luks

Using mdadm to recover from a dead disk in a Linux RAID-1 array

2.5
Yes, it's that time of the year again. A disk in my desktop-replacement laptop with 2 disks and a RAID-1 has died. Time for recovery.

This laptop has been running 24/7 for the last 3 years or such, so it's not too surprising that a disk dies. Surprisingly though, for the first time in a long series of dead disks, smartctl -a does indeed show errors for this disk. Here's a short snippet of those:

  $ smartctl -a /dev/sda
  [...]
  Error 1341 occurred at disk power-on lifetime: 17614 hours (733 days + 22 hours)
   When the command that caused the error occurred, the device was active or idle.

   After command completion occurred, registers were:
   ER ST SC SN CL CH DH
   -- -- -- -- -- -- --
   40 41 02 1f c0 9c 40  Error: UNC at LBA = 0x009cc01f = 10272799

   Commands leading to the command that caused the error were:
   CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
   -- -- -- -- -- -- -- --  ----------------  --------------------
   60 f8 08 20 c0 9c 40 00  41d+01:51:50.974  READ FPDMA QUEUED
   60 08 00 18 c0 9c 40 00  41d+01:51:50.972  READ FPDMA QUEUED
   ef 10 02 00 00 00 a0 00  41d+01:51:50.972  SET FEATURES [Reserved for Serial ATA]
   ec 00 00 00 00 00 a0 00  41d+01:51:50.971  IDENTIFY DEVICE
   ef 03 45 00 00 00 a0 00  41d+01:51:50.971  SET FEATURES [Set transfer mode]

  SMART Self-test log structure revision number 1
  Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
  # 1  Short offline       Completed: read failure       90%     20511         156170102
  [...]

The status of the degraded RAID array looks like this:

  $ cat /proc/mdstat
  Personalities : [raid1] 
  md1 : active raid1 sdb7[1]
       409845696 blocks [2/1] [_U]
  md0 : active raid1 sda6[0] sdb6[1]
       291776 blocks [2/2] [UU]

The [_U] means that one of two disks has failed, it should normally be [UU]. There are two RAID-1s actually, a small md0 (sda6 + sdb6) for /boot and the main md1 (sda7 + sdb7) which holds the OS and my data. Apparently (at first at least), only sda7 was faulty and got kicked out of the array:

  $ dmesg | grep kick
  md: kicking non-fresh sda7 from array!

Anyway, so I ordered a replacement disk, removed the dead disk (I checked the serial number and brand before, so I don't accidentally remove the wrong one), inserted the new disk and rebooted.

Note: In order for this to work you have to have (previously) installed the bootloader (usually GRUB) onto both disks, otherwise you won't be able to boot from either of them (which you'll want to do if one of them dies, of course). In my case, sda was now dead, so I put sdb into its place (physically, by using the other SATA connector/port) and the new replacement disk would become the new sdb.

After the reboot, the new disk needs to be partitioned like the other RAID disk. This can be done easily by copying the partition layout of the "good" disk (now sda after the reboot) onto the empty disk (sdb):

  $ sfdisk -d /dev/sda | sfdisk /dev/sdb

Specifically, the RAID disks/partitions need to have the type/ID "fd" ("Linux raid autodetect"), check if that is the case. Then, you can add the new disk to the RAIDs:

  $ mdadm /dev/md0 --add /dev/sdb6
  $ mdadm /dev/md1 --add /dev/sdb7

After a few hours the RAID will be re-synced properly and all is good again. You can check the progress via:

  $ watch -n 1 cat /proc/mdstat

You should probably not reboot during the resync (though I'm not 100% sure if that would be an issue in practice; please leave a comment if you know).

Also, don't forget to install GRUB on the new disk so you can still boot when the next disk dies:

  $ grub-mkdevicemap
  $ grub-install /dev/sdb

And it might be a good idea to use S.M.A.R.T. to check the new disk, just in case. I did a quick run for the new disk via:

  $ smartctl -t short /dev/sdb # Wait a few minutes after this.
  $ smartctl -a /dev/sdb
  [...]
  SMART Self-test log structure revision number 1
  Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
  # 1  Short offline       Completed without error       00%        22         -
  [...]

Looks good. So far.

How to setup an encrypted USB-disk software-RAID-1 on Debian GNU/Linux using mdadm and cryptsetup

This is what I set up for backups recently using a cheap USB-enclosure which can house 2 SATA disks and shows them as 2 USB mass-storage devices to my system (using only one USB cable). Without any further introduction, here goes the HOWTO:

First, create one big partition on each of the two disks (/dev/sdc and /dev/sdd in my case) of the exact same size. The cfdisk details are omitted here.

  $ cfdisk /dev/sdc
  $ cfdisk /dev/sdd

Then, create a new RAID array using the mdadm utility:

  $ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1

The array is named md0, consists of the two devices (--raid-devices=2) /dev/sdc1 and /dev/sdd1, and it's a RAID-1 array, i.e. data is simply mirrored on both disks so if one of them fails you don't lose data (--level=1). After this has been done the array will be synchronized so that both disks contain the same data (this process will take a long time). You can watch the current status via:

  $ cat /proc/mdstat
  Personalities : [raid1]
  md0 : active raid1 sdd1[1] sdc1[0]
        1465135869 blocks super 1.1 [2/2] [UU]
        [>....................]  resync =  0.0% (70016/1465135869) finish=2440.6min speed=10002K/sec
  unused devices: 

Some more info is also available from mdadm:

  $ mdadm --detail --scan
  ARRAY /dev/md0 metadata=1.01 name=foobar:0 UUID=1234578:1234578:1234578:1234578

  $ mdadm --detail /dev/md0
  /dev/md0:
          Version : 1.01
    Creation Time : Sat Feb  6 23:58:51 2010
       Raid Level : raid1
       Array Size : 1465135869 (1397.26 GiB 1500.30 GB)
    Used Dev Size : 1465135869 (1397.26 GiB 1500.30 GB)
     Raid Devices : 2
    Total Devices : 2
      Persistence : Superblock is persistent
      Update Time : Sun Feb  7 00:03:21 2010
            State : active, resyncing
   Active Devices : 2
  Working Devices : 2
   Failed Devices : 0
    Spare Devices : 0
   Rebuild Status : 0% complete
             Name : foobar:0  (local to host foobar)
             UUID : 1234578:1234578:1234578:1234578
           Events : 1
      Number   Major   Minor   RaidDevice State
         0       8       33        0      active sync   /dev/sdc1
         1       8       49        1      active sync   /dev/sdd1

Next, you'll want to create a big partition on the RAID device (cfdisk details omitted)...

  $ cfdisk /dev/md0

...and then encrypt all the (future) data on the device using dm-crypt+LUKS and cryptsetup:

  $ cryptsetup --verbose --verify-passphrase luksFormat /dev/md0p1
  Enter your desired pasphrase here (twice)
  $ cryptsetup luksOpen /dev/md0p1 myraid

After opening the encrypted container with cryptsetup luksOpen you can create a filesystem on it (ext3 in my case):

  $ mkfs.ext3 -j -m 0 /dev/mapper/myraid

That's about it. In future you can access the RAID data by using the steps below.

Starting the RAID and mouting the drive:

  $ mdadm --assemble /dev/md0 /dev/sdc1 /dev/sdd1
  $ cryptsetup luksOpen /dev/md0p1 myraid
  $ mount -t ext3 /dev/mapper/myraid /mnt

Shutting down the RAID:

  $ umount /mnt
  $ cryptsetup luksClose myraid
  $ mdadm --stop /dev/md0

That's all. Performance is shitty due to all the data being shoved out over one USB cable (and USB itself being too slow for these amounts of data), but I don't care too much about that as this setup is meant for backups, not performance-critical stuff.

Update 04/2011: Thanks to Bohdan Zograf there's a Belorussian translation of this article now!

HOWTO: Disk encryption with dm-crypt / LUKS and Debian [Update]

A few weeks ago I published a small HOWTO for using loop-aes to encrypt your hard drive, usb thumb drive etc.

As I have bought a new 300 GB external USB disk drive on Friday, I have tried something new this time: disk encryption using dm-crypt / LUKS. It has been suggested to me multiple times that dm-crypt is superior to loop-aes, however I didn't get a real reason. Yes, it doesn't require any kernel patches and is easier to setup. But has any serious cryptographer looked at it sharply, yet? Did it withhold his eye contact?

Anyways, here's how I encrypted my 300 GB drive. I largely followed the guide at the EncryptedDeviceUsingLUKS wiki page...

  1. Make sure you run Linux 2.6.16 or better. Previous versions suffer from an implementation problem which affects the security of dm-crypt, see Linux Kernel dm-crypt Local Cryptographic Key Disclosure.
  2. Enable the following options in your kernel:

    • Code maturity level options
      • Prompt for development and/or incomplete code/drivers
    • Device Drivers -> Multi-device support (RAID and LVM)
      • Device mapper support
      • Crypt target support
    • Cryptographic options
      • AES cipher algorithms
  3. Overwrite the whole drive with random data in order to slow down attacks on the encryption. At the same time perform a bad blocks scan to make sure the hard drive is not going to die too soon:
    badblocks -c 10240 -s -w -t random -v /dev/sdb
    Replace /dev/sdb with whatever is correct on your system. If you're really paranoid, and are willing to wait one or two days, do this:
    dd if=/dev/urandom of=/dev/sdb
  4. Install the required packages:
    apt-get install cryptsetup
    The current cryptsetup in Debian unstable already supports LUKS, which was not the case a while ago, if I'm not mistaken. So Debian testing or stable will most probably not work!
  5. Create one or more partitions on the drive:
    cfdisk /dev/sdb
    I created one big 300 GB partition, /dev/sdb1.
  6. Setup LUKS:
    cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb1
    Enter a good passphrase here. Don't spoil the whole endeavour by chosing a stupid or short passphrase.
  7. Open the encrypted device and assign it to a virtual /dev/mapper/samsung300gb device:
    cryptsetup luksOpen /dev/sdb1 samsung300gb
  8. Create a filesystem on the encrypted device:
    mkfs.ext3 -j -m 1 -O dir_index,filetype,sparse_super /dev/mapper/samsung300gb
    I used ext3 with some optimizations, see mke2fs(8).
  9. Mount the encrypted partition:
    mkdir /mnt/samsung300gb
    mount /dev/mapper/samsung300gb /mnt/samsung300gb
    That's it. Everything you write to /mnt/samsung300gb will be encrypted transparently.
  10. For unmounting use:
    umount /mnt/samsung300gb
    cryptsetup luksClose /dev/mapper/samsung300gb

After unmounting, nobody will be able to see your data without knowing the correct passphrase. Drive is stolen? No problem. Drive is broken, and you want to send it in for repair without the guys there poking in your data? No problem. You leave the USB drive at home and some jerk breaks into your house, steals your drive, rapes your wife, and kills your kids? No problem. Well, sort of, but you get the idea ;-)

There's more things you can do, thanks to LUKS: have multiple passphrases which unlock your data, change/add/remove passphrases as you see fit, etc.

Comments?

Update 2006-04-17: You have to use cryptsetup from unstable if you want LUKS support. cryptsetup in testing does not support this (thanks Ariel).

Syndicate content