linux

Recovering from a dead disk in a Linux software-RAID5 system using mdadm

RAID5 failure

As I wrote quite a while ago, I set up a RAID5 with three
IDE disks at home, which I'm using as backup (yes, I know that
RAID != backup) and storage space.

A few days ago, the RAID was put to a real-life test for the first time, as one of the disks died. Here's what that looks like in dmesg:

raid5: raid level 5 set md1 active with 3 out of 3 devices, algorithm 2
RAID5 conf printout:
 --- rd:3 wd:3
 disk 0, o:1, dev:hda2
 disk 1, o:1, dev:hdg2
 disk 2, o:1, dev:hde2
[...]
hdg: dma_timer_expiry: dma status == 0x21
hdg: DMA timeout error
hdg: 4 bytes in FIFO
hdg: dma timeout error: status=0x50 { DriveReady SeekComplete }
ide: failed opcode was: unknown
hdg: dma_timer_expiry: dma status == 0x21
hdg: DMA timeout error
hdg: 252 bytes in FIFO
hdg: dma timeout error: status=0x50 { DriveReady SeekComplete }
ide: failed opcode was: unknown
hdg: dma_timer_expiry: dma status == 0x21
hdg: DMA timeout error
hdg: 252 bytes in FIFO
hdg: dma timeout error: status=0x58 { DriveReady SeekComplete DataRequest }
ide: failed opcode was: unknown
hdg: DMA disabled
ide3: reset: success
hdg: dma_timer_expiry: dma status == 0x21
hdg: DMA timeout error
hdg: 252 bytes in FIFO
hdg: dma timeout error: status=0x58 { DriveReady SeekComplete DataRequest }
ide: failed opcode was: unknown
hdg: DMA disabled
ide3: reset: success
hdg: status timeout: status=0x80 { Busy }
ide: failed opcode was: 0xea
hdg: drive not ready for command
hdg: lost interrupt
hdg: task_out_intr: status=0x50 { DriveReady SeekComplete }
ide: failed opcode was: unknown
hdg: lost interrupt
hdg: task_out_intr: status=0x50 { DriveReady SeekComplete }
ide: failed opcode was: unknown

That's when I realized that something was horribly wrong.

Not long after that, these messages appeared in dmesg. As you can see the software-RAID automatically realized that a drive died and removed the faulty disk from the array. I did not lose any data, and the system did not freeze up; I could continue working as if nothing happened (as it should be).

 md: super_written gets error=-5, uptodate=0
 raid5: Disk failure on hdg2, disabling device.
 raid5: Operation continuing on 2 devices.
 RAID5 conf printout:
  --- rd:3 wd:2
  disk 0, o:1, dev:hda2
  disk 1, o:0, dev:hdg2
  disk 2, o:1, dev:hde2
 RAID5 conf printout:
  --- rd:3 wd:2
  disk 0, o:1, dev:hda2
  disk 2, o:1, dev:hde2

This is how you can check the current RAID status:

 $ cat /proc/mdstat
 Personalities : [raid6] [raid5] [raid4] 
 md1 : active raid5 hda2[0] hde2[2] hdg2[3](F)
       584107136 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]

The "U_U" means two of the disks are OK, and one is faulty/removed. The desired state is "UUU", which means all three disks are OK.

The next steps are to replace the dead drive with a new one, but first you should know exactly which disk you need to remove (in my case: hda, hde, or hdg). If you remove the wrong one, you're screwed. The RAID will be dead and all your data will be lost (RAID5 can survive only one dead disk at a time).

The safest way (IMHO) to know which disk to remove is to write down the serial number of the disk, e.g. using smartctl, and then check the back side of each disk for the matching serial number.

 $ smartctl -i /dev/hda | grep Serial
 $ smartctl -i /dev/hde | grep Serial
 $ smartctl -i /dev/hdg | grep Serial

(ideally you should get the serial numbers before one of the disks dies)

Now power down the PC and remove the correct drive. Get a new drive which is at least as big as the one you removed. As this is software-RAID you have quite a lot of flexibility; the new drive doesn't have to be from the same vendor / series, it doesn't even have to be of the same type (e.g. I got a SATA disk instead of another IDE one).

Insert the drive into some other PC in order to partition it correctly (e.g. using fdisk or cfdisk). In my case I needed a 1 GB /boot partition for GRUB, and the rest of the drive is another partition of the type "Linux RAID auto", which the software-RAID will then recognize.

Then, put the drive into the RAID PC and power it up. After a successful boot (remember, 2 out of 3 disks in RAID5 are sufficient for a working system) you'll have to hook-up the new drive into the RAID:

 $ mdadm --manage /dev/md1 --add /dev/sda2
 mdadm: added /dev/sda2

My new SATA drive ended up being /dev/sda2, which I added using mdadm. The RAID immediately starts restoring/resyncing all data on that drive, which may take a while (2-3 hours, depends on the RAID size and some other factors). You can check the current progress with:

 $ cat /proc/mdstat 
 Personalities : [raid6] [raid5] [raid4] 
 md1 : active raid5 sda2[3] hda2[0] hde2[2]
       584107136 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U]
       [>....................]  recovery =  0.1% (473692/292053568) finish=92.3min speed=52632K/sec

As soon as this process is finished you'll see this in dmesg:

 md: md1: recovery done.
 RAID5 conf printout:
  --- rd:3 wd:3
  disk 0, o:1, dev:hda2
  disk 1, o:1, dev:sda2
  disk 2, o:1, dev:hde2

In /proc/mdstat you'll see "UUU" again, which means your RAID is fully functional and redundant (with three disks) again. Yay.

 $ cat /proc/mdstat
 Personalities : [raid6] [raid5] [raid4] 
 md1 : active raid5 sda2[1] hda2[0] hde2[2]
       584107136 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

Btw, another nice utility you might find useful is hddtemp, which can check the temperature of the drives. You should take care that they don't get too hot, especially so if the RAID runs 24/7.

 $ hddtemp /dev/hda
 dev/hda: SAMSUNG HD300LD: 38 °C
 $ hddtemp /dev/hde
 dev/hde: SAMSUNG HD300LD: 44 °C
 $ hddtemp /dev/sda
 dev/sda: SAMSUNG HD322HJ: 32 °C

Running homebrew and open-source software on your Wii, using the Twilight Hack and the Homebrew Channel

Wii

I've been owning a Nintendo Wii for quite a while now, but so far only played an occasional game of tennis or the like. Over the holidays I got The Legend of Zelda: Twilight Princess, which is a really nice game in itself. What is even more interesting though, is that this game allows you to run homebrew software on your Wii.

What you usually want to do, is to use the so-called Twilight Hack to install the Homebrew Channel on your Wii (you'll need an SD card and the Twilight Princess game for this). Once the process has finished you won't need the game anymore though.

So, here's a quick tutorial on how to install the Homebrew Channel. I basically followed the Setting up your Wii for Homebrew HOWTO from wiibrew.org.

Requirements

An SD card max. 2GB in size (not an SDHC card!), an SD card reader in your PC or laptop, and the Twilight Princess game (which you must play at least once).

Backup

First, you'll want to backup your Twilight Princess save games (if any). For that, we format the SD card (it needs a FAT16/FAT32) filesystem.

 $ mkfs.vfat /dev/mmcblk0p1 (you may need to change "mmcblk0p1" to whatever fits your setup)

Now insert the SD card into the Wii, start the Wii, go to Wii Options --> Data Management --> Save Data --> Wii and copy your Twilight Princess save games onto the SD card. Then, poweroff, remove the SD card, insert it into your laptop/PC and copy the save games to a safe place.

 $ mount -t vfat /dev/mmcblk0p1 /mnt
 $ mv /mnt/private ~/zelda_savegames

Twilight Hack and Homebrew Channel

The Legend of Zelda: Twilight Princess

Next up: download the Twilight Hack and Homebrew Channel, and put the files on the SD card:

 $ wget http://hbc.hackmii.com/dist/twilight-hack-v0.1-beta1.zip
 $ unzip twilight-hack-v0.1-beta1.zip
 $ cp -r private /mnt
 $ wget http://hbc.hackmii.com/dist/the_homebrew_channel-1.0.1.tar.gz
 $ tar xfvz the_homebrew_channel-1.0.1.tar.gz
 $ cp the_homebrew_channel-1.0.1/boot.dol /mnt
 $ umount /mnt

Now remove the SD card, insert it into the Wii and power-up the Wii. Go to Wii Options --> Data Management --> Save Data --> Wii, delete the Wii Twilight Princess save games, then copy the "Twilight Hack" save game from the SD card onto the Wii (you need to select the correct one for your region).

Quit the menu, start Twilight Princess, load the "Twilight Hack" savegame and finally talk to the person in front of you (do not perform any other actions, or the Wii may crash!). Follow the remaining instructions on the screen and you're done. You now have an additional channel named Homebrew Channel where all your homebrew programs/games (and your own Free Software programs you may write/port) will appear.

Installing homebrew software

So far, there are no real applications in the Homebrew Channel, you need to put all the homebrew software you want on the SD card. There's a huuuge list of applications and games to choose from, many of them Free Software with source code, some however are binary-only.

Basically, you put each application in a sub-directory of apps/ on the SD card, the most important file in every sub-directory is always boot.dol. If you have a boot.elf file instead, you'll probably need to convert it using the ELF to DOL converter.

Restoring Twilight Princess save games

You may now want to restore your Twilight Princess save games onto the Wii, you no longer need the "Twilight Hack" save game. Put your ~/zelda_savegames directory on the SD card and name it private/ again. Insert the SD card into the Wii and copy the save games on the Wii (similar procedure as above).

Have fun with homebrew software on your Wii, or even better write your own software or port existing Linux applications/games!

Playing Starcraft on Linux using Wine

Wine logo

Here's a quick HOWTO for using Wine to play Starcraft on a Linux machine.

Starcraft Installation

  $ apt-get install wine (as root)
  $ winecfg

The winecfg (graphical) utility will setup some config file defaults in your ~/.wine directory. Click on Graphics and activate Allow DirectX apps to stop the mouse leaving their window. Also, click on Audio (a dialog will pop up, just click OK). This will autodect your soundcard and setup Wine to use it. Under Drives click Add (this will add D:) and change the path to /media/cdrom, so that Wine knows about your CD-ROM drive. Finally click OK to close winecfg and save the settings.

winecfg screenshot

The next step is to insert the Starcraft CD-ROM into the drive and start the installer using Wine:

  $ mount /media/cdrom (as root)
  $ wine /media/cdrom/setup.exe

Follow the instructions in the installer until the Starcraft install is finished (you'll need your CD key number), then exit the installer (don't start playing Starcraft right away).

The next step is to get the latest patch and get rid of the need to insert the CD-ROM every time.

  $ wget http://ftp.blizzard.com/pub/starcraft/patches/PC/SC-1161.exe
  $ wine SC-1161.exe

After the patch is installed click OK and Starcraft will be started (very annoying). Leave the game again. We'll get rid of the CD-ROM requirement now:

  $ cp /media/cdrom/install.exe ~/.wine/drive_c/Programme/Starcraft/StarCraft.mpq

That's a pretty big file, it may take a while. You might have to change "Programme" in the path (I have the German Starcraft version). That's it. You can now play Starcraft (without needing the CD-ROM) using:

  $ wine ~/.wine/drive_c/Programme/Starcraft/StarCraft.exe

Notes

A good thing is, it even works nice and fast with the open-source nv NVIDIA driver (no need to install the proprietary driver).

I noticed one very annoying "bug" with the mouse behaviour at first. The mouse would sometimes just get stuck during the game (which is a total disaster of course, if you're in the middle of a fast-paced game). Left-clicking somewhere would "unstuck" the mouse, but it's still very bad. After many, many hours of reading bugreports and trying various patches I finally found out the root cause for the problem.

It's somehow related to my window manager (IceWM); whenever you move the mouse to the bottom of the Starcraft screen (where the IceWM status bar is, even though it's not on top or even visible, and even though Wine/Starcraft runs in full-screen mode!), something funny happens with X11/IceWM and the mouse gets stuck. I haven't yet found out if/which IceWM option could fix this behavior, but I have a small work-around. Just start Wine directly on a second X11 server with Starcraft (without any window manager being involved):

  $ xinit -e '/usr/bin/wine ~/.wine/drive_c/Programme/Starcraft/StarCraft.exe' -- :1

No patches needed (stock Wine from Debian unstable works fine, that's version 1.0.1 right now). I hope this saves other people some debugging time...

Brood War Installation

In order to play the Brood War expansion you can follow a similar procedure. Insert the Brood War CD-ROM, then:

  $ mount /media/cdrom (as root)
  $ wine /media/cdrom/setup.exe
  $ cp /media/cdrom/install.exe ~/.wine/drive_c/Programme/Starcraft/BroodWar.mpq
  $ wget http://ftp.blizzard.com/pub/broodwar/patches/PC/BW-1161.exe
  $ wine BW-1161.exe

After you've done that, you can start both Starcraft (classic) and Brood War via:

  $ wine ~/.wine/drive_c/Programme/Starcraft/StarCraft.exe

You will be asked in the game whether you want to actually play the Starcraft or Brood War variant.

Reducing CPU load

As of version 1161 for the Starcraft / Brood War patch, there's a new game option which can drastically lower the CPU load while playing Starcraft. First fire up Starcraft and start any game. Then, press F10, select Options / Game speed, and check the "Enable CPU Throttling box". You'll probably need to restart Starcraft afterwards.

Multiplayer and Firewalls

Multiplayer LAN games work just fine (didn't try BattleNet that much yet), but if you use a strict firewall rule set as I do (which blocks most ingress as well as egress traffic) you have to open a number of different ports. Here's what I added to my firewall script:

  $IPTABLES -A OUTPUT -m state --state NEW -p udp --dport 6111 -j ACCEPT
  $IPTABLES -A INPUT -m state --state NEW -p udp --dport 6111 -j ACCEPT
  $IPTABLES -A OUTPUT -m state --state NEW -p udp --dport 6112 -j ACCEPT
  $IPTABLES -A OUTPUT -m state --state NEW -p tcp --dport 6112 -j ACCEPT # BattleNet

Starcraft on netbooks

One A110 netbook running Starcraft

Starcraft works just fine on various netbooks; for instance, I tested it on my One A110 netbook (VIA VX800) with 256 MB of RAM, and the whole .wine directory being on a USB thumb drive (thus slow; but my internal SSD was already full). I bet it'll also work fine on the ASUS Eee PC and other netbooks...

Audio works fine, and game speed is quite OK, the only minor "problem" is that you should use an external USB mouse, the touchpad is just too small (and too slow to use) for such a fast-paced game.

The full Wine package (and all dependencies) consume quite a lot of space on the (usually very small) hard drive or SSD of a netbook, but luckily you can get away with only a minimal Wine install for playing Starcraft:

  $ apt-get install wine-bin libwine-alsa (as root)

That's sufficient, and a lot smaller than installing the full wine package.

Update 2010-06-23: There's a contributed Hungarian translation now (thanks!)
Update 2009-03-04: Added info about patch 1161 and CPU load reduction.
Update 2008-12-19: Added Starcraft-on-netbooks section.
Update 2008-12-13: Added BroodWar and multiplayer info.

jhead - List and modify EXIF fields in JPEG photos

jhead is a very nice and very powerful command line utility to mess with JPEG headers (esp. EXIF fields).

  $ apt-get install jhead

It can display/extract a great amount of metadata fields from JPEG files and also extract the thumbnails stored in JPEG files (if any). The following will list all known metadata fields from a sample photo:

  $ wget http://farm4.static.flickr.com/3173/3061542361_60acb0904b_o.jpg
  $ jhead *.jpg
  File name    : 3061542361_60acb0904b_o.jpg
  File size    : 1074172 bytes
  File date    : 2008:11:26 23:38:04
  Camera make  : Panasonic
  Camera model : DMC-FZ18
  Date/Time    : 2008:03:05 15:45:52
  Resolution   : 3264 x 2448
  Flash used   : No
  Focal length : 4.6mm  (35mm equivalent: 28mm)
  Exposure time: 0.0100 s  (1/100)
  Aperture     : f/3.6
  ISO equiv.   : 100
  Whitebalance : Auto
  Metering Mode: matrix
  Exposure     : program (auto)
  GPS Latitude : N %:.7fd %;.8fm %;.8fs
  GPS Longitude: E %;.8fd %:.7fm %;.8fs
  GPS Altitude : 174.00m
  Comment      : Aufgenommen auf dem <a href="http://www.froutes.de/TT00000014_Ars_Natura">Kunstweg Ars Natura</a>.
  ======= IPTC data: =======
  Record vers.  : 4
  Headline      : Felsburg auf dem Felsberg
  (C)Notice     : www.froutes.de
  Caption       : Aufgenommen auf dem <a href="http://www.froutes.de/TT00000014_Ars_Natura">Kunstweg Ars Natura</a>.

As you can see there's a huge amount of potentially privacy-sensitive metadata in your typical JPEG as generated by your camera (including camera type, settings, date/time, maybe even GPS coordinates of your location, etc).

You can extract the thumbnail stored in all JPEGs in the current directory with:

  $ jhead -st "&i_t.jpg" *.jpg
  Created: '3061542361_60acb0904b_o.jpg_t.jpg'

Random flickr image and its differing thumbnail

Note that the JPEG thumbnail does not necessarily show the same picture as the JPEG itself. Depending on the image manipulation software that was used to create the edited/fixed/cropped JPEG, the thumbnail may still reflect the original JPEG contents (see sample image on the right-hand side). This is a huge potential privacy issue. There have been a number of articles about this some years ago, in case you missed them:

Thus, an important jhead command line to know is the following, which removes all metadata (including any thumbnails) from all JPEG images in the current directory:

  $ jhead -purejpg *.jpg
  Modified: 3061542361_60acb0904b_o.jpg

As you can see the result is that only very basic information can be gathered from the file afterwards:

  $ jhead *.jpg
  File name    : 3061542361_60acb0904b_o.jpg
  File size    : 1052506 bytes
  File date    : 2008:11:26 23:38:04
  Resolution   : 3264 x 2448
  $ jhead -st "&i_t.jpg" *.jpg
  Image contains no thumbnail

I recommend doing this for most photos you make publically available on sites like flickr etc. (unless you have a good reason not to). Finally, see the jhead(1) manpage for lots more options that the tool supports.

OpenOCD, a Free Software JTAG utility with ARM and MIPS support

JTAG adapter for parallel port

Just FYI, I've recently updated the OpenOCD Debian package in unstable. OpenOCD is a Free Software JTAG utility which currently supports quite a large number of JTAG adapters and various CPUs/targets (many ARM and now also some MIPS ones). It's being used by a number of Free Software related projects such as OpenMoko and many others.

Here's an example of how you usually use the (new) OpenOCD with a cheapo parallel port JTAG device. First, start the OpenOCD server, providing it an interface config file and a target config file (you can copy/adapt them from /usr/lib/openocd/{interface,target}/*.cfg, or use those files directly if they work for your target, of course).

  $ openocd -f parport.cfg -f lpc2148.cfg

Then, in another xterm for example, connect to the now-running OpenOCD telnet server. Here you can now run various commands to probe, control and program the JTAG device(s). Try help for a list of commands. As an example, for flashing a binary onto some LPC2148 eval board you would do something like this:

  $ telnet localhost 4444
  Trying 127.0.0.1...
  Connected to localhost.
  Escape character is '^]'.
  Open On-Chip Debugger
  > reset init
  JTAG device found: 0x4f1f0f0f (Manufacturer: 0x787, Part: 0xf1f0, Version: 0x4)
  srst pulls trst - can not reset into halted mode. Issuing halt after reset.
  target state: halted
  target halted in Thumb state due to debug-request, current mode: Supervisor
  cpsr: 0x800000f3 pc: 0x7fffd2a2
  requesting target halt and executing a soft reset
  target state: halted
  target halted in ARM state due to debug-request, current mode: Supervisor
  cpsr: 0x800000d3 pc: 0x00000000
  > flash write_image /home/foo/program.bin 0
  wrote 1236 byte from file /home/foo/program.bin in 0.533683s (2.261701 kb/s)
  > resume 0

The final resume 0 will start to execute your program on the ARM LPC2148 microcontroller.

Check out the openocd info page (info openocd on the command line) for lots more documentation.

Syndicate content