Hard drive crash
At least a couple of you out there are probably wondering what the hell just happened. The mp3s disappeared, the home directories were missing…it’s like a hard drive failed!
For reference, here’s what the drive layout in fiona looked like last week: everything booted off of an 80GB WD drive, which also contained all the main partitions—home directories, /var, /usr and the cvs repository. The mp3s, movies, and miscellaneous other were on a volume spread across two drives, an 80GB IBM Deathstar and a 160GB Maxtor. Over the weekend I purchased another 160GB Maxtor drive, with the intention of using that to replace the 80GB Deathstar, in turn using that to replace the 20GB drive I currently use in bunbun. Everything started off fine. I got the extra drive shoved in there, started LVM’s excruciatingly slow move process off of the IBM drive (it wouldn’t be so bad if there were more numbers scrolling by faster. Maybe 10000 units for a 80GB move instead of 2000), played some mp3s while this was all running as a demonstration to myself that LVM is pretty badass, oh, but then that Western Digital drive exploded. The system crashed and became unbootable, and the pvmove stopped in a state that left my mp3s inaccessible. I had fortunately burned backups of the contents of the dead drive earlier that day, but I didn’t have the sense to backup the data that I was actually screwing around with, so while I could have restored most of the system fairly easily, I would have lost three weeks of new mp3s and the horrendously tedious tagging of ripped CDs. It was pretty late by this point, so I just restored enough data to make bunbun’s 20GB drive look kind of like fiona if you turned your head sideways and squinted, booted off that and went to bed.
The next evening I was able to examine the situation a little more closely. The WD drive wasn’t totally dead, but it had some rather crucial bad blocks where the root partition used to be. To add more insult to the matter, the music volume wasn’t turning on because LVM couldn’t find a single physical extent out of about 4500, and the damage done by the interrupted pvmove could only be undone if I could get to the metadata backups on the bad partition of the dying drive. My eventual course of action was to make an image of the dying partition, seeking and skipping around bad blocks so I could get a mountable file with a few holes instead of an unusable block device, copied the lvmconf files off of that, and reset the music drives to their state just before the third drive was added. After that I moved everything off the 80GB IBM drive, successfully this time, set that up as the new primary drive and put the 20GB back in bunbun. I’m back where I started as far the workstation, but the mp3 server has another 80GB to play with, so I guess that’s a success. Also, doing backups now.