[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Using mdadm to recover from a multi-drive RAID5 failure

Posted on 2006-07-20
5
Medium Priority
?
4,212 Views
Last Modified: 2013-12-16
I have a Centos4.2 server that has a software RAID-5 volume consisting of 4 disks, /dev/sd{a-d}7.  The other day it reported a failure on /dev/sdc7.  I was able to start rebuilding it but during the rebuild it reported another failure.

With the older raidtools I knew how to edit /etc/raidtab and set the known bad disk to failed-disk and then force a rebuild of the array.  I'd like to do that with this machine to see if I can recover the array, but I've never done it using mdadm.  /etc/mdadm.conf doesn't have much useful information in it, only:

ARRAY /dev/md0 super-minor=0
ARRAY /dev/md1 super-minor=1
...

How can I go about trying to tell mdadm that /dev/sdc7 is the truely failed disk and to try to rebuild using the other 3 disks?

-Bruce
0
Comment
Question by:brucepennypacker
  • 2
  • 2
5 Comments
 
LVL 43

Expert Comment

by:ravenpl
ID: 17152425
If You have two disks failed in raid5 volume You can't recover. That's the design of raid5.

If only sdc7 is failed, then
mdadm /dev/mdX -f /dev/sdc7 # hot-fail
mdadm /dev/mdX -r /dev/sdc7 # hot-remove sdc7 from mdX
mdadm /dev/mdX -a /dev/sdc7 # hot-add and start rebuilding
0
 
LVL 41

Assisted Solution

by:noci
noci earned 60 total points
ID: 17153252
Be aware that if /dev/sdc7 failed,
other partitions on /dev/sdc might also fail...., it's a bit depending on the error, if it's just bad block you might get away with it for now...

Also have a look at the smartmontools, these can help diagnose health of disks before failure.
(disks that is, not partitions) /dev/sd? , /dev/hd? etc.

Better prepare for other partitions on /dev/sdc failing.
0
 

Author Comment

by:brucepennypacker
ID: 17153830
ravenpl - As I said in my original post I have successfully recovered from multiple-disk RAID5 failures using raidtools.  It's possible to have multiple disks fail simultaneously if a drive controller fails, if a cable that multiple drives are on is loose, if an external disk array loses power, etc.  Here's a web page that describes how to do this using raidtools:

http://software.cfht.hawaii.edu/linuxpc/RAID_recovery.html

What I would like to know is how to do it using mdadm since that's replaced raidtools.
0
 
LVL 43

Accepted Solution

by:
ravenpl earned 90 total points
ID: 17153905
If the array is out of sync - You can't. If it is, just plug the disk - kernel will find new disk and use in array.

Or try assembling array from scratch
mdadm -A /dev/mdX -YourOptions -l5 -n4 /dev/sda7 /dev/sdb7 missing /dev/sdd7
but it will propably fail, if disks are unsyc.
0
 

Author Comment

by:brucepennypacker
ID: 17187474
You were close.  I just had to do the following:

mdadm --assemble  --force /dev/md5 /dev/sda7 /dev/sdb7 /dev/sdc7 /dev/sdd7

This has recreated the array successfully, still with one failed disk.  I was able to mount it, and after it recovered its journal I was able to copy all the data off before replacing the drive & rebuilding the array.
0

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Linux users are sometimes dumbfounded by the severe lack of documentation on a topic. Sometimes, the documentation is copious, but other times, you end up with some obscure "it varies depending on your distribution" over and over when searching for …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Suggested Courses
Course of the Month19 days, 3 hours left to enroll

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question