RAID 5 drives offline

Dell Poweredge 2850 Windows Server 2003 w/ Perc 4e/Di
Five 146GB Drives in RAID 5 Configuration
The machine was running fine, but a rookie tech pulled a drive to see what size it was and then put it back.  No problem until he pulled a second drive 60 seconds after replacing the first.  System hung and then blue screened. When rebooting the machine now says:

1 Logical Drive found on host adapter
1 Logical Drive failed
1 Logical Drive handeled by bios

I then go into the RAID configuration utility and see the following:
Raid Ch-0        Raid Ch-1
Onlin A00-00    Onlin A00-02
FAIL A00-01     Onlin A00-03
                         FAIL A00-04

I do not see anyway to force the 2 drives back online and I am at a loss for how to proceed.
We have some but not of the data backed up so I really would like to get the data back. The drives are not bad so surely there is a way to get them back online and get the RAID working again.

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Double-disk failure on a RAID5 set, not much you can do to recover from this unfortunately even though it was only reseated.  I have heard good things about this toolk but haven't used it personally:

Also, some manual rebuild information from Dell here:

Hope this helps, good luck
skwokIT ConsultantCommented:
On Intel Based RAID Cards it is possible to mark the drives as back online, this was a common problem with the original SAF-T Backplanes where it would fail drives and mark them offline. Rather than blow the array and reinitialize the array and reload the server, all that was needed to be done was to just mark the drives back online.  This required using the RAID configuration utlities or firmware before the OS loads.
PowerEdgeTechIT ConsultantCommented:
I presume you know then which disk he pulled FIRST and which disk he pulled SECOND?

You need to force online ONLY the one pulled SECOND.

CTRL-M during POST, Objects, Physical Disks, hit Enter on the drive that was pulled SECOND, and Force Online.

Once that has been forced online and you are able to boot to Windows ok (there may be further damage to the OS that may prevent successful booting at this point from the unexpected shutdown - but that's another story :)), then you can REBUILD the drive that was pulled FIRST.  You do NOT want to force online both drives.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
SolarWinds® IP Control Bundle (IPCB)

Combines SolarWinds IP Address Manager and User Device Tracker to help detect IP conflicts, quickly identify affected systems, and help your team take near instantaneous action. Help improve visibility and enhance reliability with SolarWinds IP Control Bundle.

PowerEdgeTechIT ConsultantCommented:
Any luck?  How's it going?
4allmemoryAuthor Commented:
I decided to hire a consultant to help because I wanted to have the highest likelyhood of recovering the data. Turns out he did essentially what you suggested except the drive order did not matter. He had me put both drives online at the same time and then everything was fine. Had I followed your advice I would have saved some money for sure, but regardless.. THANKS for offering the solution.
PowerEdgeTechIT ConsultantCommented:
Just to offer some friendly advice ... you may share it with your consultant or use it to decide whether to use him in the future ... but with two drives offline, they should NEVER be forced online together.  Let's say there was 5 minutes difference between the time that drive A and drive B were pulled.  Any data that had changed in those five minutes would be irreconcilible by the controller and lost - scrambled - as the controller tried to figure out which had the correct data.  You only force online drive B, because the very instant that drive B went offline, no further changes are made, thus when it is forced back online, all data matches across the disks.  Then, as the integrity of the data has been preserved, drive A can be REBUILT to match the array.

Anyway ... I'm glad it seems to be working ok.

Another money/data-saving tip?  In the US and Canada, tech support is always free for Dell servers, regardless of the status of the warranty.  They could have walked you through this correctly - with no cost to you.

Take care :)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Server Hardware

From novice to tech pro — start learning today.