Restoring RAID-5 configuration after RAID controller failure

Slavyan
Slavyan used Ask the Experts™
on
Hello,
We have HP ProLiant DL100 G2 with RAID-5 consisting of 2 logical volumes/arrays. The other day the RAID controller failed with a BSOD Fatal Hardware Malfunction (see screenshots attached). After a phys check on the controller and re-inserting it, rebooting the box shows No Boot Media present, but at least we can log into the controller configuration though it does NOT show any members now, arrays - none. If we go into Disk Utlities menu in the same controller menu, we can see all disks fine. So the controller lost the configuration for both arrays and now need to re-initialize the disks and re-create the arrays meaning it will wipe all data from the old members, which we DO NOT really want. We are expecting a new controller very shortly.
Now the question: how do we restore the old members to the new controller w/o losing the data? We dont care for the system volume (we have an image), but it's the data volume which is important (~500GB of data and no backup (I know what you think now, but the server has a secondary role and RAID-5 was sufficient redundancy for it all this time)).
Appreciate your help.
S.
IMG00104.jpg
IMG00105.jpg
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®

Commented:
Have you removed any of the drives?

Your first priority must be with drive order. It can not be moved. If any of the drives were taken out I would strongly suggest you try to get the driver order right.

Author

Commented:
No, all drives are intact and in their original bays.
Is Server under warranty?
If yes i strongly recommend talking to support from HP.
It would be worth for sure.
11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

Generally RAID information is stored on the disc, but get the exact same raid controller.

That was mentioned here for adaptec, but should be quite similar for hp:
http://serverfault.com/questions/52481/adaptec-3405-failed-recover-raid-5-set
Hope you have ordered exactly same controler.

Author

Commented:
"Generally RAID information is stored on the disc, " - Which one? Or is it striped across all RAID members?
The HP warranty has expired a while ago (not a critical server). The controller is Adaptec and we ordered the exact model, but we'll check anyway upon arrival. Checking your links now
Top Expert 2009
Commented:
Hopefully its just a controller issue...as you are aware, both your controller and drives store the RAID configuration.  If you delete your array from the controller with the drives installed, they're both gone.  If you initialize the drives on another controller, the array configuration on the drives will be gone.  Unless there some user intervention, it shouldn't have deleted the configuration from the disks.  When you install the new controller, just reinsert the drives.  It should identify your array.  
If it doesn't identify the array (as you are assuming), do not attempt to initialize the drives!  
If your array is identified, awesome, import the configuration from the disks.  If its not, use software to recreate the array and extract an image.  I prefer RAID Reconstructor - http://www.runtime.org/raid.htm.  With an image intact, create a new array and restore the image.
 
Be careful at each and every key press and make sure what you are doing, if have questions post them here, not matter if they sound stupid, your 500gb is at risk.

If it is adaptec, then go through the links and read all, may surely help.

Author

Commented:
Reading the links.
We're not touching the disks, the box is powered off and waiting for the controller. If your guys are right and the controller config is mostly stored on the disks and we get the correct controller, then we should be golden in theory, but then again the harsh reality may still screw up everything.
I'll continue reading and will update you with more questions or when we have the new controller. We'll certainly try the Runtime utility.
Thank you,
S.

Author

Commented:
Looks like we cannot run the RAID Reconstructor because it is run under Windows. We cannot get to Windows as we don't have any environment to connect all disks to a non-raid controller box.
Top Expert 2009

Commented:
That won't be a problem...plugins are available so it can be run from Windows PE or Bart PE.
http://www.runtime.org/peb.htm
 

Author

Commented:
but:

"Notes
We do not endorse running our programs under WinPE or a boot CD-ROM created with PE Builder (BartPE). You assume all risks from the use of our software. Since WinPE is a very reduced version of Windows XP you should run our software only if special circumstances prohibit the use in a full Windows XP. Some features might not be available and our software might behave unpredictably. Due to limited resources on WinPE our software can close unexpectedly. When using our software under a boot CD-ROM the help system and foreign language modules are not available."
Top Expert 2009

Commented:
I wouldn't worry about it...its a pretty standard disclaimer.  I've used it many times, works fine.  You don't have to purchase the software unless it works, and its read only, so you won't hurt your disks.  There's no harm in trying...and it may just s.y.a.

Commented:
Do you have another HP server which can be used to insert drives and see what it picks up from the drives?

Author

Commented:
no. see details in upper thread.
Top Expert 2009

Commented:
To answer your previous question: each disk contains your array configuration on the last 16 blocks of each drive in the array (RAID table or metadata).  Not all controllers do this the same, but the adaptec code is pretty standard.  If you still don't want to use raid reconstructor and the controller doesn't pick up your array, you can attempt to recreate the array if your controller allows you to create an array by using Skip Init, which will create a new array without initializing the drives...it basically only updates metadata.  
http://ask.adaptec.com/scripts/adaptec_tic.cfg/php.exe/enduser/std_adp.php?p_faqid=3918&p_created=1026942349&p_sid=pfP*IwIj&p_accessibility=0&p_redirect=&p_lva=1227&p_sp=cF9zcmNoPTEmcF9zb3J0X2J5PWRmbHQ6MSZwX2dyaWRzb3J0PSZwX3Jvd19jbnQ9NDIsNDImcF9wcm9kcz0mcF9jYXRzPSZwX3B2PSZwX2N2PSZwX3NlYXJjaF90eXBlPWFuc3dlcnMuc2VhcmNoX25sJnBfcGFnZT0xJnBfc2VhcmNoX3RleHQ9cmFpZCB0YWJsZQ**&p_li=&p_topview=1
 Did you try disconnecting the drives and trying to boot?  See if there is still an error.  What controller are you using?  You may want to try a low level firmware reset or firmware update.

Author

Commented:
Moderator:
This question is not abandoned. We're preparing feedback today-tomorrow. The radio silence was due to our waiting for the replacement parts as indicated above. Keep it open.
S.

Author

Commented:
gikkel,

our controller is Adaptec AAR 2410SA, it is SATA-150, not SCSI, so the article from Adaptec web site is not quite relevant. We did try disconnecting the drives and hot-plugging them back in, the controller showed "could connect to SATA ports 0, 1, 2, 3 and same error after the drives were back in bays. The Skip Init option you mention is not available on this controller, or at least we couldn't find it. We reset the firmware with the same one (as it is EOL old controller) successfully but to no effect.
We received the new controller the other day, will be trying it today. I will update the thread.
S.

Author

Commented:
correction: should read ""could NOT connect to SATA ports 0, 1, 2, 3"
Top Expert 2009

Commented:
Cool, even easier :)  Take out the drives and put them in another machine, or use a different controller (in non-raid, standard ide emulation mode).  Do not do anything to the drives, just make sure they're not listed in the boot menu.  From there either boot to Windows from another drive or use the windows pe cd.  RAID Reconstructor will help from there.  Just so you know, if it looks like I'm pushing raid reconstructor, its only because I know it works...and very well.
Are you using a backplane?  Try connecting directly to the controller.  At this point I'm unsure whether the new controller is going to identify the array, you should at least try raid reconstructor (it is free to try)...

Author

Commented:
Everyone:

We received the new 1:1 identical RAID controller and tested it - NO GO. The controller does not see any arrays, it can only initialize the drives, but this is not what we were hoping for. So this leads us to believe that the old bad controller kept all config info on itself, not on the drives, but its memory went bad now or smth and didn't come back even after the firmware reset. Next we tried the RAID Reconstructor (RR), prepared the BartPE disk as instructed with the RR plug-in and all that -> booted up to Bart -> RR didn't see any controller, hence no disks. We then re-booted Bart with F6 + Adaptec driver manually uploaded - same result, no controller visible in RR or in Captain Nemo or in any other Bart utility. In Step 1 of RR when you click on the drive to add there is only 1 available option to do - Select virtual image. So so much for the RR.
Any other suggestions before we wipe the drives back to the dark age, gentlemen?
Thanks,
S.

Author

Commented:
We don't have another box with 4 bays, only 2 is possible, but 2 won't work.

Author

Commented:
What do you mean by "backplane" in this context? Is it the board that sits right behind the drive bays and is hard-integrated with them? There is one and the 4 SATA cables are connected to it on the front and to the RAID controller at the back. The original box is a standard HP DL100 G2 server. Do you mean there are 4 SATA ports somewhere on the system board where we can connect the drives to bypassing the controller?
Slavyan: I am not sure if any solution works or not, but you should at the end or may be now, try to call some company who does this data recovery from RAID if your data is worth the recovery cost.

Again that is a risk as well, if company is not that skilled, they may just return back after screwing it and saying cant recover, you cant do anything, but atleast some hope is there.

Trying different solutions may (may not) make this worst to recover or simply bymistake may just get re-initialize, and its history.
Right now it is like playing with fire, you know that right?

If i were you, i would not have tried till here if the data was really important. I honestly respect all the EE Experts here and we all will try our best to provide solutions to you without any issues.
Probably other Experts may object on my comment but at the end its you and your data.

If you think data is not worth, then lets try our best to fix it.

Think twice!!
Top Expert 2009
Commented:
Ok, maybe I should be giving you a little more direction on booting from windows pe/bart pe...
I really think your data is still accessible for repair, don't give up!  Please read the entire comment before starting your mission, should you choose to accept :)
Yes, the backplane is that board sitting directly behind the drives, which allows you to safely hot spap the drives...take it out of the mix.  This isn't permanent, jury-rig up something to safely hold your drives in place while they're connected to the motherboard (i typically use a spare drive caddy).  I don't think the dl100 has an onboard controller  (but double check to see if there are sata ports on the board).
To make things easier, you should really try to bypass your raid controller.  You don't need to do this on a server, you can do it on any machine...there are so many desktops that have 2 onboard controllers with 6 ports or more that it shouldn't be difficult.  Its a lot easier working from windows on an intel controller (which is one of the reasons they say they don't support that method).  If its configured in standard ide emulation mode, you likely won't even need a driver.
This isn't permanent, jury-rig up something to safely hold your drives in place while they're connected to the motherboard (i typically use a spare drive caddy).  
Your issue now is that you're trying to get bart pe to recognize disks on a raid controller.  Manually adding the driver isn't going to work, you need to "inject" the driver...
Example:
http://hotware.wordpress.com/2006/09/27/creating-a-bart-pe-boot-cd-fur-dell-poweredge-2950-servers-with-perc-5i-controller/

If you want to continue with your current method, let me know...i'll help you with any questions.  Try your best to find another machine...

Author

Commented:
gikkel:

i dont know why but i didnt see your latest comment dated Sep 25, neither did i see the notification of this comment by email. anyway it's too late now as we wiped the disks 2 days ago. data is history now, we can replace it by bits and pieces from different locations hopefully.
s.

Author

Commented:
problem was not solved but tips helped anyway.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial