One Drive Down, But Still No Option to Rebuild RAID5 Array

I have two very old servers which have been in production for about 7 years.  They are both HP ProLiant DL380 G5 with dual CPU, dual PSU, 64GB RAM and 8x300GB 10K SAS drives.  They are identical in every way.  Because they're old, I keep a spare PSU and several spare drives on hand, but I've never had an issue.

One was being used for some basic hosting stuff for some of our internal services - FTP for onsite backup replication to our datacenter, secondary DNS server for a few zones; like that.

The other was hosting about 7 virtual machines.  One was an old mail server that was servicing one domain for a friend and it also ran some secondary DNS zones.  All of the others were for customers, but were secondary servers for other services.  One VM had important data that (I found out) the customer never backed up.

Yesterday morning, the VM host server went offline due to a failed array.  The array on that server is a RAID5 using 7 of the 8 disks.  The 8th disk was a hot spare.

Booting into the CLI Array Configuration Utility during POST, I found that disks 1 and 2 weren't listed at all.  Disks 3, 4, 5 &  7 were good.  Disk 6 was bad.  Disk 8 was good and was still the hot spare.

I swapped Disk 6 with a new drive and rebooted and received a "drive mismatch" error in the ACU.

Assuming the RAID controller had died a horrible death, I migrated the services off of the other DL380, removed the drives and placed the drives from the VM server into the chassis in the same order.

Booted up with exact same result.

I downloaded the offline ACU ISO from HP and created a bootable USB thumb drive.  Booted up with only Drive 1 inserted.  The ACU showed Drive 1 was good and the others were missing.  I added Drive 2 and rescanned the system and then refreshed.  Drive 2 came up as good.  Same thing with  3, 4, 5,7, and 8 and all came up as good.  Inserted Drive 6 and it came up as bad.  I rebooted with all 8 drives inserted and 1-5 were good, 6 was bad, 7-8 were good.

This behavior was exactly the same when I put the drives back into the original chassis.

The end result is 6 of the 7 drives in the array are recognized and are "Good".  The hot spare is also recognized and is "Good".

My problem is - even though they're all detected, are "Good" and are known members of the array, I still can't boot and I still don't have an option to rebuild the array.

Any ideas would be greatly  appreciated.
LVL 16
Joseph HornseyPresident and JanitorAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Derek SouterITO Svc Delivery Cons IIICommented:
it's been a while since I did hardware - but I seem to recall a similar issue.  HP's definition of a hot spare is not the same as the rest of the universe.

the correct steps (at least from memory, someone else may be able to help with more recent information) - remove failed drive, replace with hot spare, ensure rebuild working, install new drive, assign as hot spare.

I would take disk 8 (your hot spare), and put it in slot 6 (your faulty disk) - just remove the faulty disk 6 and dispose of it.  

I notice that you said you replaced disk 6 with a new drive - but then you say that it keeps showing as bad - does the replacement drive exactly match the original - or is at least a physically larger drive?   if it is not a good enough match, then

it would be useful to see screenshots of your ACU to better help

you can also have a look through similar issues on this link - https://community.spiceworks.com/topic/235027-hp-dl380-g5-raid-1-hdd-rebuild-help

you may be able to find something there to assist.
Marshal HubsEmail ConsultantCommented:
Hi Joseph Hornsey,
You need not worry about this situation all you need to do to get your data back is connect the Hard Drive No 1,2,3,4,5,6 and 7 in correct order to motherboard of a normal PC(if your motherboard does not have enough ports then buy a SATA extension card). Remember to not put in the 8th drive since it still is a hot spare and does not have any data in it. Once you’ve done this download Stellar Phoenix Windows Data Recovery-Technician from here https://www.stellarinfo.com/windows-raid-recovery.php , Select Raid Recovery and then correct the Hard Disk order and put the specifications of the RAID array which include Stipe Size, Parity Order, Probable Start Sector and Parity Repetition(PS: Even if you don’t know all of these specifications, you can recover the data with the help of the software’s Don’t Know Feature.) Once all of this is set your Parity Order Map will be shown correctly with all the parity bits in a diagonal order. Then press Build Array, once its complete your RAID array and hard drives will be visible. Now start the Recovery Process by selecting your RAID Volume( Please be careful not to select the Volume your current OS is running or it may become damaged ) and start the scan.

Once the scan is complete the recoverable files would be shown in the ‘Tree View’, and then you can recover the wanted files easily.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
andyalderSaggar maker's framemakerCommented:
>I swapped Disk 6 with a new drive and rebooted

You should never swap a disk on an HP Smart Array controller with it powered off. The controller will read the metadata off the disk (if it is second hand), realise it was once part of an array and refuse to use it in case there is important data on it. Removing it and plugging it in hot will sort out that problem. The controller may even disable the original logical disks if it can't make sense of the mismatched disks which sounds like what has happened to you.

If you run the ACU off SmartStart CD or under Windows (or under VMware but you have to use HPACUCLI for that) and upload it I can tell you what the problem is. Haven't used an ACU ISO but it probably has a diagnostic tab where you can generate an ADU report same as SmartStart CD.

Old or new controller it doesn't matter, the ADU report is the same format.
Need More Insight Into What’s Killing Your Network

Flow data analysis from SolarWinds NetFlow Traffic Analyzer (NTA), along with Network Performance Monitor (NPM), can give you deeper visibility into your network’s traffic.

Joseph HornseyPresident and JanitorAuthor Commented:
@andyalder - I did the swap while it was powered on.  The HP ACU ISO is a download which is basically  the same thing as booting from the SmartStart CD, except it only  includes the various ACU's - GUI, CLI, etc.

@Derek - I'll definitely give this a try.  I was pretty disappointed with the "hot spare" thing... I mean, seriously:  If you  have to manually swap it, it's NOT a "hot" spare.

@Marshal  - That looks very  promising.  Can I boot with it, or do I have to put the drives in a PC?  That would be a little difficult.

I'll also download the report and upload it here.
Marshal HubsEmail ConsultantCommented:
Hi Joseph,

Glad you found it promising, no the software is not bootable, you’ll have to install the software on a Windows PC and then connect the RAID hard drives in the correct order. Please remember to attach an additional storage equal to the size of the array so that the recovery takes place without any stoppages.
Joseph HornseyPresident and JanitorAuthor Commented:
Marshall, I downloaded Stellar's Phoenix and it was exactly what you described.  However, I poked around a bit and chose to use RStudio instead (https://www.rstudio.com/).

Here's the hardware I purchased as well:
https://tinyurl.com/yb6gtayq
https://tinyurl.com/y9dt9f8g

Unfortunately, the VHD's I needed were all corrupt.  But, I was at least able to recover some data.

Thanks!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Server Hardware

From novice to tech pro — start learning today.