HP Proliant ML115 - Raid5 Drive Failure - Raid Failure?

Got called out to a customer site.  They have a HP Proliant ML115 Server.  They are using the onboard Nvidia Controller and 3 x 250GB SATA hard drives in a RAID5 configuration.

One of the drives has reported SMART failure (Drive 0.1)

Although only one drive has reported failure as shown in the screen shot below:
Post Drive Listing
When booting up I get the following message while detecting the arrays:
Bootup Raid Error
When I press F8, The screen now shows two arrays of one drive only in a error state as shown (it used to show only the one array with all three drives healthy):
Array Listing
I contacted HP and they sent a replacement drive.  THey could not offer any intelligent advice as to what I can do just to "replace the drive and hope for the best"
Unfortunatley the customer also reports that they have not had a sucessful backup in over two weeks.  Better than nothing I suppose but if possible I would like to retain the data.  THe other two drives I can assume are ok and have the intact parity, I have never dealt with an onboard array controller like this though and am seeking some advice based on experience.

I have hte replacement drive, the question I suppose is, do I replace the drive that has the smart array warning, or the drive that doesnt show up in the array configuration utility at all.

Drive 1.0 doesnt show up when I press F8, but shows fine in the post drive listing.

Advice is appreciated.  
Who is Participating?

Improve company productivity with a Business Account.Sign Up

DavidConnect With a Mentor PresidentCommented:
No do not swap out the drive, and hope for the best.  Buy a product from runtime.org, "raid reconstructor".  (Free to try, pay to buy). Hook up the 2 disks to a non-raid controller, and it will rebuild. It is a major pain, but it will likely get 99.99% of it back.  

Root cause for this situation is that you had bad blocks on the surviving disks.  You'll end up with some lost blocks but better than 100% loss.

The disk is throwing a SMART error, in part, because there are unreadable blocks there.

You need to educate the customer about proper maintenance. They need to do regular data consistency checks.  Also, that controller they are using is low-end "junk".  It is not suitable for RAID5. They would be much better off in speed and performance, if instead of getting 500GB replacement disks and going R5, then get 2 x 1TB replacement disks and going RAID1.  Use the other replacement disk in an external USB enclosure for backup, or in a PC for data.
In a nutshell, my suggestion will be replace both drive. the smart error you are getting, that drive is going to be failure soon. take the data backup ASAP. if data is not important, you may take the R&D.
TheMetalicOneAuthor Commented:
it was a 3 drive raid 5 array.  I cannot boot the server to do a data backup.  Do you believe the third drive which is not showing in the array configuration is also faulty?
Easily Design & Build Your Next Website

Squarespace’s all-in-one platform gives you everything you need to express yourself creatively online, whether it is with a domain, website, or online store. Get started with your free trial today, and when ready, take 10% off your first purchase with offer code 'EXPERTS'.

what is the status of HDD led? you may take a chance for recovering your data, by booting your server through any live CD, like ubuntu and so and so..which is supporting your raid controller driver. i beleive that ubuntu consist that driver and see if you are able to recover that data. remember RAID can't replace the backup.
TheMetalicOneAuthor Commented:
These are not hotswap drives.

All I can tell you is in post, it the server sees all three drives.
Once the array detection comes up though it only sees two drives.  But, it has separated the two drives into their own little array somehow.  

Booting to ubuntu didnt work.  Since the array wont load, I can only assume that is why it cant see anything.

Fully aware that raid cant replace backup...they have backup.  Just looking to avoid hours and hours of a complete rebuild.
now this leads many questions.
You may consider upgrading the firmware. as per scenario, not enough sure through it seems drive is failed which is not showing in the array.
again i gone through your screenshot. however the drive is having SMART failure my be the drive which is not showing in raid array, you may do onething, disconnect the smart failure drive and see, if still there are two drives in array. not to worry about the raid configuration as you know this configuration is on two places, one is in controller and another on disk. you may give a try.
sorry for above comment. you may give a try by reconnecting the drive, whether it still shows two or one only. if not it may the problem of both drive.
TheMetalicOneAuthor Commented:
if i disconnect the smart failure drive, it does not show in the array config utility anymore  

I supect that the third master may be my issue.  it doesnt show in the array config utility anywhere.

The second master is the one showing smart failure, but at least it shows in the array config utility.  If I remove the second master it disappears from the config utility, but then I only have one drive in the array.  

Primary master is index 1
Secondary master is index 2
currenlty the two drives are somehow separated into two separate arrays.

Strange thing is the third master is no where to be seen in the config utility
When you click on the first array
When you click on the second array
rabindrajhaConnect With a Mentor Commented:
coming to the conclusion, as it's raid 5, so there is no question for erxclusion of the 3rd drive so both of the hdd is faulty. you might have to replace the same.
check this if it can help you about hdd failure
TheMetalicOneAuthor Commented:
Thx  Drives are not making any noises, thank you for trying anyways
I have to admit that I haven't seen what you are describing on any of my ML115 servers.

From what you have described, I think you are saying that originally there was on RAID 5 array of 3 drives, and the Nvidia raid controller i now only seeing two drive and two arrays.

If this is the case you have probably lost all of the data on the array.

I would try swapping out the drive that the array controller doesn't see and see if it will rebuild. I've also found this http://thegreenbutton.com/forums/t/54005.aspx  I'm not quite sure how usefull it will be, but it could explain why you are seeing two arrays.
TheMetalicOneAuthor Commented:
Thanks dlethe.  Great tip.  

I ended up doing a complete rebuild of the server, and as it turns out the customer did indeed have complete backups so all is well.  I have the old drive still so I plan to try the raid reconstructor.

-  rabindrajha, thank you for your help and attempts.  The effort was much appreciated.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.