Solved

HP Proliant ML115 - Raid5 Drive Failure - Raid Failure?

Posted on 2011-02-14
13
2,469 Views
Last Modified: 2012-05-11
Got called out to a customer site.  They have a HP Proliant ML115 Server.  They are using the onboard Nvidia Controller and 3 x 250GB SATA hard drives in a RAID5 configuration.

One of the drives has reported SMART failure (Drive 0.1)

Although only one drive has reported failure as shown in the screen shot below:
Post Drive Listing
When booting up I get the following message while detecting the arrays:
Bootup Raid Error
When I press F8, The screen now shows two arrays of one drive only in a error state as shown (it used to show only the one array with all three drives healthy):
Array Listing
I contacted HP and they sent a replacement drive.  THey could not offer any intelligent advice as to what I can do just to "replace the drive and hope for the best"
Unfortunatley the customer also reports that they have not had a sucessful backup in over two weeks.  Better than nothing I suppose but if possible I would like to retain the data.  THe other two drives I can assume are ok and have the intact parity, I have never dealt with an onboard array controller like this though and am seeking some advice based on experience.

I have hte replacement drive, the question I suppose is, do I replace the drive that has the smart array warning, or the drive that doesnt show up in the array configuration utility at all.

Drive 1.0 doesnt show up when I press F8, but shows fine in the post drive listing.

Advice is appreciated.  
0
Comment
Question by:TheMetalicOne
13 Comments
 
LVL 3

Expert Comment

by:rabindrajha
ID: 34889982
In a nutshell, my suggestion will be replace both drive. the smart error you are getting, that drive is going to be failure soon. take the data backup ASAP. if data is not important, you may take the R&D.
0
 
LVL 5

Author Comment

by:TheMetalicOne
ID: 34890002
it was a 3 drive raid 5 array.  I cannot boot the server to do a data backup.  Do you believe the third drive which is not showing in the array configuration is also faulty?
0
 
LVL 3

Expert Comment

by:rabindrajha
ID: 34890074
what is the status of HDD led? you may take a chance for recovering your data, by booting your server through any live CD, like ubuntu and so and so..which is supporting your raid controller driver. i beleive that ubuntu consist that driver and see if you are able to recover that data. remember RAID can't replace the backup.
0
 
LVL 5

Author Comment

by:TheMetalicOne
ID: 34890128
These are not hotswap drives.

All I can tell you is in post, it the server sees all three drives.
Once the array detection comes up though it only sees two drives.  But, it has separated the two drives into their own little array somehow.  

Booting to ubuntu didnt work.  Since the array wont load, I can only assume that is why it cant see anything.

Fully aware that raid cant replace backup...they have backup.  Just looking to avoid hours and hours of a complete rebuild.
0
 
LVL 3

Expert Comment

by:rabindrajha
ID: 34890229
now this leads many questions.
You may consider upgrading the firmware. as per scenario, not enough sure through it seems drive is failed which is not showing in the array.
0
 
LVL 3

Expert Comment

by:rabindrajha
ID: 34890260
again i gone through your screenshot. however the drive is having SMART failure my be the drive which is not showing in raid array, you may do onething, disconnect the smart failure drive and see, if still there are two drives in array. not to worry about the raid configuration as you know this configuration is on two places, one is in controller and another on disk. you may give a try.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 3

Expert Comment

by:rabindrajha
ID: 34890268
sorry for above comment. you may give a try by reconnecting the drive, whether it still shows two or one only. if not it may the problem of both drive.
0
 
LVL 5

Author Comment

by:TheMetalicOne
ID: 34890466
if i disconnect the smart failure drive, it does not show in the array config utility anymore  

I supect that the third master may be my issue.  it doesnt show in the array config utility anywhere.

The second master is the one showing smart failure, but at least it shows in the array config utility.  If I remove the second master it disappears from the config utility, but then I only have one drive in the array.  

Primary master is index 1
Secondary master is index 2
currenlty the two drives are somehow separated into two separate arrays.

Strange thing is the third master is no where to be seen in the config utility
When you click on the first array
When you click on the second array
0
 
LVL 3

Assisted Solution

by:rabindrajha
rabindrajha earned 150 total points
ID: 34890562
coming to the conclusion, as it's raid 5, so there is no question for erxclusion of the 3rd drive so both of the hdd is faulty. you might have to replace the same.
check this if it can help you about hdd failure
http://datacent.com/hard_drive_sounds.php
0
 
LVL 5

Author Comment

by:TheMetalicOne
ID: 34890646
Thx  Drives are not making any noises, thank you for trying anyways
0
 
LVL 36

Expert Comment

by:ArneLovius
ID: 34891997
I have to admit that I haven't seen what you are describing on any of my ML115 servers.

From what you have described, I think you are saying that originally there was on RAID 5 array of 3 drives, and the Nvidia raid controller i now only seeing two drive and two arrays.

If this is the case you have probably lost all of the data on the array.

I would try swapping out the drive that the array controller doesn't see and see if it will rebuild. I've also found this http://thegreenbutton.com/forums/t/54005.aspx  I'm not quite sure how usefull it will be, but it could explain why you are seeing two arrays.
0
 
LVL 47

Accepted Solution

by:
dlethe earned 350 total points
ID: 34894579
No do not swap out the drive, and hope for the best.  Buy a product from runtime.org, "raid reconstructor".  (Free to try, pay to buy). Hook up the 2 disks to a non-raid controller, and it will rebuild. It is a major pain, but it will likely get 99.99% of it back.  

Root cause for this situation is that you had bad blocks on the surviving disks.  You'll end up with some lost blocks but better than 100% loss.

The disk is throwing a SMART error, in part, because there are unreadable blocks there.

You need to educate the customer about proper maintenance. They need to do regular data consistency checks.  Also, that controller they are using is low-end "junk".  It is not suitable for RAID5. They would be much better off in speed and performance, if instead of getting 500GB replacement disks and going R5, then get 2 x 1TB replacement disks and going RAID1.  Use the other replacement disk in an external USB enclosure for backup, or in a PC for data.
0
 
LVL 5

Author Closing Comment

by:TheMetalicOne
ID: 34899332
Thanks dlethe.  Great tip.  

I ended up doing a complete rebuild of the server, and as it turns out the customer did indeed have complete backups so all is well.  I have the old drive still so I plan to try the raid reconstructor.

-  rabindrajha, thank you for your help and attempts.  The effort was much appreciated.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

More or less everybody in the IT market understands the basics of Networking, however when we start talking about Storage Networks, things get a bit dizzier, and this is where I would like to help.
The article will include the best Data Recovery Tools along with their Features, Capabilities, and their Download Links. Hope you’ll enjoy it and will choose the one as required by you.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now