Solved

Data Corruption after RAID Rebuild on HP ML150 G5

Posted on 2010-11-19
22
1,758 Views
Last Modified: 2016-10-27
Hey, I have "upgraded" hard drive on one of my client server wich were in RAID1. This is how I done it (Done it this way many time and never had problems):

He had 2x160gigs and I upgrade them to 2x500 RE3 WDC
1- Shutdown server and plug 2x 500 gigs inside
2- Start server and create the array using the utility (pressing F8 in this case)
3- Shut down server, unplug a 160gigs and 500gigs, plug them in another PC
4- Boot the other PC with Acronis True Image Echo and start the cloning of the 160 to the 500
5- Unplug the "empty" 500gigs of the server and plug the freshly cloned 500gigs in it
6- Start the server with only the cloned 500gigs, check to see if everything works = Yeah, everythings is OK
7- Shutdown server, plug the other 500gigs in it
8- Start server and in Windows Server, start a rebuild using the HP Storage Manager

So from now everything was fine, they had access to everything and nothing was lost or whatever. The reason I did the cloning on another PC is because the Estimate Time of the copy on the server was more than 14 hours and on the other PC it only took 30 minutes (I think it's Acronis that wasn't able to properly communicate with the HP raid controller).

SO for the problem;
After 2 days, the rebuild was completed BUT the server cold restarted by itself and after that EVERYTHING was corrupt on the OS partition. So, I checked the log I could see and they had Power Outage all weekend long, so that made me to believe this screwed the rebuild or data (They are on a APC Smart-UPS 1000).

So the monday morning I go there and reclone the hard drive, restore there data from backup, and everything is fine once again. BUT once again, the RAID finished Rebuild and BAM Data corruption again! So that lead me to a defective hard drive... Run test on both of them and they return OK.

So Now, I'm right there at the moment with 2 new RE3 500gigs and trying to do the same thing, but now I'm really afraid of what would happen... What can it be? The server runned perfectly for over 2 years juste before I swap the drive.

Any idea on this ?

Thank you!
0
Comment
Question by:GIP
  • 14
  • 6
  • 2
22 Comments
 
LVL 12

Expert Comment

by:mccracky
ID: 34174209
I'd start to suspect the Acronis cloning writing the 500G so that there is a mismatch in the sizes (something there saying it was a 160G drive).

Does your RAID controller allow online expansion?  If so, I'd think about pulling one of the 160G drives and plugging in one of the 500G drives and doing the rebuild.  Once that is finished pull the other 160G and replace with the 500G and do the rebuild again.  Then, when you have both 500G drives in, expand it out to the full 500G RAID.
0
 

Author Comment

by:GIP
ID: 34174297
Thanks for the fast answer!

THey had 2x 160gigs that were full, so I swapped them with 2x 500gigs and adjusted partition in Acronis. I don't think Acronis is the culprit here because the system Boot perfectly after a cloning and on 1 drive.

They RAID controller is really basic, it'S actually the onboard Raid controller and don't have much fonctionnality on it. That's why I do this this way and it always worked great!

OK here an UPDATE:
So while writting this question, I cloned the 160gigs on 2 new 500gigs (one after the other) then plugued them in the server. Booted the server with both them in it (RAID1 array was created prior to cloning) and it booted in windows server BUT it hang when I tried to access data(2-click My Computer), So I rebooted the server and it hang on the Windows Server 2003 screen. So I unplugued 1 drive and it booted inside the OS but asking for a chkdsk... SO that lead me to the RAID Controller failling to work with those 2x 500gigs...

Because when it only runs on 1 of the 500gigs (Before I add the other) it works perrfectly, the user were able to use the server while it was rebuilding for about 2 days but as soon as the 2nd drive finished rebuilding it failed! And today the same...

I'm going to update the firmware if I can find it since it'S hp....
0
 

Author Comment

by:GIP
ID: 34174341
And FIY, today I work with 2 new brand new RE3 from WDC
0
 
LVL 12

Expert Comment

by:mccracky
ID: 34175223
Have you tried it with the second 500G drive uninitialized before putting it in?  

(Clone the 160G to one of the 500G drives, put in only the one 500G drive that you say works, put in the 2nd 500G uninitialized (with the MBR and partition data completely zeroed out/randomized), and then adding it to the degraded RAID array).
0
 
LVL 12

Expert Comment

by:mccracky
ID: 34175276
I don't know which model you have of the ML150 G5, but the specs (http://h18000.www1.hp.com/products/quickspecs/12901_na/12901_na.HTML) seem to indicate that the 500G drives shouldn't be a problem.
0
 

Author Comment

by:GIP
ID: 34175337
Yeah that what I did first.

They were both brand new so nothing on them
Created the RAID
Cloned the 1st then plugued only him in the server = Boot windows correctly with RAID Dregaded status (wich is normal)
Then plug the second one inside and it boot windows correctly, It's only ONCE the rebuild is done that the corruption arise... If I only have 1 disk in it it works!

Today I put 2 new one inside but this time I cloned both of them one at a time then put both inside the server and it start OK with raid said as Optimal, but then it hang and start to corrupt I guess because it didn'T want to reboot with both of them inside and when I unpluggued one drive it booted and telling me to do chkdsk...

Now i'm trying to update the firmware but it doesn'T recognize the controller and the other file I have doesn'T fit on a floppy and it told me to put them on a floppy...

Thanks for your time man, I really need to have ti working !
0
 

Author Comment

by:GIP
ID: 34175353
The documentatino you found is exactly the model I have. But I use the embedded e200i controller not the "Performance" one that you can add
0
 
LVL 12

Expert Comment

by:mccracky
ID: 34175490
What I meant was using the second drive without first putting it in to create the RAID.  Put it in just as if you bought it off the shelf.
0
 

Author Comment

by:GIP
ID: 34175516
OK but I must initialize the RAID before with 2 disk, no?

And I just saw that the server isn'T the e200i controller but the HP Embedded SATA Raid controller by Adaptec...
0
 

Author Comment

by:GIP
ID: 34175641
IT says somewhere that if you use sata drive you need 1.5g sata drive... could this be the problem?
0
 
LVL 12

Expert Comment

by:mccracky
ID: 34176065
That's for a different controller (SC44Ge PCI-Express Controller).  And I think it's a typographical mistake (I don't think they sell 1.5G drives.)

What I was wondering is if you can get the controller to recognize the one 500G drive as part of a degraded raid set and then put the other one is as a replacement "new" drive that the board should initialize and put into the array as a replacement.  I'm not familiar with that controller, though.  I'm just guessing.
0
Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 

Author Comment

by:GIP
ID: 34176671
It should work like you say but the problem is whenever the rebuild complte the corruption apear! So doing so won't change anything , no?

Anyway since it's work fine on 1 drive, I'll setup is server on 1 drive and recopy is data on it. I ordered the exact HP Part # for the 500gb that goes into this server and will receive them monday, and monday night I'm gonna swap the drive again!

Thanks for your help, will let you know if it works!
0
 

Author Comment

by:GIP
ID: 34176910
BTW I will do what you said about adding the 2nd drive a a "new" one when I'll do this monday. Will initiate with the HP disk and a WDC then when I'll put back the 2nd HP it will be like a new drive to the array.
0
 

Author Comment

by:GIP
ID: 34177477
There is one thing I forgot to mention about this setup. The first time I added the 500gigs, when the 160s were still inside and plugued, I saw that one of the 160 was at 1.5gbs and the other at 3gbs... Why that? Maybe it as something to do with my problem?
0
 
LVL 55

Expert Comment

by:andyalder
ID: 34186817
Regarding firmware update I don't think it's possible since the embedded controller is just the one in the Intel chipset.
0
 

Author Comment

by:GIP
ID: 34190169
Yeah, at first I searched for the e200i firmware update but when trying to update it didn't find the controller. So at this time I saw it was only the HP Embedded Sata RAID controller and there is no update on their website.

I did alot of research about this controller this weekend and all I can say is it's really a cheap one that doesn't work well in Rebuilding raid while in OS. So what I'll do is create a new array raid1 with new 500gigs HDDs, do a "Clear" instead of "Quick Init" then clone the working one on them while they are in RAID1. There is only 1 problem with this is Acronis takes 14hours to clone on this raid controller... I bought myself the new Acronis 2011 and I'll see if this one is faster.

Thanks for the answer!
0
 
LVL 55

Expert Comment

by:andyalder
ID: 34190560
Have you tried cloning it with the onboard RAID disabled? I think that "controller" still boots the disks OK when RAID is disabled (but I'll admit I don't touch the 100 series).
0
 

Author Comment

by:GIP
ID: 34191954
No didn't try but if I do it like this the raid will have to be rebuilded and that's what I want to avoid!

So theoriccaly, if I initiate the 2x new 500gigs and to a "Clear" instead of "Rebuild" or "Quick Init", the raid will show as "Optimal" and if I clone the hard drive directly on the RAID1 the data will be written on both drive at same time thus avoiding to rebuild the raid. Am I right?
0
 

Accepted Solution

by:
GIP earned 0 total points
ID: 34205207
I Think it worked!

Cloned with Acronis 2011 from one drive not connected to the raid controller to my new RAID1 (2 brand new HP tagged Seagate drive) that I have "Quick Inited". It tooks 50 minutes to clone about 180gigs and everything booted well on the new RAID1 Array after that and the RAID was still at "Optimal" state!

Will wait some days before closing this question but anyway thanks alot for your help :) I think the solution proposed by McCracky would have worked but didn't want to do a rebuild anymore.
0
 
LVL 12

Expert Comment

by:mccracky
ID: 34205675
Glad to hear things are working now.  
0
 

Author Comment

by:GIP
ID: 34309815
Well, after about 2 weeks, everything runs smooth and there is no more sign of corruption. So the job is done and everyone is happy :)

Thanks alot for your help!
0
 

Author Closing Comment

by:GIP
ID: 34341531
Can't give points to others because I did it my way but they were really helpful and fast!
0

Featured Post

Get up to 2TB FREE CLOUD per backup license!

An exclusive Black Friday offer just for Expert Exchange audience! Buy any of our top-rated backup solutions & get up to 2TB free cloud per system! Perform local & cloud backup in the same step, and restore instantly—anytime, anywhere. Grab this deal now before it disappears!

Join & Write a Comment

Hyper-convergence systems have taken the IT world by storm and have quickly started to change our point of view of how the data center should and could be architected. In this article, I’ll explain the benefits of employing a hyper-converged system …
this article is a guided solution for most of the common server issues in server hardware tasks we are facing in our routine job works. the topics in the following article covered are, 1) dell hardware raidlevel (Perc) 2) adding HDD 3) how t…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now