Solved

Perc 5/E crashed during raid 5 expansion

Posted on 2014-02-21
4
685 Views
Last Modified: 2016-11-23
We have a Dell server with a Perc 5/e controller attached to a MD1000 storage unit.  There was one raid 5 virtual disk array with five 300GB drives in a raid 5 configuration that is not split up.  We had a hot spare that was exactly the same as the others so I decided to un-assign it as the hot spare.  Then I followed Dell's instructions to add this drive to the raid 5 array to expand it.  I used the Dell server administration software to do all this.

The reconstruction phase seemed to be going smoothly and got to about 66% and the server locked up and crashed.  With the MD1000 unit attached with the SAS, the server would not boot (the OS boot drives are on the internal raid controller of the server and not on the MD1000).  We disconnected the SAS cable from the MD1000 just to make sure the server OS would still boot and it does fine.  

We connected the SAS back to the MD1000 and using the built-in raid bios management it shows that this new drive is in a foreign state and it fails when we try to import it.  So I decided to pull the new drive we added to the VD just to see if the array and server would come up and of course in our haste we pulled the wrong drive.  We didn't swap any drives around as I know that is a no-no with raid 5.  So we put the accidentally pulled drive back in and the raid 5 started rebuilding and still in the process now.

Since it never finished the expansion and the new drive is stuck in a foreign state what happens to the array?  Is it still expecting to see the new drive or will we be able to clear the configuration of the new drive and bring up the raid 5 array?
0
Comment
Question by:convergint
  • 2
  • 2
4 Comments
 
LVL 77

Assisted Solution

by:arnold
arnold earned 500 total points
ID: 39878812
Foreign State means the drive is mislabeled, you need to first clear its "Association Label"
Presumably, the drive still has the "hot spare" label.

The difficulty I see is whether the current rebuild will restore access to the RAID 5 pre-expansion. Using the ServerAdministrator, you should be able to look at the physical disks and use the option "clear foreign config" this will remove the label from the disk.
Hopefully, you have a good full backup of the data prior to your expansion attempt.


Not sure whether you needed to remove the drive from HotSpare status prior to the expansion.
NOTE: RAID 5 especially with 5x300GB going to 6x300 will take a while to complete the expansion process.

Once you have access, it might be better to pull the formerly hot spare drive, now foreign state labeled, and use a different system to clear the foreign state label on the drive.  

Make sure the firmware on the perc and md1000 are current.  The version you have might be a bit outdated such that the scenario you have was addressed by an update.
updates to firmware usually should be done when the RAID is in optimal state.
0
 
LVL 10

Author Comment

by:convergint
ID: 39879868
All our firmware us up to date.  We tried all means to recover the file tables with no luck.  It is now with a data recovery company.

I did the same thing expanding the array with two drives two years ago without any issues so that's why I assumed it wouldn't be an issue this time.  Just never took into consideration that the server would crash during the process.

The system basically sees the raid 5 array as being perfect (without the new drive) and is even mounted with the right drive letter but the file tables seems destroyed.
0
 
LVL 77

Accepted Solution

by:
arnold earned 500 total points
ID: 39879943
I suspect the issue here was with the drive being labeled and then you attempted to expand onto it.
There are different possibilities including the state of the raid ahead I.e. Checkpoint might have been needed.
Not sure which/what caused the crash.  The possibility that the drive you pulled (presumably was labeled as failed) was kicked out leading to a "two device" failure.

Expansion of a raid 5 with newer drives are one thing, an expansion with the presence of olderdrives(>3 years) have a higher likelihood of a failure.
0
 
LVL 10

Author Comment

by:convergint
ID: 39894125
Data recovery guys couldn't rebuild the raid as it was in a state of flux.  It had some data on spread onto the new drive but the file tables were half corrupted.  

Before I expanded to the spare drive, I did remove it from the hot drive status and the system saw the drive as being unallocated.

There's no chance of rebuilding the old array so I'm closing this.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Need Command Line Sent to File 7 64
NAS with google authentication 6 63
Windows 2008 Server C Drive too full 25 37
Problem to router 7 14
Usually shares are where we want them for our users and we tend to take them for granted. There are times, however, when those shares may disappear causing difficulty for your users. One of the first things to try is searching for files that shou…
Learn about cloud computing and its benefits for small business owners.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now