PowerEdge T110 - Replaced failed drive in RAID10. How to get RAID 10 configuration back to normal operation?

It is using a PERC S100 controller.

There is one Virtual Disk. Configured as RAID 10. State is degraded.

Before replacing the drive:

There were 3 hard drives visible. Split between Span 0 and Span 1.

In Span 0, the drives visible were 0:0 and 0:1.

In Span 1: the drive visible was 0:3.

This meant drive 0:2 had failed and so I replaced it. However, after replacing it; the new drive was placed into it's own RAID 0 by the controller.

I removed that RAID 0 and tried adding the new drive to the RAID10 but the only option I saw available was 'Assign Dedicated Hot Spare'. I selected this and the Virtual Disk started 'rebuilding'.

But now when I go into the Virtual Disk details I see:

In Span 0, the drives visible are 0:0 ,0:1 and 0:2.

0:0 is in a 'Ready' State. 0:1 and 0:2 are 'Online'.

In Span 1: the drive visible is still only 0:3.

The way I interpret is that now it is rebuilding Span 0 with drives 0:1 and 0:2 but not including 0:0...is that correct?

How would I add a drive back into Span1?
SeeDkAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

arnoldCommented:
A hot spare will be auto used.

Which controller is in use?
Sound as though the drives are not hot swap, do you have Dell openmanage server node Instalked through which you can manage the hardware.

The rebuild will take time to complete at which point the volume should be back in optimal mode.

Based your you have a mirror of stripes, this takes longer to rebuild compared to stripe of mirrors. Where a one to one rebuild takes place.

Using the Dell GUI tool, you shoukd see the status of the rebuild.
Gerald ConnollyCommented:
Why oh why did Dell have to change the names of everything!! Why did a stripe become a Span, Spanning means something different to a Stripe!

If you really do have ONE mirror of multiple stripes that is NOT RAID10 (well maybe to Dell). the accepted definition (by the rest of the storage industry) is that RAID10 is a single stripe of multiple mirrors! The alternative (ie mirror of stripes) is normally seen as being prone to data-loss!
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
S100 is software/chipset based RAID? Use Open Manage to manage the RAID setup.

Using the service tag on Dell's support site please confirm that the drive cage is hot swap.

I suggest rebooting the server into the RAID "BIOS" to manage the drives if all else fails.
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

arnoldCommented:
Gerald,
What is a stripe In that conext?
Not sure the origin, Solaris used striping to convey the software meta creation.

Dell's span conveys the idea single volume spans multiple drives.
Gerald ConnollyCommented:
The standard def of striping is that data is in chunks that are sequentially striped across the spindles.

Spanning is where multiple spindles are combined into one volume but the disks fill up sequentially, ie disk 1 is used first and when that is consumed disk 2 , etc etc

Dell have re-invented the stripe (as a Span) and confused everybody by defining RAID10 as a single mirror of multiple stripes - which is exactly what we wanted to get away from - with the ambiguity of RAID1+0 and RAID0+1, an ambiguity that Dell fostered against Digital back in the last century! Essentially Digital(DEC) [RAID0+1] and Dell [RAID1+0] were referring to the same Striped Mirrorset RAIDset organisation, and this confused those not in the know. The industry decided to use a standard definition ie RAID10 - a single stripe of multiple mirrorsets, but last time i looked Dell were confusing the issue with the opposite definition, and causing even more confusion by stealing the Span definition as well
arnoldCommented:
The issue might be less Dell, but the asker's statement that he has a raid10
If not mistaken since perc4 raid10 is a stripe of mirrors.
Their technical writers, might, the controller setup ....
SeeDkAuthor Commented:
The rebuild is done and this is the result....

3_RebuildDone.JPG

So as I thought it took out 0:0 from Span0 and put 0:2 in its place instead of placing 0:2 into Span1 where it is needed...not sure why.
And now 0:0 is shown as an available disk...

4_RebuildDone.JPG
What is my next option here? Try assigning 0:0 as a global hot spare?
SeeDkAuthor Commented:
Update: I assigned as global hot spare and yes 0:0 is now rebuilding itself in Span1...not sure why this wouldn't have worked when assigning a disk as dedicated hot spare. From what I understand, that should not make a difference in this case.
Shouldn't Global vs Dedicated only matter if there is more than one Virtual Disk on the server? This server has only one Virtual Disk.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
arnoldCommented:
You in effect have three
Span0
Span1
Resulting mirroring span0 and span1


Have not worked with the one you have, but r other RAID controllers with hot-swap when a failed drive is removed and its replacement inserted the controllers detects the insertion of the new device at which point a rebuild is auto-triggered.
SeeDkAuthor Commented:
Yes I know other RAID controllers should do it automatically but this one doesn't support hot swap.
Per Dell, if the chassis has to be opened to access the drives, it is not hot swappable..this chassis needs to be opened.
I am surprised it even rebuilded Span0 at all since that one was healthy.

I don't understand why selecting 'Assign Dedicated Hot Spare' on the Virtual Disk caused it to rebuild the healthy Span 0 while ignoring the degraded Span1.

Selecting 'Assign Global Hot Spare' from the Physical Disk menu did correctly assign the rebuild of the disk to Span1.

Also from what I understand the way Dell has designed this is that each Span is only a RAID1 holding half of the data...so if that disk 0:3 fails during the rebuild all the data is gone. Is this industry standard or Dell standard?
arnoldCommented:
I do not believe it is rebuilding the Span 0, but indicates that span 0 and span 1 are in a rebuilding state
It can not rebuild span 0
The display is confusing.
The disk display should reflect the drive status

Are you able to look at virtual volumes
Volume 1 status rebuilding 5%
Or this controller, does not do that.

It has to mirror with span0 as the reference to create span1
SeeDkAuthor Commented:
The first screenshots I posted was after the initial rebuild which rebuilt Span0...not the one I am doing now. This is the current status:

5_Rebuild-2.JPG
6_Rebuild-2.JPG
SeeDkAuthor Commented:
And yes it is a bit confusing because it says rebuilding for the Virtual Disk as a whole rather than showing which hard disk exactly is being rebuilt.
But this last screenshot occurred after I set up 0:0 as a Global Hot Spare. It is even shown in the screenshot that it is a Global Hot Spare. So it must be rebuilding Span1 on disk 0:0.
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
RAID 10 is a stripe across two mirrored pairs. We don't use it anymore because we had one drive fail then its mirrored pair died not long after replacing the bad one and the rebuild had begun.

So yes, that is what it is for RAID 10.
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
It's rebuilding the array. The existing mirror partner is being replicated to the replacement drive. 1% eh? How long has it been running?

EDIT: Make sure the backups are good!
SeeDkAuthor Commented:
At the time of  the screenshot - maybe 30 minutes. This controller is very slow to rebuild. It took a day and a half for the Span0 rebuild which I did not want to do.
I imagine the same length of time for this - assuming 0:3 doesn't fail. Bracing myself for a failure!
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
We stopped using software/chipset RAID a long time ago. Too buggy and tended to go full-stop if a drive failed while a hardware RAID controller would keep going plus have a buzzer to indicate drive failure.

Make sure the backups are good. The extra stress on the mirror partner during rebuild could indeed trigger a failure.
SeeDkAuthor Commented:
Well, must be running some good luck. The rebuild completed with no problems. It took long, just north of 48 hours but it finished. Now to replace that degraded disk and it will be in great shape.
arnoldCommented:
What do you mean, with the rebuild complete, the raid shoukd be in optimal state.
If you have a disk that has a predictive failure, you shoukd backup the data. The replacement of the failing drive, will require the same rebuild process.....
SeeDkAuthor Commented:
Yes, the RAID is listed as optimal and yes i mentioned in my last post i will also replace the other hard disk which is degraded/predictive failure.
arnoldCommented:
the age of the remaining drives are similar, make sure that you have setup a good backup process.
While at the same time replacing the failing drive.
SeeDkAuthor Commented:
Found correct menu option on this controller
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Dell

From novice to tech pro — start learning today.