A client running a Dell R620 with three disks in a RAID 5 reported amber blinking light on one of the disks though the OS (ESX) was running fine. I suggested buying a replacement which was delivered complete with hot-swap tray. The client swapped out the drive, but the Amber light continued. I thought it might be tagged as Foreign, so I scheduled a visit.  
When I rebooted the server, I went into the RAID controller to clear the (New) Foreign disk config and add it as hot spare so that it would begin rebuilding the RAID.

However, when I got in there the first disk (0) shows as MISSING, the New Disk (1) as Failed, and the third (2) as Online. I shut down and re-seated all of the drives, but no change.

The RAID is in a FAILED State, and with RAID 5 (which this is) you can’t rebuild if two disk fail or are unreachable.

I tried putting the old disk back into Slot 1, but now when I reboot both Disk 0 and 1 show as MISSING.

I also receive "There are offline or missing virtual drives with preserved cache. Please chack the cables and ensure that all drives are present." message.

I read some articles about re-tagging the Virtual Disk. Does this sound like a viable solution?

Any help would be appreciated.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Robert RComputer Service TechnicianCommented:
It seems like two drives have died and before you could recovery the raid by replacing the drive the second drive has failed. It looks like you are out of luck in recovering from this point. When two drives fail in a raid 5 system the data is gone. The only way to recover is from a backup.
Either that, or your client pulled out the wrong disk and caused it to go offline.

If this is somehow the Boot disk set, you can just reinstall ESX and load the VM images.  If this is the Data unit with the VMs, you'll have to create a new RAID and restore from backup.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
BMaenpaaAuthor Commented:
Many thanks for the quick response. I think the client DID swap the wrong drive. I just spoke with him and fortunately this was part of three physical servers in a HA group, so the VM's are intact. I will probably just rebuild and re-join.
Thanks again...
Rowby Goren Makes an Impact on Screen and Online

Learn about longtime user Rowby Goren and his great contributions to the site. We explore his method for posing questions that are likely to yield a solution, and take a look at how his career transformed from a Hollywood writer to a website entrepreneur.

How big are the boot disks?  They should just be in a 2 disk mirror instead of a RAID 5.  RAID 5 doesn't make sense and will cause confusion.  ESX does not need a lot of disk space.  They now sell servers with 8GB and/or 16GB SD cards as the boot drives for ESX.

Since it's in HA with 2 other servers, you don't really even need RAID.
BMaenpaaAuthor Commented:
Thanks Serialband,
I agree, this is not how I would have built it, and didn't. I like the SD CARD RAID 1 modules for my ESX OS too!
My client inherited this and is looking to pass it off to me. This particular server was demoted to internal Admin stuff and all production VMs moved to a new platform a while back.
We might not even bother rebuilding it. I just don't like stuff blowing up in my hands.
Thanks for your time!
Next time you should use PERCCLI to check if there are foreign disks rather than reboot. It's never good to reboot or power off when it is degraded.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.