I have two very old servers which have been in production for about 7 years. They are both HP ProLiant DL380 G5 with dual CPU, dual PSU, 64GB RAM and 8x300GB 10K SAS drives. They are identical in every way. Because they're old, I keep a spare PSU and several spare drives on hand, but I've never had an issue.
One was being used for some basic hosting stuff for some of our internal services - FTP for onsite backup replication to our datacenter, secondary DNS server for a few zones; like that.
The other was hosting about 7 virtual machines. One was an old mail server that was servicing one domain for a friend and it also ran some secondary DNS zones. All of the others were for customers, but were secondary servers for other services. One VM had important data that (I found out) the customer never backed up.
Yesterday morning, the VM host server went offline due to a failed array. The array on that server is a RAID5 using 7 of the 8 disks. The 8th disk was a hot spare.
Booting into the CLI Array Configuration Utility during POST, I found that disks 1 and 2 weren't listed at all. Disks 3, 4, 5 & 7 were good. Disk 6 was bad. Disk 8 was good and was still the hot spare.
I swapped Disk 6 with a new drive and rebooted and received a "drive mismatch" error in the ACU.
Assuming the RAID controller had died a horrible death, I migrated the services off of the other DL380, removed the drives and placed the drives from the VM server into the chassis in the same order.
Booted up with exact same result.
I downloaded the offline ACU ISO from HP and created a bootable USB thumb drive. Booted up with only Drive 1 inserted. The ACU showed Drive 1 was good and the others were missing. I added Drive 2 and rescanned the system and then refreshed. Drive 2 came up as good. Same thing with 3, 4, 5,7, and 8 and all came up as good. Inserted Drive 6 and it came up as bad. I rebooted with all 8 drives inserted and 1-5 were good, 6 was bad, 7-8 were good.
This behavior was exactly the same when I put the drives back into the original chassis.
The end result is 6 of the 7 drives in the array are recognized and are "Good". The hot spare is also recognized and is "Good".
My problem is - even though they're all detected, are "Good" and are known members of the array, I still can't boot and I still don't have an option to rebuild the array.
Any ideas would be greatly appreciated.