Proliant DL380G3 won't boot, RAID or drive issue issue

I do not believe any data consistency checks are performed on this server. The SQL data is backed up daily but there are a lot of other files on the server that are likely backed up somewhere.  I'm not the normal caretaker for this machine, it has fallen to me because the caretakers are out of town for a few weeks.  Naturally the server has run fine for four+ years until yesterday.

The machine indicates it is RAID 5 when viewing the logical drive configuration area.

Drive two does spin up and is green like the others and the logical drive config area shows it being okay.  I'd entered the F2 option ( which warned there could be data loss ) where the machine was originally stuck after power cycling several times. The machine seemed to drive two back to life.  The RAID config area now shows all drives being okay.

I've learned that drive two had turned orange early in the day that it died. It was several hours after this was first noticed that the machine stopped answering its SQL duties and I was contacted to come over and see what I can do with it.

A background parity check is automatically run whenever the server is idle for more than 15 seconds, but for once that hasn't helped. Looks like it's restore time.

Thank you all so much for the insight.  Yes, this is a business server, for a public safety organization actually.  I've either got to make it run again or rebuild it, need to do one or the other today.  The data records are accumulating on portable devices in the field.  I do have solid backup of the SQL data.  I've also learned this morning that this server also functions as a domain controller for a small set of personnel in one of the offices.  There are some miscellaneous files, word documents and such that probably are not backed up. The folks in the office used this server as a bit of a backup and file share for their desktop PCs. The SQL data is the main thing and I've restored a backup to a different server to ensure that the backup was good, it is.

So I gather that the thinking is I should reconfigure the server from scratch..?  I'll put all new drives in it.  I have no idea where the HP CDs are. This server was handed down from another department and I am thinking there is no hope for finding the CDs.  What do I need driver-wise and is all of this downloadable from the HP site?

Thank you,
Bill

Should I use RAID 5 or RAID 0 in a rebuild?

Smartstart CD is on HP's web - http://www.hp.com/#Support enter machine details, then OS and download firmware CD (as you may as well get it up to date now) and then smartstart CD. Smartstart will put the right drivers on for you so don't have to mess about downloading individual ones.

RAID 0 is asking for trouble, there's no parity. Use RAID 10 if you have enough capacity since it's faster than RAID 5.

Good heavens, there are seemingly hundreds of downloadable drivers an firmware for this machine at HPs site.  I'm downloading the HP SmartStart CD x32 version 8.25 from May of this year.  Will this CD image provide all that I need or do I need to grab other things as well?

Looks like RAID 5 is what I need to do based on the storage space requirements.

Everyone agree that using new drives is the way to go?  I'm thinking 4 like are currently in the server and adding one for a hot spare. What should I do with the sixth slot, add a second hot spare?

Thank you
Bill

Technically, brand spankin' new drives are higher risk of failure then ones that have been in service for a few months to several years .. due to high infant mortality.  Don't be so concerned with new vs old.  I'll take 4-year-old enterprise SCSI class disks over 1TB consumer-class SATA disks any day of the week.

Read the specs on the disks,  make sure 24x7 duty cycle & look at the ECC correction.   Go with RAID5 if you must, but I would partition it into 2 logical disks so when time allows you could add a pair of disks, mirror (RAID1) them, and migrate the O/S + scratch table space + log files to the RAID1.

On a related note on HP quality, am doing a cross compile on an ancient HP VMS (alpha) based system for some disk diagnostics.  Look how many cumulative reads/writes they have over the years, and the UNRECOVERED read/write totals are all zero.  That is almost unheard of.
===========================
The COMPAQ SCSI disks were manufactured in 2000!  - they don't make them like this any more
Discovered COMPAQ BD0096349A S/N "3BV120SH" on LAB153$DKC0: (Not  Enabling SMART)(8678 MB)
 Inquiry Text Page Data - ANSI defined fields
   Device Type:                         disk
   Removable Device:                    NO
   Vendor Identification:               COMPAQ
   Product Identification:              BD0096349A
   Firmware Revision:                   3B12
   SAF-TE Enclosure services available: NO
   Total Capacity (In Bytes):           9100044288
   Total grown defects:                 0
   Total Primary (factory) defects:     308
   Board serial number:                 H0000V1230T45
   Servo RAM Release number:            83470910
   Servo ROM Release number:            00000001
   Servo RAM Release date:              512A
   Servo ROM Release date:              2000
   ETF Log date MMDDYYYY:               12/16/2000
  Write errors corrected with possible delays: 0
   Total Write errors: 0
   Write errors corrected: 0
   Times correction algorithm processed (on Writes): 0
   Bytes processed (on Writes): 1857795739648
   Unrecovered errors (on Writes): 0
   Read errors corrected without substantial delay: 6304116
   Read errors corrected with possible delays: 0
   Total Read errors: 0
   Read errors corrected: 6304116
   Times correction algorithm processed (on Reads): 6304116
   Bytes processed (on Reads): 403237462176768
   Unrecovered errors (on Reads): 0
   Verify errors corrected without substantial delay: 6
   Verify errors corrected with possible delays: 0
   Total Verify errors: 0
   Verify errors corrected: 6
   Times correction algorithm processed (on Verifys): 6
   Bytes processed (on Verifys): 9100044288
   Unrecovered errors (on Verifys): 0

Well, I've been running some of the diagnostic tests with SmartStart.  For the drive array is is telling me that drive 1 has failed, yet drive two originally had the amber fail light on.  It tells me I should replace drive one.  I suppose I should replace drive 1 on the chance that it brings the machine back to life..?  Can I swap the drives while in SmartStart?  Curious if it is safe to pull on out and insert another in its place with the server on and SmartDrive running or should I power off and allow the server to try and boot normally?

The test show all of these drive have been running for 61 months.

Thank you for all of your assistance.

Bill

You can swap while it's running SmartStart, the controller doesn't know you're running from CD, it'll rebuild just like if it was running an OS from disks.

I've swapped the disk, let it rebuild and I'm right back to were I was.  Server get to the point were it should boot into Windows and it just sits.   The diagnostics from SmartStart now show all four drives passing the test but all four have a yellow textbox indicating that some information wasn't written to the logs.

So I am guessing this means it is time to do the complete rebuild..?