HP Proliant DL380e Gen 8 RAID configuration

Hello,
I have an HP Proliant DL380e Gen that I inherited.  SInce I didn't set it up, I'm not quite sure if I'm understanding the configuration.   I am hoping somebody can clarify some things for me.  Currently it has 8 500GB drives in it.  The array controller is showing that bays 5,6,7, and 8 are one logical drive, RAID 5.  So does this mean that 1,2,3, and 4 are not raided or how exactly would you describe the redundancy aspect of this machine?  It's not showing any unassigned drives so why aren't drives 1,2,3, or 4 listed as anything, neither unassigned nor part of any array.  It appears that there is around 3TB of storage all together but I can't picture how this works.  My understanding is that RAID 5 will give me 1/3 of the space available, so out of the 8 drives physically showing 4TB, we'd see 3TB after RAID.  I have Vmware running on it and the vmware logs are showing that drives 1,2,3, and 4 are exceeding the thermal threshold that is defined.  I guess my other question would be, how can I replace drives 1, 2, 3, and 4 if they are, in fact, overheating or if they ever fail?  It doesn't appear (to me) that I can just pop a new drive in and they will just rebuild.  Please help!  I have the VMs intermittently going down and the whole server becomes unresponsive and I can't even reboot the vmware box remotely via CLI.  Have to have the guy walk upstairs and power it off manually.  It's a nightmare!
JasonAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Zephyr ICTCloud ArchitectCommented:
Strange config apparently, maybe it's configured as two RAID5, since that would amount to the 3TB you're seeing? How many (local) datastores can you see in VMware?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Have you checked using the Array Configuration Utility ? (which means booting the Server, from the Smart Start DVD), or checking in the Array Controller POST, Press F8 at BOOT.

If the array controller is only showing 5,6,7 and 8 disks in the Array, is the size of the datastore currently, approx 1.5TB ?

8 Disks in RAID 5, would give 3TB.
0
JasonAuthor Commented:
Thanks for the quick response!  There are two datastores in vmware.  1.5TB each.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

JasonAuthor Commented:
..and yes, I've looked at the array config util.  Did you mean the F5 at boot?
0
JasonAuthor Commented:
So yea, it's showing only 1 logical drive in the array config using 5,6,7, and 8 but I see two 1.5GB datastores.
0
Zephyr ICTCloud ArchitectCommented:
Than that would mean there's two logical drives of 1.5TB each ... So in theory you should be able to change the hard disks out (hot or cold swap).
0
JasonAuthor Commented:
Also, I was checking hard drive serial numbers to verify which ones were showing the heat issue, and removing drive 1 crashes the server.  When powered off, if I replace the drive 1 with a brand new drive, the server will not boot.  Can't find the OS.
0
JasonAuthor Commented:
@spravtek:  Why doesn't it show 2 logical drives in the array config?  It specifically says 1 logical drive and lists 5, 6, 7, and 8 underneath it.  No mention of 1,2,3, or 4 anywhere.
0
Zephyr ICTCloud ArchitectCommented:
That's a good question, the only reason I can think of there's something wrong, either hardware (since you're also seeing overheating issues) or firmware ... So first thing to check is if you have the latest firmware for the controller and maybe for the disks as well.
0
arnoldCommented:
Not sure whether the issue is that there are two separate controllers one serving a single channel. 1-4  and the second 5-8
The internal/boot controller possibly has 5-8.

Does the util reflecting 5-8 logical volume reflect the 1-4 disks in the listing?

Look at the lspci to see what controllers there are.
0
Zephyr ICTCloud ArchitectCommented:
Yes, check in VMware if the datastores are using the same controller, would be strange if they weren't though.
0
JasonAuthor Commented:
There shows only one contoller and 1 logical drive consisting of disks 5, 6, 7, and 8.  Do u think somehow the other array was broken yet somehow maintaing a 3TB capacity?  That makes zero sense to me but I'm just throwing anything out there at this point.
0
Zephyr ICTCloud ArchitectCommented:
Well, something definitely is happening that doesn't make sense, first the temperature issue, second, you only see 1 logical drive. There's not much options to test these things... So hence I'd first try the firmware route. If that doesn't help ... Backup everything and then do some tests with the logical drive(s)?
0
JasonAuthor Commented:
Yeah, I just started this job so I'm trying to figure everything out and how it was setup.  The guy that's been working on the intermittent shutdowns recently was even working with VMware and they ran some kind of diags and were seeing I/O errors on the disks within VMware.  I was thinking either the heat and or the I/O errors were cause and effect of one another.  The guy that was in charge of this server before me, wasn't doing his job correctly so I replaced him, but supposedly he ran all the HP hardware diagnostics and saw no problems.  This has been going on for a month or more.
0
Zephyr ICTCloud ArchitectCommented:
Hmmm ... The only other thing that could have happened is that disks have been moved around, the so-called disk roaming feature ... That would be a cause for not seeing the second logical drive so it appears (page 63).

But still, disks getting too hot really sounds like a hardware issue no?
0
JasonAuthor Commented:
So the fact that pulling out drive 1 has a catastrophic effect, doesn't that prove that drives 1,2,3, and 4 aren't part of any raid array?  But then how the hell am I still seeing 3TB?  I'm not at the server right now and am working on getting in remotely so I can't check any of the suggestions above just yet but I want to make sure I've got all my shit together before I actually get in there.
0
Zephyr ICTCloud ArchitectCommented:
If the datastore fails when you pull out 1 drive than that would mean either there's no RAID5 or there's another drive in the array that is not working like it should ... Or, there's 3 disks configured as RAID0?? All kinds of possibilities.

There were HP Advisories for overheating issues with older BIOS ROMs... But your probably up to date if you ran all those tests, or your predecessor did ... Best to doublecheck though.
0
JasonAuthor Commented:
So if drives 1234 are not on the raid, shouldnt they show up as unassigned in the array config util?
0
arnoldCommented:
If it is a single controller, the drives should be listed there.  They might be presented as individual drives to vmware and are managed there, while the other 4 are combined as a single volume repository,

............
If possible and if not already, transitioning to a newer more stable system/setup should  be considered.
Are there multiple hosts serving the vmware infrastructures? I.e. Other hosts are part of .......
0
Zephyr ICTCloud ArchitectCommented:
Yes they should, but that will not be the case in your situation because then you wouldn't see the 1.5TB drive in VMware... Or, they're available but also still in a "ghost" config or something??
0
JasonAuthor Commented:
There is just this one server, hosting two vms.  Both vms are there dcs and one is the dns and dhcp.  Retarded because now when this server goes down, the whole company is screwed.
0
Zephyr ICTCloud ArchitectCommented:
The only solutions I'm seeing is either a second server ... Or, boot VMware from a USB drive if that's not the case yet, which would maybe free up the disks showing the issues and allowing you to take them out.
0
arnoldCommented:
Setting up a temp/intermediary with one Dc outside that either physical or Vm with DHCP/DNS (split scope DHCP for all scopes) exported from current using netsh dhcp server dump which can then be adjusted and loaded on the new.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, if you have two datastores 1.5TB each, then you have two RAID Arrays, Logical Drives of 1.5TB RAID 5 each.

Possibly....

but of the RAID controller is not showing two logical disks.....

they've not don't something weird here... like span three disks ?
0
JasonAuthor Commented:
I'm not a vmware guru but is it similar to hyper-v then where we can just copy the vmx, vmd, or wmwhatever files and spin them up on some junk server we have lying around temporarily?
0
JasonAuthor Commented:
@andrew that's why I'm at a loss.  Where could this other 1.5TB logical be originating from if the controller (the only one listed in the array config) isn't seeing the drives?
0
JasonAuthor Commented:
...and if 1234 are raid 5, why can't I pop a drive out Without catastrophe?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It would seem, 1,2,3,4 and not RAID 5.

There should be two logical drives of 4 disks, of 1.5TB each.

can you screen Storage Adaptors, and look at the devices....

Yes, you can backup like that, but it's easier if you do this...see my EE Article

Part 10: HOW TO: Backup (Export) and Restore (Import) virtual machines to VMware vSphere Hypervisor 5.1 for FREE
0
JasonAuthor Commented:
Thanks Andrew!  So the question still remains, where are these hard drives 1234 and why wouldn't the be attached to that raid controller?  I've got an HP tech goin there to give the hardware a once-over to see any hardware errors and update firmware if need be but I'm at a loss.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Check the following:-

select the host > configuration > storage adaptors > devices

check here, or screenshot and we can inspect...
0
arnoldCommented:
VMWARE host has the 1,2,3,4 presented to it as a JBOD set (individual drives) which I've not, but if not mistaken, vmware host runs on a linux platform, see if on the host you can cat /proc/mdstat to see whether this is where the software handling is.
if you can not boot when disk 1 is removed, it might be the VMWARE OS Boot drive with the remaining three spanned in RAID 0 as mentioned before.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JasonAuthor Commented:
That's exactly what it is, Arnold.  After review with HP, drives 1-4 are just like you described with no raid enabled onboard.  Upgrading the firmware on the drives to see if the heat issue remains but gonna have to rebuild this sooner than later.  Thank you!
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Migrate the VMs off the dataatore which is not RAIDED to the other datastore which is!

BUT, if ESXi is installed on a single disk, or JBODs, I would recommend re-installing ESXi onto a USB flash drive or SD card.

Also you want to ensure, you are using the OEM HP version of ESXi from HP site.
0
JasonAuthor Commented:
Only one vm is on the unraided drives and datastore 2 is almost full.  We're gonna move it off to a NAS before rebuild.  VMware recommended doing a clone to a NAS. Thanks for heads up on the OEM.  Any other issues you forsee?
0
JasonAuthor Commented:
How big of a flash drive would you use for the esxi?  And you'd recommend that over putting it on the HDD??
0
arnoldCommented:
I think, a 2,4,8,16,32,GB SD or Usb will do
How big is the vm that is on this "data store" and is it "resource heavy I.e. Drive I/O"?


The reason for either gives you the option if drives 1,2,3,4 can be deused in favor of the usb/sd and migrated to another data store, you can repurpose them while booted by usb/sd into a new additional raided data store.
0
JasonAuthor Commented:
The whole vm is currently about 35GB short of its 1.5TB capacity.  It is a dc and houses a handful of sql dbs.  I would like to do the ESXi stuff  on a usb, enable RAID on the controller, raid5 that thing with 4 500GB drives, and call it a day.  Any link to the HP ESXi oem before I gotta google it?
0
arnoldCommented:
Make sure to locate and copy the esxi's serial/key I think it is part of the config on the boot drice, might be visible in the gui.

The other issue deals with which version is installed?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
ESXi - I would recommend a 8GB branded, e.g. HP, SanDisk, Kingston, Lexmar, flash drive or SD Card.

If you want top performance, put all the spindles in the same array.

e.g. all disks, and make one a hot spare.

more spinldles = more disks = more performance = MORE IOPS!
0
JasonAuthor Commented:
Thanks guys!

I'll wait to close this until I finish this week.

I appeappreciate all your insight.
0
JasonAuthor Commented:
Andrew, so in your last statement, were you saying put all 8 drives on one array and just put the two different datastores on the same logical volume?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Well, you would have a single datastore or 3TB, over 8 disks (RAID 5 of 8 discs, and logical volume of 3TB)

OR....

Put all eight disks, in the array, and then carve out logical volumes of 2 x 1.5TB.
0
arnoldCommented:
Not sure you want to do that due to both performance impact of a failure of a single resource and the duration of the rebuild and the consequences of a second failure during that time.

While 2 500 GB are expanded for parity, a failure of one drive, will only impact the single data store allowing for tolerating a failure of a drive in the other logical volume.

Though have not researched it, maybe another expert who has raid expertise,
Would the rebuild of the same 500GB drive in a 1.5TB space take a shorter amount of time as compared to the same 500GB in a 3T space (4 spindles versus 7)
Taking into account that if each datastore hs equal baseline of Disk IO when combined, not sure they are summed, or there is a weighted adjustment .......
0
andyalderCommented:
Note the machine is DL380e, not DL380p.

Probably came with HP Dynamic Smart Array B120i Controller which only supports up to 6 drives. Sounds like someone's put an additional Smart Array controller in it (or a 3rd party JBOD controller - can't tell without taking the lid off it) and cabled the single backplane to two different controllers.

While that does work as far as the data paths are concerned it makes a disaster for SES which looks after the backplane LEDs and temperature sensors on the disks because they can't both talk to the little chip on the disk backplane. You need to either get another backplane or more sensibly cable both plugs on this backplane to a single PCIe Smart Array controller (and not a dynamic/fakeraid one).
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Server Hardware

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.