Slow write speeds on Poweredge R510

Matt Kendall
Matt Kendall used Ask the Experts™
on
Hi,

I have a server running Vmware ESXi on a Dell Power Edge R510.  It's setup with RAID with 4 10K SAS drives.  The capacity is 1.64 TB.  All 4 of the SAS drives have been recently replaced.  I'm not at all familar with Vmware as I've always used Hyper-V and other VM solutions.  There's only on VM setup running Server 2016.  Xeon X5675 (8 processors) with 48 GB RAM allocated to the VM.  The host has 64 GB RAM. 12 CPUs.  A software vendor (I'll call vendor A) recently upgraded their software on the server and it looks like they're using SQL 2012.  They have another db on the server that also uses SQL 2012 well call from vendor B.  There were a bunch of log errors from vendor B's database.  10 GB worth.  They cleared them but their storage benchmark said that the write speeds to the storage were around 22 Mbps.  On 10K SAS drives that seems excessively slow.  I've logged into the ESXi navigator and have spent a few hours going through all of the settings in there.  Is there a way that I can test if all four of the SAS drives are working properly?  Under Storage, Datastore01 and Monitor, there are a few lines from June 13th (nothing listing anything after that).  

Device naa.614xxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 47132 microseconds to 14417 microseconds.      Thursday, June 13, 2019, 03:10:40 -0700      Info
Device naa.614xxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 238928 microseconds to 47132 microseconds.      Thursday, June 13, 2019, 03:10:36 -0700      Info
Device naa.614xxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 7416 microseconds to 238928 microseconds.      Thursday, June 13, 2019, 03:10:30 -0700      Warning

So to recap quickly.  In the past 2 months, all 4 SAS 10K drives were replaced.  Last week, a new SQL2012 database was added to the server.  This morning, everything was working fine for the two QB users and the 4 vendor A database users.  When users using vendor B's database came in, everything slowed down for the other users.  Vendor B's tech logged into the server and cleared 10 GB of log files and restarted their service.  Everything seems to be working okay now except that the write speeds on the server are 22 Mbps and that vendor says that there's something wrong with the hardware on the server.  I'm trying to figure out how to diagnose the problems on the ESXi Navigator page.  Please let me know if you need any further info and thanks in advance for your time and help!
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Adam BrownSenior Systems Admin
Top Expert 2010

Commented:
What kind of RAID are you using? RAID 0, 1, or 5? RAID 5 will put a significant performance hit on the drive array, but not up to 238928ms. What caused the drives to be replaced? Was there a crash on those drives? If that's the case, I'd also look at the RAID controller, because that can certainly cause issues with IO speed if it is wonky.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Do you have any cache enabled for the RAID controller, e.g. Read/Write cache ?

4 x 10K SAS in what RAID configuration?

Deleting 10GB of log files would have caused a broadcast storm on the Datasore, which would affect ALL VMs.
Top Expert 2014

Commented:
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Matt KendallTech / Business owner operator

Author

Commented:
Adam, we have RAID 5.  The manager reported that the server crashed and the former IT group replaced all four of the 10K SAS drives.  I'm guessing that they lost more than one drive.  Probably with some time in between the first drive failure and the second failure.  Since it's running Vmware, I don't know how I can run Dell Openmanage to see what the status is of the drives and the status of the Perc7 controller.  

Andrew, I haven't enabled cache for the RAID controller but I'm still trying to figure out how I see what the RAID status is and the physical drive status as it's running Vmware.  I'm used to working with Hyper-V where I can install Openmanage on the hypervisor and monitor the RAID and physical drive status.  the 4 x 10K SAS is in RAID 5 configuration.  

Andy, I cannot run those Perccli commands on the VM, correct?  I don't have a physical server to run this on.  I connect to Vmware dashboard by going to 192.168.1.40 on a workstation on the network.

Thanks for your help!
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
ut I'm still trying to figure out how I see what the RAID status is and the physical drive status as it's running Vmware.

You don't because VMware just "sees" the RAID LUN presented to it.

But you should be able to look at hardware.

You'll need OSMA installed in ESXi to actually see the disks that make up the RAID 5 array (LUN). or VD as DELL calls it, when you do a CTRL-R at POST.

To run those commands you need to be at the ESXi console (bash shell) or remotely via SSH (and they need to be installed!)
Matt KendallTech / Business owner operator

Author

Commented:
Thanks Andrew.  I'll be at the console tomorrow and I'll give it a try.
Matt KendallTech / Business owner operator

Author

Commented:
Thank you all for your help!  I was able to diagnose the problem at the ESXi console.  Thanks Andrew for the recommendations.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial