Slow write speeds on Poweredge R510
Hi,
I have a server running Vmware ESXi on a Dell Power Edge R510. It's setup with RAID with 4 10K SAS drives. The capacity is 1.64 TB. All 4 of the SAS drives have been recently replaced. I'm not at all familar with Vmware as I've always used Hyper-V and other VM solutions. There's only on VM setup running Server 2016. Xeon X5675 (8 processors) with 48 GB RAM allocated to the VM. The host has 64 GB RAM. 12 CPUs. A software vendor (I'll call vendor A) recently upgraded their software on the server and it looks like they're using SQL 2012. They have another db on the server that also uses SQL 2012 well call from vendor B. There were a bunch of log errors from vendor B's database. 10 GB worth. They cleared them but their storage benchmark said that the write speeds to the storage were around 22 Mbps. On 10K SAS drives that seems excessively slow. I've logged into the ESXi navigator and have spent a few hours going through all of the settings in there. Is there a way that I can test if all four of the SAS drives are working properly? Under Storage, Datastore01 and Monitor, there are a few lines from June 13th (nothing listing anything after that).
Device naa.614xxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 47132 microseconds to 14417 microseconds. Thursday, June 13, 2019, 03:10:40 -0700 Info
Device naa.614xxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 238928 microseconds to 47132 microseconds. Thursday, June 13, 2019, 03:10:36 -0700 Info
Device naa.614xxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 7416 microseconds to 238928 microseconds. Thursday, June 13, 2019, 03:10:30 -0700 Warning
So to recap quickly. In the past 2 months, all 4 SAS 10K drives were replaced. Last week, a new SQL2012 database was added to the server. This morning, everything was working fine for the two QB users and the 4 vendor A database users. When users using vendor B's database came in, everything slowed down for the other users. Vendor B's tech logged into the server and cleared 10 GB of log files and restarted their service. Everything seems to be working okay now except that the write speeds on the server are 22 Mbps and that vendor says that there's something wrong with the hardware on the server. I'm trying to figure out how to diagnose the problems on the ESXi Navigator page. Please let me know if you need any further info and thanks in advance for your time and help!
I have a server running Vmware ESXi on a Dell Power Edge R510. It's setup with RAID with 4 10K SAS drives. The capacity is 1.64 TB. All 4 of the SAS drives have been recently replaced. I'm not at all familar with Vmware as I've always used Hyper-V and other VM solutions. There's only on VM setup running Server 2016. Xeon X5675 (8 processors) with 48 GB RAM allocated to the VM. The host has 64 GB RAM. 12 CPUs. A software vendor (I'll call vendor A) recently upgraded their software on the server and it looks like they're using SQL 2012. They have another db on the server that also uses SQL 2012 well call from vendor B. There were a bunch of log errors from vendor B's database. 10 GB worth. They cleared them but their storage benchmark said that the write speeds to the storage were around 22 Mbps. On 10K SAS drives that seems excessively slow. I've logged into the ESXi navigator and have spent a few hours going through all of the settings in there. Is there a way that I can test if all four of the SAS drives are working properly? Under Storage, Datastore01 and Monitor, there are a few lines from June 13th (nothing listing anything after that).
Device naa.614xxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 47132 microseconds to 14417 microseconds. Thursday, June 13, 2019, 03:10:40 -0700 Info
Device naa.614xxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 238928 microseconds to 47132 microseconds. Thursday, June 13, 2019, 03:10:36 -0700 Info
Device naa.614xxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 7416 microseconds to 238928 microseconds. Thursday, June 13, 2019, 03:10:30 -0700 Warning
So to recap quickly. In the past 2 months, all 4 SAS 10K drives were replaced. Last week, a new SQL2012 database was added to the server. This morning, everything was working fine for the two QB users and the 4 vendor A database users. When users using vendor B's database came in, everything slowed down for the other users. Vendor B's tech logged into the server and cleared 10 GB of log files and restarted their service. Everything seems to be working okay now except that the write speeds on the server are 22 Mbps and that vendor says that there's something wrong with the hardware on the server. I'm trying to figure out how to diagnose the problems on the ESXi Navigator page. Please let me know if you need any further info and thanks in advance for your time and help!
Adam Brown
What kind of RAID are you using? RAID 0, 1, or 5? RAID 5 will put a significant performance hit on the drive array, but not up to 238928ms. What caused the drives to be replaced? Was there a crash on those drives? If that's the case, I'd also look at the RAID controller, because that can certainly cause issues with IO speed if it is wonky.
Do you have any cache enabled for the RAID controller, e.g. Read/Write cache ?
4 x 10K SAS in what RAID configuration?
Deleting 10GB of log files would have caused a broadcast storm on the Datasore, which would affect ALL VMs.
4 x 10K SAS in what RAID configuration?
Deleting 10GB of log files would have caused a broadcast storm on the Datasore, which would affect ALL VMs.
What does perccli show for the disk and cache status?
https://www.dell.com/support/article/uk/en/ukdhs1/sln283135/how-to-use-the-poweredge-raid-controller-perc-command-line-interface-cli-utility-to-manage-your-raid-controller?lang=en
https://www.dell.com/support/article/uk/en/ukdhs1/sln283135/how-to-use-the-poweredge-raid-controller-perc-command-line-interface-cli-utility-to-manage-your-raid-controller?lang=en
ASKER
Adam, we have RAID 5. The manager reported that the server crashed and the former IT group replaced all four of the 10K SAS drives. I'm guessing that they lost more than one drive. Probably with some time in between the first drive failure and the second failure. Since it's running Vmware, I don't know how I can run Dell Openmanage to see what the status is of the drives and the status of the Perc7 controller.
Andrew, I haven't enabled cache for the RAID controller but I'm still trying to figure out how I see what the RAID status is and the physical drive status as it's running Vmware. I'm used to working with Hyper-V where I can install Openmanage on the hypervisor and monitor the RAID and physical drive status. the 4 x 10K SAS is in RAID 5 configuration.
Andy, I cannot run those Perccli commands on the VM, correct? I don't have a physical server to run this on. I connect to Vmware dashboard by going to 192.168.1.40 on a workstation on the network.
Thanks for your help!
Andrew, I haven't enabled cache for the RAID controller but I'm still trying to figure out how I see what the RAID status is and the physical drive status as it's running Vmware. I'm used to working with Hyper-V where I can install Openmanage on the hypervisor and monitor the RAID and physical drive status. the 4 x 10K SAS is in RAID 5 configuration.
Andy, I cannot run those Perccli commands on the VM, correct? I don't have a physical server to run this on. I connect to Vmware dashboard by going to 192.168.1.40 on a workstation on the network.
Thanks for your help!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks Andrew. I'll be at the console tomorrow and I'll give it a try.
ASKER
Thank you all for your help! I was able to diagnose the problem at the ESXi console. Thanks Andrew for the recommendations.