2012 R2 Hyper-V Slow HDD Performance
Physical server - Dell PowerEdge 520 with H710 Raid Controller
Host OS - 2012 R2 fully updated and only Hyper-V service running
Host OS can copy data between logical volumes at high rates of speed - 200 MB/s upwards
Server has all SAS 10k RPM drives
Server is brand new
Guest OS 1 - 2012 R2 fully updated with AD, DHCP and DNS running
Guest OS 2 - 2012 R2 fully updated with Exchange 2013 running
Everything functions perfectly with regard to applications and core functionality.
Problem - data transfer to Guest OS (either including random test VM) is beyond terribly slow. Transfer speed is in the bytes and might sometimes get as high as 3-4 Mb/s.
Guest to Guest same thing. Host to Guest same thing.
Host B volume to Host C or D volume extremely fast.
VM Hard Drive is VHDX (tried fixed and dynamic)
Tried Generation 1 with IDE controller
Currently running Generation 2 with SCSI controller
Tried changing the channels for the SCSI drives and making sure they're unique
Currently updating Firmware and Drivers for RAID controller.
What am I missing??? I have never seen performance this poor. My test lab is the same setup and is lightning fast between Host and Guest.
Host OS - 2012 R2 fully updated and only Hyper-V service running
Host OS can copy data between logical volumes at high rates of speed - 200 MB/s upwards
Server has all SAS 10k RPM drives
Server is brand new
Guest OS 1 - 2012 R2 fully updated with AD, DHCP and DNS running
Guest OS 2 - 2012 R2 fully updated with Exchange 2013 running
Everything functions perfectly with regard to applications and core functionality.
Problem - data transfer to Guest OS (either including random test VM) is beyond terribly slow. Transfer speed is in the bytes and might sometimes get as high as 3-4 Mb/s.
Guest to Guest same thing. Host to Guest same thing.
Host B volume to Host C or D volume extremely fast.
VM Hard Drive is VHDX (tried fixed and dynamic)
Tried Generation 1 with IDE controller
Currently running Generation 2 with SCSI controller
Tried changing the channels for the SCSI drives and making sure they're unique
Currently updating Firmware and Drivers for RAID controller.
What am I missing??? I have never seen performance this poor. My test lab is the same setup and is lightning fast between Host and Guest.
RaithZ
What are the disk configurations? Like a Raid 5 and then how is that divided? Which volume are the guest machines stored on? are the files being transferred from that volume, to the guest servers? If so then you are reading and writing from the same logical partition/array which will significantly decrease disk performance. It sounds like your host C and D drives are on separate arrays, which could explain why the transfer is so fast.
ASKER
Now I'm getting concerned that I have royally screwed myself because I've just spent 2 days prepping this entire network and just now realized the performance problem! What's weird is the entire time I was installing and configuring everything (even remotely) I was surprised at how smoking fast the server actually is. It just seems to be when moving data around that you notice something is wrong. Application performance is good. I just can't copy data from the old servers to the new ones because it said it would take 3945 hours!
I had to install Dell OMSA to review the RAID configuration. The server has (and my first post was wrong) six (6) SAS 300 Gb 15k RPM drives and 1 SATA 2 TB 7200 RPM.
Virtual Drive 0 - Raid 1 - 300 Gb Total (2 300Gb Drives) OS / Hyper-V
Virtual Drive 1 - Raid 5 - 900 Gb Total (4 300Gb Drives) VM and VHD storage for Hyper-V
Virtual Drive 2 - Raid 0 - 2 TB SATA Backup Drive (also SAS)
I am storing the Hyper-V VM and VHD locations on the same volume (VD-1 Raid 5) if that matters. The VM's run super fast. Everything runs super fast. I am using the 2012 R2 ISO at 4 Gb as the test file. If I copy from VD 0 to 1 no problem. Even if I copy it from VD 2 (the sata drive) to VD 0 or 1 no problem. If I login to any VM and copy it from anywhere to the VM it's slow as Christmas.
I had to install Dell OMSA to review the RAID configuration. The server has (and my first post was wrong) six (6) SAS 300 Gb 15k RPM drives and 1 SATA 2 TB 7200 RPM.
Virtual Drive 0 - Raid 1 - 300 Gb Total (2 300Gb Drives) OS / Hyper-V
Virtual Drive 1 - Raid 5 - 900 Gb Total (4 300Gb Drives) VM and VHD storage for Hyper-V
Virtual Drive 2 - Raid 0 - 2 TB SATA Backup Drive (also SAS)
I am storing the Hyper-V VM and VHD locations on the same volume (VD-1 Raid 5) if that matters. The VM's run super fast. Everything runs super fast. I am using the 2012 R2 ISO at 4 Gb as the test file. If I copy from VD 0 to 1 no problem. Even if I copy it from VD 2 (the sata drive) to VD 0 or 1 no problem. If I login to any VM and copy it from anywhere to the VM it's slow as Christmas.
ASKER
ASKER
This is the best its been yet. I updated RAID driver and firmware along with Bios and Intel Chipset. It seems to have improved dramatically compared to what it was before. It still seems to be bouncing from 20 mb/s to nothing up and down. Driver issue?
The transfer that is slow, is the one that is going from one VM on the Raid 5 to any place else on the Raid 5.. or from any place to a VM... or from VM to VM right?
All the tests you say are fast are going from one array to another, therefore the controller can read and write at the same time.
What you are seeing is not really abnormal for the situations described, though you might benefit from a RAID controller with more cache memory or a different controller.
There are several factors that I think are contributing to your issue, each mentioned above.
So it sounds like its reading data, then writing it to the destination back and forth. If you monitor the disk transfer at the source does it also go back and forth?
All the tests you say are fast are going from one array to another, therefore the controller can read and write at the same time.
What you are seeing is not really abnormal for the situations described, though you might benefit from a RAID controller with more cache memory or a different controller.
There are several factors that I think are contributing to your issue, each mentioned above.
So it sounds like its reading data, then writing it to the destination back and forth. If you monitor the disk transfer at the source does it also go back and forth?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Forced to call Dell and have Tier 3 Microsoft support determine cause of problem. Took them 30 minutes to fix something I researched for 10 hours.
By default: No matter the server manufacturer Broadcom Gigabit physical NIC ports _MUST_ have VMQ disabled.
Broadcom for whatever reason chose to ignore the VMQ specification, since it was designed for 10GbE and higher, and enable it on 1GbE ports. This breaks everything as there is not enough network traffic on 1Gb for things to work. :(
Broadcom for whatever reason chose to ignore the VMQ specification, since it was designed for 10GbE and higher, and enable it on 1GbE ports. This breaks everything as there is not enough network traffic on 1Gb for things to work. :(