Solved

2012 R2 Hyper-V Slow HDD Performance

Posted on 2013-12-11
8
8,379 Views
Last Modified: 2016-11-23
Physical server - Dell PowerEdge 520 with H710 Raid Controller
Host OS - 2012 R2 fully updated and only Hyper-V service running
Host OS can copy data between logical volumes at high rates of speed - 200 MB/s upwards
Server has all SAS 10k RPM drives
Server is brand new
Guest OS 1 - 2012 R2 fully updated with AD, DHCP and DNS running
Guest OS 2 - 2012 R2 fully updated with Exchange 2013 running
Everything functions perfectly with regard to applications and core functionality.
Problem - data transfer to Guest OS (either including random test VM) is beyond terribly slow. Transfer speed is in the bytes and might sometimes get as high as 3-4 Mb/s.
Guest to Guest same thing. Host to Guest same thing.
Host B volume to Host C or D volume extremely fast.
VM Hard Drive is VHDX (tried fixed and dynamic)
Tried Generation 1 with IDE controller
Currently running Generation 2 with SCSI controller
Tried changing the channels for the SCSI drives and making sure they're unique
Currently updating Firmware and Drivers for RAID controller.
What am I missing??? I have never seen performance this poor. My test lab is the same setup and is lightning fast between Host and Guest.
0
Comment
Question by:CUBLA1
  • 5
  • 2
8 Comments
 
LVL 6

Expert Comment

by:RaithZ
ID: 39712965
What are the disk configurations?  Like a Raid 5 and then how is that divided?  Which volume are the guest machines stored on?  are the files being transferred from that volume, to the guest servers?  If so then you are reading and writing from the same logical partition/array which will significantly decrease disk performance.  It sounds like your host C and D drives are on separate arrays, which could explain why the transfer is so fast.
0
 

Author Comment

by:CUBLA1
ID: 39712987
Now I'm getting concerned that I have royally screwed myself because I've just spent 2 days prepping this entire network and just now realized the performance problem! What's weird is the entire time I was installing and configuring everything (even remotely) I was surprised at how smoking fast the server actually is. It just seems to be when moving data around that you notice something is wrong. Application performance is good. I just can't copy data from the old servers to the new ones because it said it would take 3945 hours!

I had to install Dell OMSA to review the RAID configuration. The server has (and my first post was wrong) six (6) SAS 300 Gb 15k RPM drives and 1 SATA 2 TB 7200 RPM.

Virtual Drive 0 - Raid 1 - 300 Gb Total (2 300Gb Drives) OS / Hyper-V
Virtual Drive 1 - Raid 5 - 900 Gb Total (4 300Gb Drives) VM and VHD storage for Hyper-V
Virtual Drive 2 - Raid 0 - 2 TB SATA Backup Drive (also SAS)

I am storing the Hyper-V VM and VHD locations on the same volume (VD-1 Raid 5) if that matters. The VM's run super fast. Everything runs super fast. I am using the 2012 R2 ISO at 4 Gb as the test file. If I copy from VD 0 to 1 no problem. Even if I copy it from VD 2 (the sata drive) to VD 0 or 1 no problem. If I login to any VM and copy it from anywhere to the VM it's slow as Christmas.
0
 

Author Comment

by:CUBLA1
ID: 39712995
Office File Transfer to VM Desktop
0
PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

 

Author Comment

by:CUBLA1
ID: 39712998
This is the best its been yet. I updated RAID driver and firmware along with Bios and Intel Chipset. It seems to have improved dramatically compared to what it was before. It still seems to be bouncing from 20 mb/s to nothing up and down. Driver issue?
Win2012 File Transfer to VM desktop
0
 
LVL 6

Expert Comment

by:RaithZ
ID: 39714520
The transfer that is slow, is the one that is going from one VM on the Raid 5 to any place else on the Raid 5.. or from any place to a VM... or from VM to VM right?  


All the tests you say are fast are going from one array to another, therefore the controller can read and write at the same time.

What you are seeing is not really abnormal for the situations described, though you might benefit from a RAID controller with more cache memory or a different controller.  

There are several factors that I think are contributing to your issue, each mentioned above.

So it sounds like its reading data, then writing it to the destination back and forth.  If you monitor the disk transfer at the source does it also go back and forth?
0
 

Accepted Solution

by:
CUBLA1 earned 0 total points
ID: 39772634
Actually it was very abnormal and Dell created a knowledge base article with regard to this particular problem. Seems there is a problem in 2012 R2 with Hyper-V and Virtual Machine Queues. We disabled VMQ on the physical NIC's and everything shot through the roof.
0
 

Author Closing Comment

by:CUBLA1
ID: 39781842
Forced to call Dell and have Tier 3 Microsoft support determine cause of problem. Took them 30 minutes to fix something I researched for 10 hours.
0
 
LVL 38

Expert Comment

by:Philip Elder
ID: 40469110
By default: No matter the server manufacturer Broadcom Gigabit physical NIC ports _MUST_ have VMQ disabled.

Broadcom for whatever reason chose to ignore the VMQ specification, since it was designed for 10GbE and higher, and enable it on 1GbE ports. This breaks everything as there is not enough network traffic on 1Gb for things to work. :(
0

Featured Post

Active Directory Webinar

We all know we need to protect and secure our privileges, but where to start? Join Experts Exchange and ManageEngine on Tuesday, April 11, 2017 10:00 AM PDT to learn how to track and secure privileged users in Active Directory.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The following article is comprised of the pearls we have garnered deploying Hyper-V since Virtual Server 2005 and subsequent 2008 RTM+ standalone and clustered.
In this article, I am going to show you how to simulate a multi-site Lab environment on a single Hyper-V host. I use this method successfully in my own lab to simulate three fully routed global AD Sites on a Windows 10 Hyper-V host.
Email security requires an ever evolving service that stays up to date with counter-evolving threats. The Email Laundry perform Research and Development to ensure their email security service evolves faster than cyber criminals. We apply our Threat…
I've attached the XLSM Excel spreadsheet I used in the video and also text files containing the macros used below. https://filedb.experts-exchange.com/incoming/2017/03_w12/1151775/Permutations.txt https://filedb.experts-exchange.com/incoming/201…

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question