Solved

Hyper-V networking slow

Posted on 2013-01-04
12
4,328 Views
Last Modified: 2016-11-23
Dell PowerEdge T420, Windows Server 2008 R2 with Hyper-V installed and a single 2008 R2 virtual machine setup within the Hyper-V environment. The two NICs on the T420 are connected to the same Gigabit switch, one for the physical machine's use and the other for the virtual network.

The only thing being done in the VM is Remote Desktop Services and then end-users running a medical records program which is accessing a database on completely separate server. The physical server just has Hyper-V, a domain controller and DNS.

With no activity, ping'ing the guest from the physical machine or vice versa, or a separate database server from the guest machine, we get under 1 ms response times. However, if there is any activity in the guest OS, ping times vary drastically -- between 1 ms and 200 ms.

I have the various Offloads disabled in the virtual network adapters properties but that does not seem to make any difference. Looking at the Resource Monitor on the virtual machines, there is generally under 5% of the CPU and 10% of the memory being used, and network traffic when the application is running is under 100 Kbps (usually under 50 Kbps).

On the physical server, the CPU & network traffic are basically the same as the virtual machine, with 80% of the memory in use (since it is all allocated to Hyper-V).

Any thoughts on how to solve the slow networking?

Thanks in advance!
0
Comment
Question by:A-p-u
  • 4
  • 2
  • 2
  • +4
12 Comments
 
LVL 18

Expert Comment

by:Andrej Pirman
Comment Utility
Seems like driver problem, or compatibility with network card. Read here, links to solution, too:
http://social.technet.microsoft.com/Forums/en/winserverhyperv/thread/29c669db-30fe-4196-9b95-a9d5e48ac318?prof=required
0
 
LVL 20

Expert Comment

by:Svet Paperov
Comment Utility
It doesn’t seem as a network problem to me with one condition: did you disable the “Allow management operating system to share this network adapter” option on the virtual NIC?

What is the storage setup? RAID, disks, volumes, where is the VHD file located, is it fixed or dynamic?

What is the total RAM of the physical server and how much is the RAM reserved for the VM and the Hyper-V host?

What is the size of the domain, how many users do you have?
0
 
LVL 1

Author Comment

by:A-p-u
Comment Utility
Driver & firmware updated with the latest on Dell's website. (Broadcom NetXtreme Gigabit adapters).  All the "offloads" are disabled in the virtual machine, and I've tried with them enabled or disabled on the physical machine with no change.

"Allow management operating system to share this network adapter" is disabled on the one NIC used by the VM.

Physical machine has a RAID 5 array across three 500 GB SATA drives. One 80 GB system partition, the rest is a data partition that only has the virtual machine files. 32 GB of RAM in the server.

Virtual machine has a single disk (dynamic - currently using 10 GB out of 80 GB allocated to it) - and is allocated 24 GB of RAM.
0
 
LVL 1

Accepted Solution

by:
A-p-u earned 0 total points
Comment Utility
The solution to the problem turns out to be...

The Broadcom NICs in this Dell server have a setting "Virtual Machine Queues" that defaults to enabled. Once disabled, the guest OS has the same network speed as the physical machine does.
0
 
LVL 20

Expert Comment

by:Svet Paperov
Comment Utility
Thanks for sharing.

However, I am little bit surprised by the solution. I have several Dell R710 servers running Hyper-V role of Windows Server 2008 R2 with Broadcom and Intel adapters and I do not have problems with this option enabled (by default; I didn’t know its existence until today). It actually should help to increase the throughput according to this document http://www.dell.com/downloads/global/power/ps1q10-20100101-Chaudhary.pdf

Did you have the latest NIC drives? Did you run other tests for real network throughput? For example, a free I/O benchmark utility from Microsoft called SQLIO simulates database requests with different block size.

Thanks again
0
 
LVL 1

Author Comment

by:A-p-u
Comment Utility
Based on the description of the feature, it should definitely help being enabled. But, from a practical standpoint, disabling it immediately resolves the problem and enabling it recreates the problem.

I did install the latest drivers and firmware on Dell's site without any change prior to trying this. A different Dell server purchased around the same time (PowerEdge T620) has Intel NICs and had this setting disabled from the factory, and had no problems with a similar single Hyper-V VM running on it. (We did not try enabling it and seeing if that caused a performance problem on that server.)

No scientific benchmarking. Just the real-world performance of the client's medical records program, or doing a file copy in Windows Explorer (~ 2 MB/s when enabled, ~ 10 MB/s when disabled).

(P.S. One potentially significant correction to original post - the server's two NICs are connected to the same 100Mbps switch, not a Gigabit switch. The Dell VMQ white paper and some other documentation on Microsoft's site about it, only reference it with Gigabit and 10GE NICs so it might only work with Gigabit or better switches.)
0
Do email signature updates give you a headache?

Do you feel like all of your time is spent managing email signatures? Too busy to visit every user’s desk to make updates? Want high-quality HTML signatures on all devices, including on mobiles and Macs? Then, let Exclaimer solve all your email signature problems today!

 
LVL 1

Author Closing Comment

by:A-p-u
Comment Utility
Making this change on the driver's settings resolved the problem entirely.

Looking at a different Dell server bought around the same time, but which has Intel NICs shows the default setting there is already "disabled."

Many sites reference turning of Large Send Offloads or TCP Checksum Offloads but I haven't see any reference to this "Virtual Machine Queues" setting before.
0
 

Expert Comment

by:dksgreenit
Comment Utility
"Virtual Machine Queues" worked for me, I also have a Dell server, same NIC, also defaulted to Enable.  

This was driving me NUTS because the data transfer rates to the guest system were so slow it calculated in kb/s.  I am in the process of migrating a 2003 physical machine to a 2008 R2 VM.  I need to move approx 300 GB of data and at 130 kb/s it was not going to happen any time soon.
0
 

Expert Comment

by:K-9
Comment Utility
Thank you!! This saved me on my network setup that I was doing for a customer!
0
 

Expert Comment

by:breichard
Comment Utility
I realize it's two years later, but thank you so much for this info.  I just installed a Dell T420 with Broadcom NIC's just as the A-p-u described (one NIC for host, one for VMs) and p2v'd a Server 2003 R2 machine.  Performance was terrible to the VM.  I left a 45000-byte ping running from another machine to the VM before I made the change.  The times before the change were from 2ms to 220ms.  After the change (broke the connection for 30 seconds), times went down to 1ms-2ms.  Thanks again!!
0
 

Expert Comment

by:breichard
Comment Utility
PS, for those of you reading this later - the setting is in Network and Sharing Center, then change adapter settings, properties on the adapter you want to change, then Configure on the adapter, then Advanced tab.
0
 
LVL 1

Expert Comment

by:fishturd
Comment Utility
I had same problem on a T420 with 2012 R2, disabling Virtual Machine Queues, fixed for me
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
The recent Microsoft changes on update philosophy for Windows pre-10 and their impact on existing WSUS implementations.
This tutorial will walk an individual through the steps necessary to configure their installation of BackupExec 2012 to use network shared disk space. Verify that the path to the shared storage is valid and that data can be written to that location:…
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now