?
Solved

Hyper-V networking slow

Posted on 2013-01-04
12
Medium Priority
?
4,471 Views
Last Modified: 2016-11-23
Dell PowerEdge T420, Windows Server 2008 R2 with Hyper-V installed and a single 2008 R2 virtual machine setup within the Hyper-V environment. The two NICs on the T420 are connected to the same Gigabit switch, one for the physical machine's use and the other for the virtual network.

The only thing being done in the VM is Remote Desktop Services and then end-users running a medical records program which is accessing a database on completely separate server. The physical server just has Hyper-V, a domain controller and DNS.

With no activity, ping'ing the guest from the physical machine or vice versa, or a separate database server from the guest machine, we get under 1 ms response times. However, if there is any activity in the guest OS, ping times vary drastically -- between 1 ms and 200 ms.

I have the various Offloads disabled in the virtual network adapters properties but that does not seem to make any difference. Looking at the Resource Monitor on the virtual machines, there is generally under 5% of the CPU and 10% of the memory being used, and network traffic when the application is running is under 100 Kbps (usually under 50 Kbps).

On the physical server, the CPU & network traffic are basically the same as the virtual machine, with 80% of the memory in use (since it is all allocated to Hyper-V).

Any thoughts on how to solve the slow networking?

Thanks in advance!
0
Comment
Question by:A-p-u
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 2
  • 2
  • +4
12 Comments
 
LVL 18

Expert Comment

by:Andrej Pirman
ID: 38746706
Seems like driver problem, or compatibility with network card. Read here, links to solution, too:
http://social.technet.microsoft.com/Forums/en/winserverhyperv/thread/29c669db-30fe-4196-9b95-a9d5e48ac318?prof=required
0
 
LVL 20

Expert Comment

by:Svet Paperov
ID: 38746795
It doesn’t seem as a network problem to me with one condition: did you disable the “Allow management operating system to share this network adapter” option on the virtual NIC?

What is the storage setup? RAID, disks, volumes, where is the VHD file located, is it fixed or dynamic?

What is the total RAM of the physical server and how much is the RAM reserved for the VM and the Hyper-V host?

What is the size of the domain, how many users do you have?
0
 
LVL 1

Author Comment

by:A-p-u
ID: 38747977
Driver & firmware updated with the latest on Dell's website. (Broadcom NetXtreme Gigabit adapters).  All the "offloads" are disabled in the virtual machine, and I've tried with them enabled or disabled on the physical machine with no change.

"Allow management operating system to share this network adapter" is disabled on the one NIC used by the VM.

Physical machine has a RAID 5 array across three 500 GB SATA drives. One 80 GB system partition, the rest is a data partition that only has the virtual machine files. 32 GB of RAM in the server.

Virtual machine has a single disk (dynamic - currently using 10 GB out of 80 GB allocated to it) - and is allocated 24 GB of RAM.
0
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

 
LVL 1

Accepted Solution

by:
A-p-u earned 0 total points
ID: 38749224
The solution to the problem turns out to be...

The Broadcom NICs in this Dell server have a setting "Virtual Machine Queues" that defaults to enabled. Once disabled, the guest OS has the same network speed as the physical machine does.
0
 
LVL 20

Expert Comment

by:Svet Paperov
ID: 38749471
Thanks for sharing.

However, I am little bit surprised by the solution. I have several Dell R710 servers running Hyper-V role of Windows Server 2008 R2 with Broadcom and Intel adapters and I do not have problems with this option enabled (by default; I didn’t know its existence until today). It actually should help to increase the throughput according to this document http://www.dell.com/downloads/global/power/ps1q10-20100101-Chaudhary.pdf

Did you have the latest NIC drives? Did you run other tests for real network throughput? For example, a free I/O benchmark utility from Microsoft called SQLIO simulates database requests with different block size.

Thanks again
0
 
LVL 1

Author Comment

by:A-p-u
ID: 38749484
Based on the description of the feature, it should definitely help being enabled. But, from a practical standpoint, disabling it immediately resolves the problem and enabling it recreates the problem.

I did install the latest drivers and firmware on Dell's site without any change prior to trying this. A different Dell server purchased around the same time (PowerEdge T620) has Intel NICs and had this setting disabled from the factory, and had no problems with a similar single Hyper-V VM running on it. (We did not try enabling it and seeing if that caused a performance problem on that server.)

No scientific benchmarking. Just the real-world performance of the client's medical records program, or doing a file copy in Windows Explorer (~ 2 MB/s when enabled, ~ 10 MB/s when disabled).

(P.S. One potentially significant correction to original post - the server's two NICs are connected to the same 100Mbps switch, not a Gigabit switch. The Dell VMQ white paper and some other documentation on Microsoft's site about it, only reference it with Gigabit and 10GE NICs so it might only work with Gigabit or better switches.)
0
 
LVL 1

Author Closing Comment

by:A-p-u
ID: 38766429
Making this change on the driver's settings resolved the problem entirely.

Looking at a different Dell server bought around the same time, but which has Intel NICs shows the default setting there is already "disabled."

Many sites reference turning of Large Send Offloads or TCP Checksum Offloads but I haven't see any reference to this "Virtual Machine Queues" setting before.
0
 

Expert Comment

by:dksgreenit
ID: 38871226
"Virtual Machine Queues" worked for me, I also have a Dell server, same NIC, also defaulted to Enable.  

This was driving me NUTS because the data transfer rates to the guest system were so slow it calculated in kb/s.  I am in the process of migrating a 2003 physical machine to a 2008 R2 VM.  I need to move approx 300 GB of data and at 130 kb/s it was not going to happen any time soon.
0
 

Expert Comment

by:K-9
ID: 39094807
Thank you!! This saved me on my network setup that I was doing for a customer!
0
 

Expert Comment

by:breichard
ID: 39173816
I realize it's two years later, but thank you so much for this info.  I just installed a Dell T420 with Broadcom NIC's just as the A-p-u described (one NIC for host, one for VMs) and p2v'd a Server 2003 R2 machine.  Performance was terrible to the VM.  I left a 45000-byte ping running from another machine to the VM before I made the change.  The times before the change were from 2ms to 220ms.  After the change (broke the connection for 30 seconds), times went down to 1ms-2ms.  Thanks again!!
0
 

Expert Comment

by:breichard
ID: 39173818
PS, for those of you reading this later - the setting is in Network and Sharing Center, then change adapter settings, properties on the adapter you want to change, then Configure on the adapter, then Advanced tab.
0
 
LVL 1

Expert Comment

by:fishturd
ID: 39705797
I had same problem on a T420 with 2012 R2, disabling Virtual Machine Queues, fixed for me
0

Featured Post

Are your AD admin tools letting you down?

Managing Active Directory can get complicated.  Often, the native tools for managing AD are just not up to the task.  The largest Active Directory installations in the world have relied on one tool to manage their day-to-day administration tasks: Hyena. Start your trial today.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A safe way to clean winsxs folder from your windows server 2008 R2 editions
After seeing many questions for JRNL_WRAP_ERROR for replication failure, I thought it would be useful to write this article.
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…
Suggested Courses
Course of the Month14 days, 16 hours left to enroll

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question