Strange Speed issue with Server 2012 Team / Hyper V virtual Networks

Posted on 2014-09-22
Last Modified: 2016-11-23
Trying to understand what is happening and why it's happening.  Basically, I am getting great file transfer speeds from 1gb nics most of the time but then get terrible speed other times.  First some background:

I have 3 Dell R620 in a Hyper V / Failover cluster.  Each node has the same broadcom 4 x 1gb nic setup in it with current drivers.  All run server 2012r2 DC
--2 nic ports are teamed together with built in nic teaming.  They have a 10.1.10.x address.  
--A separate nic port on each server is just a single port with a 192.168.1.x address
--The nic team and single port go to different switches but those switches are on the same network, just separated by VLAN's.  All switches are 1gb ciscos
--The nic team and the static port are then assigned to virtual nics for hyper V guests to use.  Both virtual nics allow management access

1st strange thing - When both the Nic Team and stand alone ports are all connected, I try to transfer a single 2gb file from one machines c drive to another machines c drive.   The file copies at 220mb/s and I can see that both virtual nic's are using their 1gb connection.  But how / why is the single file getting split across two network connections and 2 vlans?  The speed is normal and makes sense but how are packets split and recombined?

2nd issue - If I disable the single port network connection, forcing all traffic over the team, then the same file copy drops to 36mb/s.  I can't seem to figure out why the speed is so slow.

If I disable the nic team and force the traffic over the single 1gb port, then the file copy works at 110mb/s.  I guess this is actually normal and seems to be the only time that traffic flows as I would expect it to.

Any insight here? - Thanks
Question by:CWadmins
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
LVL 16

Accepted Solution

choward16980 earned 250 total points
ID: 40337726
Hyper-V, Nic Teaming and VLans are not recommended and that's insane the packets are reassembled on the other end.  I'm assuming that's something to do with your switches functionality or an incorrectly set VLAN

Should I use VLANs with NIC Teaming in Hyper-V?
No. When using Hyper-V to virtualize workloads, our recommendation is to configure all VLANs for VMs within the Hyper-V Virtual Network Switch instead of configuring VLANs on team interfaces.  Taking this approach ensures that virtual machines are capable of being configured for any VLAN traffic that is passed to the NIC team and prevents confusing configurations that can occur when certain VLAN traffic is split off on a separate VLAN from the Hyper-V Virtual Switch.
LVL 58

Assisted Solution

by:Cliff Galiher
Cliff Galiher earned 250 total points
ID: 40337764
The answer to your first question is an easy one. That is a feature called SMB Multichannel and was introduced in Server 2012. If two servers see multiple network paths between each other, any data transfers that use the SMB protocol (and file transfers fit this) then it will use all available paths simultaneously.

As far as the second strange thing, there is a Broadcom NIC driver bug that doesn't play well when in a 2012 team and when VMQ has been turned on. You can fix this one of two ways. First, update the Broadcom driver. Broadcom claims it is fixed so this should be done no matter what. Having little faith in Broadcom, however, I also recommend the second fix, which is disable VMQ. VMQ has no benefit unless you have 10GB NICs anyways, and the OS disables it regardless of the user setting on 1GB NICs. But the Broadcom bug still kicks in, so  you get the bug without the benefit, Going in and turning the VMQ setting off sidesteps the bug and you won't take any performance hit.


Author Comment

ID: 40337771
I am using VLans on my network but they are not set within Hyper V anywhere.  The different network ports are simply to allow different hyper V guests to communicate on different networks.  I am not nic teaming across vlans or anything like that.  It's just one nic going to one switch and the other two going to a different switch so each host has access to each network for it's guest vms.

Author Comment

ID: 40337809
I have looked into the Broadcom / VMQ issue for awhile now and knew there were some issues with it.  So I actually already have VMQ disabled for every port on all the servers.  I also have it disabled on the Nic Team driver properties (but I don't think that one matters).

I did read about SMB multichannel and forgot about that.  That makes perfect sense.  Thanks,

Featured Post

Raise the IQ of Your IT Alerts

From IT major incidents to manufacturing line slowdowns, every business process generates insights that need to reach the people required to take action. You need a platform that integrates with your business tools to create fully enabled DevOps toolchains.

You need xMatters.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The following article is comprised of the pearls we have garnered deploying virtualization solutions since Virtual Server 2005 and subsequent 2008 RTM+ Hyper-V in standalone and clustered environments.
Veeam Backup & Replication has added a new integration – Veeam Backup for Microsoft Office 365.  In this blog, we will discuss how you can benefit from Office 365 email backup with the Veeam’s new product and try to shed some light on the needs and …
In this Micro Tutorial viewers will learn how they can get their files copied out from their unbootable system without need to use recovery services. As an example non-bootable Windows 2012R2 installation is used which has boot problems.
Michael from AdRem Software explains how to view the most utilized and worst performing nodes in your network, by accessing the Top Charts view in NetCrunch network monitor ( Top Charts is a view in which you can set seve…

691 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question