Strange Speed issue with Server 2012 Team / Hyper V virtual Networks

Trying to understand what is happening and why it's happening.  Basically, I am getting great file transfer speeds from 1gb nics most of the time but then get terrible speed other times.  First some background:

I have 3 Dell R620 in a Hyper V / Failover cluster.  Each node has the same broadcom 4 x 1gb nic setup in it with current drivers.  All run server 2012r2 DC
--2 nic ports are teamed together with built in nic teaming.  They have a 10.1.10.x address.  
--A separate nic port on each server is just a single port with a 192.168.1.x address
--The nic team and single port go to different switches but those switches are on the same network, just separated by VLAN's.  All switches are 1gb ciscos
--The nic team and the static port are then assigned to virtual nics for hyper V guests to use.  Both virtual nics allow management access

1st strange thing - When both the Nic Team and stand alone ports are all connected, I try to transfer a single 2gb file from one machines c drive to another machines c drive.   The file copies at 220mb/s and I can see that both virtual nic's are using their 1gb connection.  But how / why is the single file getting split across two network connections and 2 vlans?  The speed is normal and makes sense but how are packets split and recombined?

2nd issue - If I disable the single port network connection, forcing all traffic over the team, then the same file copy drops to 36mb/s.  I can't seem to figure out why the speed is so slow.

If I disable the nic team and force the traffic over the single 1gb port, then the file copy works at 110mb/s.  I guess this is actually normal and seems to be the only time that traffic flows as I would expect it to.

Any insight here? - Thanks
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Chris HInfrastructure ManagerCommented:
Hyper-V, Nic Teaming and VLans are not recommended and that's insane the packets are reassembled on the other end.  I'm assuming that's something to do with your switches functionality or an incorrectly set VLAN

Should I use VLANs with NIC Teaming in Hyper-V?
No. When using Hyper-V to virtualize workloads, our recommendation is to configure all VLANs for VMs within the Hyper-V Virtual Network Switch instead of configuring VLANs on team interfaces.  Taking this approach ensures that virtual machines are capable of being configured for any VLAN traffic that is passed to the NIC team and prevents confusing configurations that can occur when certain VLAN traffic is split off on a separate VLAN from the Hyper-V Virtual Switch.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Cliff GaliherCommented:
The answer to your first question is an easy one. That is a feature called SMB Multichannel and was introduced in Server 2012. If two servers see multiple network paths between each other, any data transfers that use the SMB protocol (and file transfers fit this) then it will use all available paths simultaneously.

As far as the second strange thing, there is a Broadcom NIC driver bug that doesn't play well when in a 2012 team and when VMQ has been turned on. You can fix this one of two ways. First, update the Broadcom driver. Broadcom claims it is fixed so this should be done no matter what. Having little faith in Broadcom, however, I also recommend the second fix, which is disable VMQ. VMQ has no benefit unless you have 10GB NICs anyways, and the OS disables it regardless of the user setting on 1GB NICs. But the Broadcom bug still kicks in, so  you get the bug without the benefit, Going in and turning the VMQ setting off sidesteps the bug and you won't take any performance hit.

CWadminsAuthor Commented:
I am using VLans on my network but they are not set within Hyper V anywhere.  The different network ports are simply to allow different hyper V guests to communicate on different networks.  I am not nic teaming across vlans or anything like that.  It's just one nic going to one switch and the other two going to a different switch so each host has access to each network for it's guest vms.
CWadminsAuthor Commented:
I have looked into the Broadcom / VMQ issue for awhile now and knew there were some issues with it.  So I actually already have VMQ disabled for every port on all the servers.  I also have it disabled on the Nic Team driver properties (but I don't think that one matters).

I did read about SMB multichannel and forgot about that.  That makes perfect sense.  Thanks,
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Switches / Hubs

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.