Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win


Hyper-V vNIC not responding

Posted on 2015-01-08
Medium Priority
Last Modified: 2015-01-13
Hello guys, I have a problem that Ive been pulling my hair out for quite some time.

What we have setup is a Windows Failover Cluster setup across a number of blade servers. Each server has Windows Server 2012R2 installed on it and the Hyper-V and failover clustering roles installed. Each server has 4 physical network interfaces, two HP NC373i Integrated and two HP NC373m Mezzanine. I only have 2 of 4 physical NICs connected to our switch at this time they are switch independently teamed through Windows. On top of this we have eight virtual NICs connected to a Hyper-V virtual switch:

Access/Management, Cluster network, Migration network, Replica network, and four SMB data transfer networks (for accessing VHDs on a storage server)

We have each virtual NIC on a separate VLAN and they all have statically assigned IP addresses. Occasionally, one of the vNICs will stop working and we will lose Live Migration on that blade, or it may lose communication with the cluster depending on which virtual interfaces have failed.

Ive tried updating the drivers on both sets of physical NICs, reflashing the firmware, turning on/off certain subsystems such as VMQ or RSC etc but nothing has solved this. The interesting thing to note is that if I toggle VMQ on or off it sometimes caused the affected NICs to start responding again, but only for a limited time. I should mention no where in the Network Connections does it state these NICs are malfunctioning or disconnected, it does however list it in Event Viewer as a clustering failure.

edit: when I say the NIC is not responding Im meaning the other hosts can not ping it even though it should. Yes Firewall is off
Question by:Lumenix
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4
LVL 59

Assisted Solution

by:Cliff Galiher
Cliff Galiher earned 750 total points
ID: 40538613
1)  Disable VMQ. There is no use for it on gigabit adapters.

2) Grab updated Broadcom drivers. HP has been sadly very slow on updating drivers, and Broadcom is (also sadly) regularly subpar on driver quality. So combine the two, and you have a bad Broadcom driver that they have (probably) fixed but that HP hasn't rebranded and re-released yet.  Personally, I'd go with Intel, but depending the blades you chose, that may not be an option.

3) Make sure you've implemented reasonable QoS settings on your various vNICs. Otherwise the virtual switch won't prioritize packets and you can eventually end up with a vNIC feeling starved, even after load has resumed "normal" low levels. That's the nature of dynamic teaming and running a converged network. QoS is mandatory in such a setup to ensure no single vNIC can crash the others.


Author Comment

ID: 40538907
Thanks for suggestion Cliff, Ive tried turning off VMQ on my nics using the command.

Get-NetAdapterVmq | Disable-NetAdapterVmq

This hasnt provided a permanent fix unfortunately. Ive also removed all vNICs and recreated them. The same ones are not replying to pings after this either. I checked and updated the Broadcom drivers as well and reflashed the firmware on all four NICs and of course including a reboot. Still the same problem.
LVL 39

Expert Comment

by:Philip Elder
ID: 40538968
With Cliff. Broadcom requires VMQ to be disabled on _all_ physical NIC ports that run at Gigabit speeds.

Check to see if there is a firmware update for the NICs as well.

In this scenario we would:
 Team 1: Port 0 on each: Management (VLAN for services if required)
 Team 2: Port 1 on each: vSwitch (not shared with OS) (VLAN for VMs via Hyper-V vNIC Properties)

Veeam Task Manager for Hyper-V

Task Manager for Hyper-V provides critical information that allows you to monitor Hyper-V performance by displaying real-time views of CPU and memory at the individual VM-level, so you can quickly identify which VMs are using host resources.


Author Comment

ID: 40544359
Thanks for the suggestions. I have checked and it seems VMQ is not enabled on the NIC Team but the problem persists. Is there some other way to disable it instead of in Powershell?

Ive managed to fix the problem by using a single NIC instead of a teamed one, however then we lose fault tolerant networking to the blade. Im trying to see if I can use the Broadcom utility (BASC) to configure a NIC team and see if the problem persists there. If any of you have further suggestions let me know please!
LVL 39

Expert Comment

by:Philip Elder
ID: 40544707
Not the team. The ports.

Click Start --> ncpa.cpl --> pNIC Properties --> Advanced --> Virtual Machine Queues (VMQ) --> Set DISABLED.

Do that for all physical NICs.

Author Comment

ID: 40545346
Interestingly enough that option isnt there. The model of NIC is HP NC373i and NC373m. On our HP G6 blades, which Broadcom BCM57711e 10Gbe the option is there but I have not had the VMQ issue on these blades yet.
LVL 39

Expert Comment

by:Philip Elder
ID: 40545359
10GbE works fine with VMQ. It is on 1Gb connections that things get munged.

Author Comment

ID: 40547004
Alright, Ive done a fresh OS install and configured the vNICs using the BASC team instead of Windows software teaming. Everything works fine for now Ill maybe update later on for those who stumble across this post in the future. I do have one more thing to ask however. It looks like the Physical NICs Im using do not support VMQ anyways since there is no option to turn it on or off. However when running Get-NetAdapterVMQ is shows the NIC team (BASC) as using VMQ...

I try to disable it in Powershell and it tell me it cannot set the property to disabled, any ideas?
LVL 39

Accepted Solution

Philip Elder earned 750 total points
ID: 40547065
The Broadcom management software may expose those settings.

If the actual physical NIC port does not show them then perhaps they are not supported at all as you say. If that is the case then the OS setting should be meaningless anyway.

Author Closing Comment

ID: 40547388
Wasnt the actual solution was was very valuable info for this issue. Thanks guys

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Windows Server 2003 introduced persistent Volume Shadow Copies and made 2003 a must-do upgrade.  Since then, it's been a must-implement feature for all servers doing any kind of file sharing.
New style of hardware planning for Microsoft Exchange server.
In this Micro Tutorial viewers will learn how to restore single file or folder from Bare Metal backup image of their system. Tutorial shows how to restore files and folders from system backup. Often it is not needed to restore entire system when onl…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

604 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question