Hyper-V VMs lose connectivity after live migrating

Hello,
i have a four node Hyper-V Server 2008 R2 cluster.
each node has 8 NICs
2 have been configured for iSCSI access.
1 for Live migration
1 for CSV
1 for LAN
2 for Hyper-V virtual networks
1 unassigned

the cluster was validated and built without issue
after creating the VMs they were added as a cluster service without issue

however, when i live migrate from one node to another the VMs lose connectivity altogether. sometimes i need to live migrate them back to the original host or to another in the 4 node cluster.

can someone help me?

thanks,
netrescueAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

James HaywoodCommented:
Are all your hosts setup identically with regards to virtual nics, names etc?
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
is it just one host which is causing this issue?

check you VM networking, on the host you have live migrated to.
0
netrescueAuthor Commented:
yes
all NICs are named the same
iSCSI - 1
iSCSI - 2
Hyper-V-1
Hyper-V-2
LAN
Private CSV
Private LMN

those are the names of the NICs on all servers.
all the respective private networks can see all other interfaces on all nodes.
all servers were built to same spec, OS and patch level and built all by myself at once.
0
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

netrescueAuthor Commented:
no it isnt limited to any one host.
different VMs are hosted and communicate well on all 4 hosts. there are no preferred hosts set. only when i live migrate there is an issue.
this morning i live migrated a VM from host 1 to 2 and it didnt work. i tried live migrating to host 3. still nothing. then moved it to 4, it worked.
thing is, there are VMs on all the hosts. if i live migrate from one host to the other, its a roll of the dice on whether it will work or if i need to live migrate it to another.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
is it ALL hosts? or just one host?
0
netrescueAuthor Commented:
will sending you guys screenshots of the NICs details help?
i can send you the adapter settings for each host and the names etc....
0
James HaywoodCommented:
Send the details please.

One weird thing I have experienced before is if the firewall is switched on in the VM that can cause this issue.
0
netrescueAuthor Commented:
hello
see the info for the 4 hosts NICs below
hope it displays well.
if you have trouble interpreting let me know.

i confirm no VMs firewalls are enabled. neither are there any on the hosts enabled.

HOST 1

Interface Name      Device                  IP

Hyper-V-1            Intel Gigabit ET Quat port Adapter      blank
Hyper-V-2            Intel Gigabit ET Quat port Adapter      blank
LAN            Intel Gigabit ET Quat port Adapter      172.20.2.10
Unplugged            Intel Gigabit ET Quat port Adapter      -
iSCSI – 1            Broadcom NetXtreme II GigE      192.168.130.10
iSCSI – 2            Broadcom NetXtreme II GigE      192.168.130.11
Private CSV            Broadcom NetXtreme II GigE      10.10.10.1
Private LMN            Broadcom NetXtreme II GigE      10.10.11.1
Local Area Connection      Hyper-V-1 Virtual Network            172.20.3.1
Local Area Connection      Hyper-V-2 Virtual Network            172.20.3.2

HOST 2

Interface Name            Device            IP

Hyper-V-1            Intel Gigabit ET Quat port Adapter      blank
Hyper-V-2            Intel Gigabit ET Quat port Adapter      blank
LAN            Intel Gigabit ET Quat port Adapter      172.20.2.11
Unplugged            Intel Gigabit ET Quat port Adapter      -
iSCSI – 1            Broadcom NetXtreme II GigE      192.168.130.12
iSCSI – 2            Broadcom NetXtreme II GigE      192.168.130.13
Private CSV            Broadcom NetXtreme II GigE      10.10.10.2
Private LMN            Broadcom NetXtreme II GigE      10.10.11.2
Local Area Connection      Hyper-V-1 Virtual Network            172.20.3.3
Local Area Connection      Hyper-V-2 Virtual Network            172.20.3.4

HOST 3

Interface Name            Device            IP

Hyper-V-1            Intel Gigabit ET Quat port Adapter      blank
Hyper-V-2            Intel Gigabit ET Quat port Adapter      blank
LAN            Intel Gigabit ET Quat port Adapter      172.20.2.12
Unplugged            Intel Gigabit ET Quat port Adapter      -
iSCSI – 1            Broadcom NetXtreme II GigE      192.168.130.14
iSCSI – 2            Broadcom NetXtreme II GigE      192.168.130.15
Private CSV            Broadcom NetXtreme II GigE      10.10.10.3
Private LMN            Broadcom NetXtreme II GigE      10.10.11.3
Local Area Connection      Hyper-V-1 Virtual Network            172.20.3.5
Local Area Connection      Hyper-V-2 Virtual Network            172.20.3.6

HOST 4

Interface Name            Device            IP

Hyper-V-1            Intel Gigabit ET Quat port Adapter      blank
Hyper-V-2            Intel Gigabit ET Quat port Adapter      blank
LAN            Intel Gigabit ET Quat port Adapter      172.20.2.13
Unplugged            Intel Gigabit ET Quat port Adapter      -
iSCSI – 1            Broadcom NetXtreme II GigE      192.168.130.16
iSCSI – 2            Broadcom NetXtreme II GigE      192.168.130.17
Private CSV            Broadcom NetXtreme II GigE      10.10.10.4
Private LMN            Broadcom NetXtreme II GigE      10.10.11.4
Local Area Connection      Hyper-V-1 Virtual Network            172.20.3.7
Local Area Connection      Hyper-V-2 Virtual Network            172.20.3.8
0
saqibfareedCommented:
Hi,

Can you please share more information about your setup
0
netrescueAuthor Commented:
Hi Saqib,
can you be specific. what information would you like. i really would like to provide you with any and all information necessary. but i need you to identify what information in particular you would need.
thanks,
0
netrescueAuthor Commented:
hello again
is there anyone with any knowledge of how i can resolve this issue.
please assist.
if you require further information, please do let me know specifically what you need.
0
Steve BinkCommented:
When you lose connectivity, try one of the following:

1) Disable/re-enable the NIC on the VM through the console connection, or
2) "Unplug" the NIC from the VM by setting its connected property in "Settings" to "Not Connected", then reconnect it to the proper network.  Make note of the other settings before disconnecting it - they may be reset to defaults, or
3) Reboot the VM.

I've noticed this problem on a great many VMs in our Hyper-V farm.  It is really annoying during updates, since this usually manifests after a reboot.  About half of our machines are P2V servers, but this issue is not limited to that set.  
0
James HaywoodCommented:
Are the driver versions the same on each host?

From what I can see you are only using two NICs for your VMs is this right?

Are there any error messages on either of the host when you are trying to migrate?
0
netrescueAuthor Commented:
@ routinet -
disabling / reenabling the NIC from within Windows or using Hyper-V Manager disconnecting it and reconnecting it or even switching the Hyper-V networks but that does not work.
this morning i experienced a problem with connectivity on one VM and a reboot solved the problem.

@ hhaywood000 -
i do have 2 NICs on each host designated and named identically for VM networks and i have two Hyper-V networks created in hyper-v manager
NO - The driver versions are not identical upon further investigations. two hosts have identical versions and the other two have different versions.
could this be contributing to the problem? i will schedule updates and maintanence tonight on those hosts.
0
PowerToTheUsersCommented:
Maybe these hotfixes are relevant in your situation:
http://support.microsoft.com/?id=2263829
http://support.microsoft.com/?id=2517329
0
James HaywoodCommented:
As a follow up, what antivirus are you using in the VMs?
0
msmamjiCommented:
If you have antivirus of some kind on the Hosts try adding the C:\ClusterStorage as an exception.
0
netrescueAuthor Commented:
hi
im using trend micro server protect on the VMs
i will definitlely include the C:\ClusterStorage folders on the hosts as exceptions

just FYI, a few of these VMs have been P2V if that bares any consequence.

on the matter of updating the drivers, it looks like i need to completely remove the existing drivers and reinstall a single version on all hosts and NICs as the driver udpate or installation isnt working and saying the latest driver is already installed.

any ideas before i make that step in removing and reinstalling a single driver version on all NICs on what is the best way to get all NICs to have same driver version?
0
msmamjiCommented:
I would still advise you to do the antivirus exception thing first.
http://support.microsoft.com/kb/961804

I had the same issue at one of my clients, forgot to exclude the exceptions on it. Once added it worked perfectly.

Regards,
Shahid
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Server OS

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.