Hyper-V HA Cluster on Windows 2008 R2 having Hyper-V issues after installing Backup Exec 2010

jkeegan123
jkeegan123 used Ask the Experts™
on
I have a 2 node Windows 2008 R2 Hyper-V High-Availability cluster running on (2) Dell T710 servers.  The cluster has (1) disk witness [quorum] and (1) Cluster shared volume [CSV].  The storage on the cluster is a Dell Equalogic PS5000 4TB SAN.  Each cluster node has (2) NICs dedicated to SAN traffic, and (6) NICs dedicated to Hyper-V VMs.

Everything has been running perfectly for about 3 months.  We have been able to Live-Migrate VMs for planned hardware maintenance, we have had nodes unexpectedly reboot due to power issues and the cluster-service take care of the issue to make it nearly transparent to all users, and in short, it's a perfect cluster.

We recently installed Backup Exec 2010 on one of the nodes by:  

1.  Failing over all resources (DISK, VMs) to NODE 2.
2.  Installing software, rebooting as necessary.
3.  Pushing the remote-agent to necessary VMs and NODE 2 from NODE 1.

After all of this, we attempted to live-migrate some VMs back to NODE 1, and the migration failed with a non-specific error event-id 21100, 21107, 22506.  Rather than a live-migration, we tried a quick migration with the VM shutdown.  This succeeded, however the VM would not start on NODE 1 with event-id 12140:

'Server-17': Failed to open attachment 'C:\ClusterStorage\Volume1\Server-17\Virtual Hard Disks\Windows 2003 x64.vhd'. Error: 'A device attached to the system is not functioning.' (0x8007001F). (Virtual machine ID 6DC7349E-D249-40A1-A576-57B7186EAABB)

and event-id 12080:

'Server-17' Microsoft Emulated IDE Controller (Instance ID {83F8638B-8DCA-4152-9EDA-2CA8B33039B4}): Failed to restore with Error 'A device attached to the system is not functioning.' (0x8007001F). (Virtual machine ID 6DC7349E-D249-40A1-A576-57B7186EAABB).

Further investigation into the Hyper-V service showed event-id 12660 being logged every 1-2 seconds:

Cannot open handle to Hyper-V storage provider.

Although this is failing live and quick migrations, we are able to pass disk resources to each node successfully (Quorum and CSV), and the cluster service appears to be healthy, aside from failing the live migrations.  Also, both nodes can access the CSV without issue without redirected access (a common status when the CSV is in a bad state).

So now we have a cluster that is unable to fail over any of the resources that we have clustered, can anyone shed some light?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®

Commented:
Your error MSG indicates that the vm is attached to the c drive. If this is the case it will not migrate. It needs to be attached to a shared stiorage volume using the san

Commented:
but i think this correct. the shared storage is mapped to c:\clusterstorage\... on both nodes, isnt it?

Author

Commented:
Yes, the Cluster Shared Volume is mapped to each node's "C" drive.
I was able to resolve this issue myself.  It seems that BackupExec casues an issue with the "FSDepends" (File System Dependencies) service after being installed.  The BackupExec services loadbefore the FSDepends service does (because it is set to manual), and because of this, it affects the FSDepends ability to allow Hyper-V services to access VHDs.

To fix this, we just had to set "FSDepends" to AUTOMATICAL instead of manual.  This process is more of a driver and not a service that is viewable in the services.msc screen, so changing it to automatic is done as follows:

1)    Start Registry Editor.
2)    Visit the following registry key: “HKLM\SYSTEM\CurrentControlSet\Services\FsDepends”
3)    Under the “FsDepends” key, change REG_DWORD value “Start” from “3” to “0”
4)    Restart the computer  

This causes the FSDepends service/process with the correct functionality to start with the OS, and BackupExec can load it’s own version of FSDepends when the backup is called, which no longer causes an issue.

Reference:

http://support.microsoft.com/kb/2013544

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial