Hyper-V VM will not shut off, turn off, or Migrate

It is now show as a shutdown-critical and I can not get it to shut down. All the other VM's are working but I did have to restart the host1 because machines were doing the same thing. What would cause this to happen. I am running a 2 host cluster on Server 2008 R2 and it seems every couple weeks I have an issue. But nothing shows up in the logs. Any help would be great!
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

BillSystems AdministratorCommented:
Have you thought about installing SP1 for Windows 2008 R2 on your host servers?  It has many fixes to Hyper-V and includes some excellent new features like thin RAM provisioning.

Further questions:
Are the integration services installed properly?
Are the event viewer logs on the host and guest VMs showing anything?
Is your cluster configured correctly?  Since you only have 2 physical servers are you using a shared disk as the quorum?
Kelly-BradyAuthor Commented:
1 Yes I have S1 installed on all servers
2 I will have to checck on the integration but I am almost certain that they are installed
3 nothing is showing up in the event viewer
4 Yes the cluster is configured correctly and I am using a quorom disk.
5 The only oddity here is that I am using Virsto software that allows for IO offloading though ISCI and I am starting to think that it is too finicky. This software also allows for better use of the SAN storage but not sure if it is worth it anymore.
BillSystems AdministratorCommented:
Virsto advertizes some tremendous efficiencies in dealing with the "IO blender" that comes from the more random IO that comes from multiple VMs.  I haven't worked with it personally.  It does add a significant layer of complexity.  If you have support you may be able to open up a ticket to have them help you troubleshoot your issue.  In our experience we've been able to pack dozens of VMs on a Hyper-V host with W2008 R2 SP1 and a SAN without significant IO issues.  This includes multiple SQL servers.  However some heavy applications using SQL, Exchange, etc. may require some extra attention:  a dedicated iSCSI NIC(s), MPIO, jumbo frames, a dedicated LUN, a dedicated controller on your SAN, splitting your IO on different SAN disk spindles, etc.

I'd suggest contacting Virsto and running your issue past them for advice.

Another option is to open up a MS Support Ticket.  They cost $265 per incident and they work with you for as long as it takes to resolve the issue.  I've used their service many times.  The last time they invested about 10 hours on a Remote Desktop / Terminal Server Farm and they did resolve the issue, which was rather complex.  When you really need to get to the bottom of things MS Support is a good option to consider.

Microsoft Paid Technical Support      800-936-5800

I suppose one of the logical approaches is to verify/troubleshoot each component in your environment individually and as a complete system:
1- Physical server config including NIC configs
2- Hyper-V config
3- Hyper-V integration components
4- Cluster config
5- iSCSI config
6- Visto setup

Another option might be to export/import the VMs in question to another Hyper-V host temporarily, or migrate them live via System Center, or migrate them with a VB script.  I have a script saved to do a live VM migration but I'd have to take a few min to find it.  That would take Virsto out of the equation for troubleshooting purposes.
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

Kelly-BradyAuthor Commented:
That would be great and I am going to work on Exporting them out and reloading the Hyper V servers and not renewing the Virsto Software. I think that it is screwing with the services. I currently only have 8 VM's on a two node cluster and each host has 64GB of Ram, so I think I may be over complicating the needs of the environment.
Kelly-BradyAuthor Commented:
The question I guess would be can I move them all onto one host, redo the second one. Import them to the newly configured host, redo the 1st one then create the cluster? Or does all VM's need to be off in order for the cluster to be created?
BillSystems AdministratorCommented:
Only 8 VM's on a two node cluster.  Unless those VMs are mega busy with SQL, Exchange, etc. you shouldn't have performance issues.  It might be overkill, and especially with Virsto.

Also, does it need to be clustered?  What are you clustering, only the hosts?

I've setup clusters in a lab environment without a reboot, but not in production.  If so you may be able to keep the VMs running while this happens.

Here's a simple script (I haven't tested it personally) that can move live VMs to another host.  Look near the bottom.

Questions to consider:
? Does Virsto HAVE to intercept ALL disk IO?  Perhaps you can create a new LUN (without Virsto) and move things over without having to redo several things, perhaps including the cluster.
? Would it be worth a phone call to the Virsto people to ask them what's wrong?  They appear to be a reputable company, hopefully with good tech support.

It looks like the critical steps you will need to get through are (not necessarily in order):
1- Put the VMs somewhere safe while you mess with the cluster
2- Break the cluster and redo one of the hosts (without Virsto)
3- Perhaps move the VMs onto the new host
4- Reconfig the other host
5- Add the other host into the new cluster

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
BillSystems AdministratorCommented:
>1- Put the VMs somewhere safe while you mess with the cluster

I mean RUN them somewhere safe.  That could actually be any hyper-v server(s) with sufficient resources.
Kelly-BradyAuthor Commented:
The servers are not SQl or Exchange but the reason for the cluster is high availability, I want to make sure that they will be up at all times. I have talked to Virsto in the past and I do believe in there product but I think that it was a little much for us. I do not have the time to concentrate just on that and the only issue I have is that you need to have a separate server that acts as a Live space. Then you have to find a Storage program out there that does ISCSI connections so that adds a separate piece to the puzzle. It would be a better solution if they had the ISCSI program to go along with there software and that way it support would be in one place. I think the issue I am having is one that is with the ISCSI connection and since it is not their program they can not help and it is a free program. I could pay for a separate program but I already pay enough for the Virsto that I feel all the components should be included with it.
BillSystems AdministratorCommented:
FYI, if you do need to build an inexpensive SAN Microsoft has decided to make their storage server iSCSI software available for free.  It lets you turn any Windows server into an iSCSI SAN.  We use an EMC SAN but I thought you might find this interesting.  An advantage of an end to end Microsoft solution is that Microsoft Support will cover all of the pieces.

Microsoft iSCSI Software Target

The Microsoft iSCSI Software Target 3.3 provides storage (disks) over a TCP/IP network. It turns a computer running Windows Server into a storage device which provides shared block storage. You can use Microsoft iSCSI Software Target 3.3 to perform a variety of storage-related tasks, including the following:

•Provide shared storage for Hyper-V to enable high availability and live migration
•Consolidate storage for multiple application servers (i.e. Microsoft SQL Server or Hyper-V)
•Provide shared storage for applications hosted on a Windows failover cluster
•Enable diskless computers to boot remotely from a single operating system image using iSCSI
Kelly-BradyAuthor Commented:
That would be better then what I am currently using, thank you for the info.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.