Hyper-V Machines in Stopping State

David Johnson, CD
David Johnson, CD used Ask the Experts™
on
Hyper-V Machines in Stopping State?
It's not because of Hard drive space 472 GB Free of 1.09TB
30 VM's of which 14 are in stopping state..

Server 2016 Datacenter
HP DL380 G6
96 GB Ram
2x e5649  6 core CPU's2018-12-30_20-14-31.png
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Tom CieslikIT Engineer
Distinguished Expert 2017

Commented:
Hi David.
You've said you have 30 VM's right ?
Your Host has 2 processors 6 core each,,, so 2x6x2 = 24 threads (Virtual Processors)
Microsoft VM Best Practice rule is to not go beyond virtual processor limit - 2 vp for host.
Actually mathematical  definition is  maximum VP - 2 for Host OS.
In your case: IF YOU DID ASSIGNED ONLY ONE VIRTUAL PROCESSOR (THREAD) PER VM, YOU ARE ALLOWED TO HAVE:  24 -2 =22 VM's
If for some reason few of your VM has more than one VP assigned, you have a bigger issue here.
Maybe this is your problem,,, you have too many VM's on single Host for this Hardware configuration !!!
Philip ElderTechnical Architect - HA/Compute/Storage

Commented:
On the host bring up ResMon.EXE and head to the Disk tab. Does Disk Activity show any VHDX files being worked on?
Shaun VermaakTechnical Specialist
Awarded 2017
Distinguished Expert 2018

Commented:
What do the event logs say?
OWASP: Threats Fundamentals

Learn the top ten threats that are present in modern web-application development and how to protect your business from them.

Top Expert 2016

Author

Commented:
1 Vcpu per machine..  Disk and cpu contention would only make things run super slow.


I had to actually restart the host and the vm's restarted and then there was a bunch of altaro temporary snapshots being removed.

There was an Altaro backup running at the time.  one lab has altaro and another has veeam B&R
IT Engineer
Distinguished Expert 2017
Commented:
OK, but still,,,you have more VM's than available Vcpu's
You should not because strange thinks can happen.
Top Expert 2016

Author

Commented:
I will split the vms and use them on the replica servers to see if the vcpu's are the problem
Philip ElderTechnical Architect - HA/Compute/Storage

Commented:
If there were Altaro snapshots happening at the time then that's the source of the "issue". That would have become readily apparent as well with the Disk Activity check.

What is the storage subsystem setup?

Break the Altaro backup schedules up to stagger the VM's backups. Don't do them all at once. A VSS snapshot is extremely I/O intensive thus the stalls.

EDIT: As an FYI:
1: CPU is very rarely the bottleneck. The storage subsystem is almost always the place to start with storage to compute fabric being the next step. Since this is a standalone host, storage is it.
2: A VM should have at least two vCPUs assigned. If there's a runaway process on the VM with only 1 vCPU assigned the VM is toast to access.
3: PerfMon has both host and in-guest counters. Use those to verify where all of the host's systems are at relative to load.
Top Expert 2016

Author

Commented:
The problem is that there was 0 disk activity and it is set to do 2 simultaneous backups down from the default of 4
Philip ElderTechnical Architect - HA/Compute/Storage

Commented:
The catch for me is that we've run into Altaro and VSS snapshot issues in the past. I'm still leaning towards that being the source of the problem.

vssadmin can be used to delete all snapshots. That's one of the troubleshooting steps that I suggest taking.
vssadmin delete shadows /all

Open in new window

Top Expert 2016

Author

Commented:
I have been using Veeam B&R and given the hype about altaro being equal but easier to configure I thought I'd give it a shot. Didn't notice the problem until I checked my mail and saw all of the server has been down for more than xx minutes from spiceworks.  Went into Hyper-V Manager and found they were all in stopping mode..  Get-Vm hung up ... Since this is not production stuff.. Did a Shutdown/restart

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial