Hyper-V virtual machine hangs in “Stopping” state

Hi experts,
I have a virtual machine that constantly hangs in “Stopping” state.
My configuration:
•      Dell R710 server with Windows Server 2008 R2 Hyper-V role installed
•      HVD drives are on external iSCSI disks
•      Three VMs running well: one WS2008 R2 and two WS2003
•      Integrated services are already installed on WS2008 R2
•      A newly installed WS2008 R2 virtual machine hangs on shutdown from the Hyper-V Manager console
First question is why this newly installed machine hangs?
Second question: how to kill it when it has hung?

I don’t have any clue about my first question. I will to repeat the problem with another installation on the same and on another R710 host.

About the second one: I’ve red several posts suggesting killing the vmwp.exe process of the machine but I’ve never been able to kill this process neither from the Windows Task Manager nor from an administrative command prompt (using prockill /PID xxxx /F where xxxx was the process ID). The only result that I have is that my machine enters in “Stopping-Critical” state.
Even worse, from that point (having a virtual machine hung at stopping) I am unable to manage (stop or start) any other virtual machine on the same host. The only “solution” in that case for me is to stop the Virtual Machine Management Service (vmms.exe) and to restart the physical host. Without first stopping the vmms.exe service my physical host also hangs during the restart.
Moreover, there is no any error logged in the Event Viewer.
I’ve found some other posts complaining about them problem. On all of them the only suggestion was to kill the vmwp.exe process, which obviously doesn’t work for them too.
Tried some ideas from related posts: disabling all (TCP, IP, UDP LSO) offloads into the virtual machine and configuring the host server to run in High-performance power mode; still no success.

Any suggestions, please?
LVL 20
Svet PaperovIT ManagerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

CrandellCommented:
Bios fully updated?  All firmware fully updated?  All device drivers fully updated?  Micosoft updates fully updated?  Virtual network configured correctly?  I have 2 of these boxes and 26 virtuals on one and 10 on the other and running smoothly only hiccup I had was turning of CE1 in bios to stop CPU throttling.
0
Svet PaperovIT ManagerAuthor Commented:
BIOS version: 1.2.6, Not sure about CE1 (I have to check it but it's my production server)
All device drivers are fully updated. MS updates too.
Yes, the virtual network is configured correctly. I'm using the Intel ET Quad-port adapters as virtual network adapters. They are not available to the parent partition (the check-box about chairing the network adapter with the management station is cleared). As I sad, I tried to disable the offloading into the VM without success.
I also have 2 boxes with 3 virtual machines running on each of them: one WS2008 R2 and two WS2003 without any problems. The WS2008 R2 was installed from scratch from the same media.

Another clue: the VMQ offloading is enabled on all Intel NICs (by default it is disabled).

It becomes more frustrating because I am unable to kill the wmwp.exe process even if according to the MS forums this is the only way to fully stop a VM that refuses to stop.  
0
CrandellCommented:
There is a KB article out there that addresses the CPU throttling issue and can be applied without reboot
KB975530.
0
Cloud Class® Course: Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

Svet PaperovIT ManagerAuthor Commented:
KB975530 could be helpful if you are experiencing the described stop error but I don't see how it could help me with my problem.

BTY, in one of the posts that I red on the subject, it was mentioned that this hang happens only with WS2008 R2 VMs.

Did you ever experience any freezing of your VMs?  
0
Svet PaperovIT ManagerAuthor Commented:
Another "think" that I noticed: when the virtual machine hangs in stopping state it actually doesn't free its allocated memory.  
0
CrandellCommented:
Are you using WSRM?  I have it running on the hosts (2008R2) and VM's (2008R2) with the management policy "equal_per_session".  Also power management on the NIC's off.  On vm in question have you ran CHKDSK to see if it is corrupted?  Where is your swap file and is it windows managed or have you moved it (I do) and set to 1.5X total?  What are your RAM allocations? If you total up your allocated RAM to the vm's what is left for the host?  You can oversubscribe the CPU's but you can't the RAM. On the host you need a minimum of 2GB for the kernel, 2GB for the OS and at least 1GB for each application including HV.  These are minimums I use more.  I would think you are running out of resources and causing these issues.
0
Svet PaperovIT ManagerAuthor Commented:
Hi,
I am not using WSRM. The Power Management of all NICs is OFF. The RAM is OK, I have 24GB on each of the physical servers with currently only 7GB allocated to all VMs. I kept the swap file (on the parent partition) relatively small 4G (following the best practices) since it is not used by the the virtual machines.  

I did another test: I have installed a new WS2008 R2 VM from scratch on my second R710 server. It did run well until the moment I have joined it to my domain.

I need more clues, what am I missing? As I sad I have two WS2008 R2 VMs running without any problems and I am definitely not running out of resources - there is enough RAM and enough disk space.

Thanks
0
CrandellCommented:
Your swap file should be 36GB BP is 1.5X total.
0
Svet PaperovIT ManagerAuthor Commented:
Why? What is the purpose of having a huge page file on the parent partition when the virtual machines don't use it (each VM creates its own within its partition). The only valid reason to have a page file matching the actual physical RAM would be if we want to analyze the dump data in case of BSoD which I don't need.

Again, my old VMs are not affected by the problem.
0
CrandellCommented:
The host does
0
Svet PaperovIT ManagerAuthor Commented:
Hi,
Having a huge page file in the parent partition really doesn't make sens. Check this post for more info: http://social.technet.microsoft.com/Forums/en/virtualmachinemanager/thread/a86a742b-0f0a-4032-a9ac-c4d3ef06daeb/

Nevertheless, just to be sure of that I made my page file 36GB, restarted the physical server and after 2-3 restarts of the virtual machine it hung again, so, the swap file doesn't help.

Meanwhile, I was able to narrow the problem using Process Manager from SysInternals:
 - as I noticed in my initial post, when the VM hangs I am unable to kill its vmwp.exe process
- using Process Manager I was able to kill all threads of this process except for one: msvcrt.dll.

Thanks
0
Svet PaperovIT ManagerAuthor Commented:
Hello,

After updating Dell MD3000i multi-path driver my servers are running smoothly now. I don’t know how this was related but the problem seems disappeared.

Thanks
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Svet PaperovIT ManagerAuthor Commented:
Since my last comment on this issue we do not experience the problem anymore, so I am closing the question.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Server OS

From novice to tech pro — start learning today.