DC freezes when running VMTools

Humphrey Legend
Humphrey Legend used Ask the Experts™
on
Hi Guys,


I need to help w.r.t VMTools running on a Domain Controller to a point that the VM freezes.
The work around is to stop and disable the VMTools or stop &  set the  VMTools service to "manual".
The freeze happens not everyday, but the time stamp when it freezes is always in the morning between 10:00 - 10:30AM.
>No snapshoy_IMG_Backups
>No events that shows probable cause on VMware Taks & Events.
>Time Sync is disabled - on the VMTools options.

Thank you
Zico_VMware

What could be the probable cause?
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Are you sure that there are no VMware Backups taking place ?

Are you on the latest version of the Host OS and VMware Tools ?

Author

Commented:
Hi Andrew

We are running VMware Ver. 6.0 across our environment, hardware version of the VMs is 11.
>I will have to ask the Backup Team to check if whether the jobs are still running at the time.
Note. We do not do Image_Snapshot_Backup for DCs and VMs over 500GB of disks.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
How do you backup your DCs then ?

there are 21 different builds of ESXi 6.0 ?

which one is your build ?

What OS are the DCs ?
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
Hi,

It's a IT-Technical business working in silos, we have a Windows Team, Linux, Mainframe, SAN &  Backup Team and us,  VMware Team.
>I am actually a level 2 support in this instance, the Windows are doing their own investigation and me(VMware) doing our own troubleshooting and then we can compare if there is anything common from the different teams.
In a nushell the system owner(AD Team) will state what to backup, e.g. C: = Sys_State; D:Logs, E:\Data or G: Temp z:\Backup drives.

>Its a Windows 2008 R2 Enterprise, however we have a lot on Win2k12,it's kind of a non-native environment.
vCenter 6.0.0  4541947
The problem VM  is on a Cluster running only Esxi 6.0.0 3825889
Note. We have also, 6.0.0 4192238  

>>I did confirm with the Backup Team, no backup_jobs were running at the time.

The error displayed on the WindowsLog is 'A timeout (3000 milliseconds) was reached while waiting for a transaction' response from VMTools svc.'

This is more related to VMware and specific to the Tools.

So far a remove and reinstall of the tools is not a FIX.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
The error message you have is likely because the Sync Driver or VSS driver has been called, because a snapshot has been initiated either manually via a Backup call.

if you remove VMware Tools or stop and disable the service I assume the VM does not FREEZE!

VMware Tools as part of it's Snapshot Process in association with the VSS driver in the VM, STUNS the VM - this is normal.....

Freezing of the VM is another issue.

Are you sure no backup is taking place, it does seem strange this occurs at the same time each day.

So you do not backup DCs ?

Latest version of ESXi is  ESXi-6.0.0-20170202001-standard (Build 5050593) , which was released on 27 Feb 2017.

So you are 7 updates behind, (9 months ago).

Options:-

1. Update hosts from 3825889 to 5050593.
2. Update VMware Tools to current version.

OR Disable VMware Tools, or remove the VSS Driver!

see here

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007696

https://communities.vmware.com/thread/309844

https://communities.vmware.com/thread/497388?tstart=0

Author

Commented:
Hi Andrew

Can I come to you about the backup, I need to chat to the AD Team if this specific VM is being backed up and how is it really backed up.
As mentioned, the AD guys are number 3 in this case, i.e. VMware, Backup, O/S Support and AD Teams.
I've been reading a couple of incidents around VSS & SYNC driver, just to get a good understanding w.r.t the VMTools. Also, I will be going thru the links attached...
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
VSS/Sync driver is called to flush/quiesce the VM (e.g. DC AD database) to make it consistent, when a Storage API call is made to the VM, when a Snapshot (Backup of the VM) is made.

This does cause the VM to be stunned, and can cause DC corruption, and /or freeze.

You can also look at the VMware.logs, e.g. vmware.log in the VM folder.

Author

Commented:
Hi Andrew,

You probably wondering where did I disappear to. I will give an update next week, currently the work around was to stop VMTools, however soon after we're out of the month-end period, I will raise a CHANGE to implement your suggestions, i.e. exclude the sync & vss drivers.
>I intend to PIN the VM to hosts in our DR site since we're running some of our hosts in a stretched cluster, some in active-active clusters.

Thanks you,
Humphrey_Legend
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
vmware.log folder contents ?

are backups being created ?

Author

Commented:
Hi Andrew,

>The subject server is not being backed up, however the backup is place for only 3 DCs , 1 DMZ and LAN-DC.
>I do not see anything on the log-files that relates to snapshots/backup.
>So, far the VM has freezed.

Thanks
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
The issue you have is VSS, Sync driver related, when the driver is called, possibly via a snapshot, caused by a Backup trigger.

Author

Commented:
Hi Andrew,

I 100% agree with you, I will be modifying the install of the VMTools after the month-end freeze, this must be approved by all stakeholders thru a  Change-Order....
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
no problems
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
do you require any additional assistance, to help close this question, and select a solution.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial