DC freezes when running VMTools

Are you sure that there are no VMware Backups taking place ?

Are you on the latest version of the Host OS and VMware Tools ?

Hi Andrew

We are running VMware Ver. 6.0 across our environment, hardware version of the VMs is 11.
>I will have to ask the Backup Team to check if whether the jobs are still running at the time.
Note. We do not do Image_Snapshot_Backup for DCs and VMs over 500GB of disks.

How do you backup your DCs then ?

there are 21 different builds of ESXi 6.0 ?

which one is your build ?

What OS are the DCs ?

Hi,

It's a IT-Technical business working in silos, we have a Windows Team, Linux, Mainframe, SAN &  Backup Team and us,  VMware Team.
>I am actually a level 2 support in this instance, the Windows are doing their own investigation and me(VMware) doing our own troubleshooting and then we can compare if there is anything common from the different teams.
In a nushell the system owner(AD Team) will state what to backup, e.g. C: = Sys_State; D:Logs, E:\Data or G: Temp z:\Backup drives.

>Its a Windows 2008 R2 Enterprise, however we have a lot on Win2k12,it's kind of a non-native environment.
vCenter 6.0.0  4541947
The problem VM  is on a Cluster running only Esxi 6.0.0 3825889
Note. We have also, 6.0.0 4192238  

>>I did confirm with the Backup Team, no backup_jobs were running at the time.

The error displayed on the WindowsLog is 'A timeout (3000 milliseconds) was reached while waiting for a transaction' response from VMTools svc.'

This is more related to VMware and specific to the Tools.

So far a remove and reinstall of the tools is not a FIX.

The error message you have is likely because the Sync Driver or VSS driver has been called, because a snapshot has been initiated either manually via a Backup call.

if you remove VMware Tools or stop and disable the service I assume the VM does not FREEZE!

VMware Tools as part of it's Snapshot Process in association with the VSS driver in the VM, STUNS the VM - this is normal.....

Freezing of the VM is another issue.

Are you sure no backup is taking place, it does seem strange this occurs at the same time each day.

So you do not backup DCs ?

Latest version of ESXi is  ESXi-6.0.0-20170202001-standard (Build 5050593) , which was released on 27 Feb 2017.

So you are 7 updates behind, (9 months ago).

Options:-

1. Update hosts from 3825889 to 5050593.
2. Update VMware Tools to current version.

OR Disable VMware Tools, or remove the VSS Driver!

see here

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007696

https://communities.vmware.com/thread/309844

https://communities.vmware.com/thread/497388?tstart=0

Hi Andrew

Can I come to you about the backup, I need to chat to the AD Team if this specific VM is being backed up and how is it really backed up.
As mentioned, the AD guys are number 3 in this case, i.e. VMware, Backup, O/S Support and AD Teams.
I've been reading a couple of incidents around VSS & SYNC driver, just to get a good understanding w.r.t the VMTools. Also, I will be going thru the links attached...

VSS/Sync driver is called to flush/quiesce the VM (e.g. DC AD database) to make it consistent, when a Storage API call is made to the VM, when a Snapshot (Backup of the VM) is made.

This does cause the VM to be stunned, and can cause DC corruption, and /or freeze.

You can also look at the VMware.logs, e.g. vmware.log in the VM folder.

Hi Andrew,

You probably wondering where did I disappear to. I will give an update next week, currently the work around was to stop VMTools, however soon after we're out of the month-end period, I will raise a CHANGE to implement your suggestions, i.e. exclude the sync & vss drivers.
>I intend to PIN the VM to hosts in our DR site since we're running some of our hosts in a stretched cluster, some in active-active clusters.

Thanks you,
Humphrey_Legend

vmware.log folder contents ?

are backups being created ?

Hi Andrew,

>The subject server is not being backed up, however the backup is place for only 3 DCs , 1 DMZ and LAN-DC.
>I do not see anything on the log-files that relates to snapshots/backup.
>So, far the VM has freezed.

Thanks

Hi Andrew,

I 100% agree with you, I will be modifying the install of the VMTools after the month-end freeze, this must be approved by all stakeholders thru a  Change-Order....

no problems

do you require any additional assistance, to help close this question, and select a solution.