Avatar of Humphrey Legend
Humphrey Legend
 asked on

DC freezes when running VMTools

Hi Guys,


I need to help w.r.t VMTools running on a Domain Controller to a point that the VM freezes.
The work around is to stop and disable the VMTools or stop &  set the  VMTools service to "manual".
The freeze happens not everyday, but the time stamp when it freezes is always in the morning between 10:00 - 10:30AM.
>No snapshoy_IMG_Backups
>No events that shows probable cause on VMware Taks & Events.
>Time Sync is disabled - on the VMTools options.

Thank you
Zico_VMware

What could be the probable cause?
VMware* domain controller

Avatar of undefined
Last Comment
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

8/22/2022 - Mon
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Are you sure that there are no VMware Backups taking place ?

Are you on the latest version of the Host OS and VMware Tools ?
Humphrey Legend

ASKER
Hi Andrew

We are running VMware Ver. 6.0 across our environment, hardware version of the VMs is 11.
>I will have to ask the Backup Team to check if whether the jobs are still running at the time.
Note. We do not do Image_Snapshot_Backup for DCs and VMs over 500GB of disks.
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

How do you backup your DCs then ?

there are 21 different builds of ESXi 6.0 ?

which one is your build ?

What OS are the DCs ?
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
Humphrey Legend

ASKER
Hi,

It's a IT-Technical business working in silos, we have a Windows Team, Linux, Mainframe, SAN &  Backup Team and us,  VMware Team.
>I am actually a level 2 support in this instance, the Windows are doing their own investigation and me(VMware) doing our own troubleshooting and then we can compare if there is anything common from the different teams.
In a nushell the system owner(AD Team) will state what to backup, e.g. C: = Sys_State; D:Logs, E:\Data or G: Temp z:\Backup drives.

>Its a Windows 2008 R2 Enterprise, however we have a lot on Win2k12,it's kind of a non-native environment.
vCenter 6.0.0  4541947
The problem VM  is on a Cluster running only Esxi 6.0.0 3825889
Note. We have also, 6.0.0 4192238  

>>I did confirm with the Backup Team, no backup_jobs were running at the time.

The error displayed on the WindowsLog is 'A timeout (3000 milliseconds) was reached while waiting for a transaction' response from VMTools svc.'

This is more related to VMware and specific to the Tools.

So far a remove and reinstall of the tools is not a FIX.
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

The error message you have is likely because the Sync Driver or VSS driver has been called, because a snapshot has been initiated either manually via a Backup call.

if you remove VMware Tools or stop and disable the service I assume the VM does not FREEZE!

VMware Tools as part of it's Snapshot Process in association with the VSS driver in the VM, STUNS the VM - this is normal.....

Freezing of the VM is another issue.

Are you sure no backup is taking place, it does seem strange this occurs at the same time each day.

So you do not backup DCs ?

Latest version of ESXi is  ESXi-6.0.0-20170202001-standard (Build 5050593) , which was released on 27 Feb 2017.

So you are 7 updates behind, (9 months ago).

Options:-

1. Update hosts from 3825889 to 5050593.
2. Update VMware Tools to current version.

OR Disable VMware Tools, or remove the VSS Driver!

see here

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007696

https://communities.vmware.com/thread/309844

https://communities.vmware.com/thread/497388?tstart=0
Humphrey Legend

ASKER
Hi Andrew

Can I come to you about the backup, I need to chat to the AD Team if this specific VM is being backed up and how is it really backed up.
As mentioned, the AD guys are number 3 in this case, i.e. VMware, Backup, O/S Support and AD Teams.
I've been reading a couple of incidents around VSS & SYNC driver, just to get a good understanding w.r.t the VMTools. Also, I will be going thru the links attached...
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

VSS/Sync driver is called to flush/quiesce the VM (e.g. DC AD database) to make it consistent, when a Storage API call is made to the VM, when a Snapshot (Backup of the VM) is made.

This does cause the VM to be stunned, and can cause DC corruption, and /or freeze.

You can also look at the VMware.logs, e.g. vmware.log in the VM folder.
Humphrey Legend

ASKER
Hi Andrew,

You probably wondering where did I disappear to. I will give an update next week, currently the work around was to stop VMTools, however soon after we're out of the month-end period, I will raise a CHANGE to implement your suggestions, i.e. exclude the sync & vss drivers.
>I intend to PIN the VM to hosts in our DR site since we're running some of our hosts in a stretched cluster, some in active-active clusters.

Thanks you,
Humphrey_Legend
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

vmware.log folder contents ?

are backups being created ?
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Humphrey Legend

ASKER
Hi Andrew,

>The subject server is not being backed up, however the backup is place for only 3 DCs , 1 DMZ and LAN-DC.
>I do not see anything on the log-files that relates to snapshots/backup.
>So, far the VM has freezed.

Thanks
ASKER CERTIFIED SOLUTION
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Humphrey Legend

ASKER
Hi Andrew,

I 100% agree with you, I will be modifying the install of the VMTools after the month-end freeze, this must be approved by all stakeholders thru a  Change-Order....
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

no problems
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)

do you require any additional assistance, to help close this question, and select a solution.