DC freezes when running VMTools

Hi Guys,


I need to help w.r.t VMTools running on a Domain Controller to a point that the VM freezes.
The work around is to stop and disable the VMTools or stop &  set the  VMTools service to "manual".
The freeze happens not everyday, but the time stamp when it freezes is always in the morning between 10:00 - 10:30AM.
>No snapshoy_IMG_Backups
>No events that shows probable cause on VMware Taks & Events.
>Time Sync is disabled - on the VMTools options.

Thank you
Zico_VMware

What could be the probable cause?
Humphrey LegendAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Are you sure that there are no VMware Backups taking place ?

Are you on the latest version of the Host OS and VMware Tools ?
0
Humphrey LegendAuthor Commented:
Hi Andrew

We are running VMware Ver. 6.0 across our environment, hardware version of the VMs is 11.
>I will have to ask the Backup Team to check if whether the jobs are still running at the time.
Note. We do not do Image_Snapshot_Backup for DCs and VMs over 500GB of disks.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
How do you backup your DCs then ?

there are 21 different builds of ESXi 6.0 ?

which one is your build ?

What OS are the DCs ?
0
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Humphrey LegendAuthor Commented:
Hi,

It's a IT-Technical business working in silos, we have a Windows Team, Linux, Mainframe, SAN &  Backup Team and us,  VMware Team.
>I am actually a level 2 support in this instance, the Windows are doing their own investigation and me(VMware) doing our own troubleshooting and then we can compare if there is anything common from the different teams.
In a nushell the system owner(AD Team) will state what to backup, e.g. C: = Sys_State; D:Logs, E:\Data or G: Temp z:\Backup drives.

>Its a Windows 2008 R2 Enterprise, however we have a lot on Win2k12,it's kind of a non-native environment.
vCenter 6.0.0  4541947
The problem VM  is on a Cluster running only Esxi 6.0.0 3825889
Note. We have also, 6.0.0 4192238  

>>I did confirm with the Backup Team, no backup_jobs were running at the time.

The error displayed on the WindowsLog is 'A timeout (3000 milliseconds) was reached while waiting for a transaction' response from VMTools svc.'

This is more related to VMware and specific to the Tools.

So far a remove and reinstall of the tools is not a FIX.
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The error message you have is likely because the Sync Driver or VSS driver has been called, because a snapshot has been initiated either manually via a Backup call.

if you remove VMware Tools or stop and disable the service I assume the VM does not FREEZE!

VMware Tools as part of it's Snapshot Process in association with the VSS driver in the VM, STUNS the VM - this is normal.....

Freezing of the VM is another issue.

Are you sure no backup is taking place, it does seem strange this occurs at the same time each day.

So you do not backup DCs ?

Latest version of ESXi is  ESXi-6.0.0-20170202001-standard (Build 5050593) , which was released on 27 Feb 2017.

So you are 7 updates behind, (9 months ago).

Options:-

1. Update hosts from 3825889 to 5050593.
2. Update VMware Tools to current version.

OR Disable VMware Tools, or remove the VSS Driver!

see here

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007696

https://communities.vmware.com/thread/309844

https://communities.vmware.com/thread/497388?tstart=0
0
Humphrey LegendAuthor Commented:
Hi Andrew

Can I come to you about the backup, I need to chat to the AD Team if this specific VM is being backed up and how is it really backed up.
As mentioned, the AD guys are number 3 in this case, i.e. VMware, Backup, O/S Support and AD Teams.
I've been reading a couple of incidents around VSS & SYNC driver, just to get a good understanding w.r.t the VMTools. Also, I will be going thru the links attached...
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
VSS/Sync driver is called to flush/quiesce the VM (e.g. DC AD database) to make it consistent, when a Storage API call is made to the VM, when a Snapshot (Backup of the VM) is made.

This does cause the VM to be stunned, and can cause DC corruption, and /or freeze.

You can also look at the VMware.logs, e.g. vmware.log in the VM folder.
1
Humphrey LegendAuthor Commented:
Hi Andrew,

You probably wondering where did I disappear to. I will give an update next week, currently the work around was to stop VMTools, however soon after we're out of the month-end period, I will raise a CHANGE to implement your suggestions, i.e. exclude the sync & vss drivers.
>I intend to PIN the VM to hosts in our DR site since we're running some of our hosts in a stretched cluster, some in active-active clusters.

Thanks you,
Humphrey_Legend
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
vmware.log folder contents ?

are backups being created ?
0
Humphrey LegendAuthor Commented:
Hi Andrew,

>The subject server is not being backed up, however the backup is place for only 3 DCs , 1 DMZ and LAN-DC.
>I do not see anything on the log-files that relates to snapshots/backup.
>So, far the VM has freezed.

Thanks
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The issue you have is VSS, Sync driver related, when the driver is called, possibly via a snapshot, caused by a Backup trigger.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Humphrey LegendAuthor Commented:
Hi Andrew,

I 100% agree with you, I will be modifying the install of the VMTools after the month-end freeze, this must be approved by all stakeholders thru a  Change-Order....
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
no problems
0
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
do you require any additional assistance, to help close this question, and select a solution.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
VMware

From novice to tech pro — start learning today.