Andy's VMware vSphere tip#3 - Experiments in TIME. Just call me a TIME LORD!

Real life example, a client added 20 new hosts to a large vSphere Cluster, DRS was enabled, and after a few minutes, VMs started to populate the new hosts. (via VMotion/DRS)

A very large outsourcer, then suddenly found issues with Active Directory Replication across the business and also Microsoft Exchange email servers services started to fail.

Because of TIME, TIME had skewed between Active Directory servers and Microsoft Exchange servers by 1.5 hours! and all stopped replicating, and services failed! Active Directory and Microsoft Exchange failed. Logons and email stopped working for many thousands of employees!

The reason, a few of the new ESXi hosts, were syncing time from a rouge time server, which was giving the WRONG time, the time in fact was 1.5 hours ahead.

You may be thinking, ah, they are using VMware Tools to sync to host! - No!

This function has been removed from VMware Tools for many years!

BUT here is a golden nugget, did you know that a vMotion of a VM will sync time with the host unless disabled!

Time is resynchronized when you migrate the virtual machine using vMotion, take a snapshot, restore to a snapshot, shrink the virtual disk, or restart the VMware Tools service in the virtual machine (including rebooting the virtual machine).

Source
https://kb.vmware.com/s/article/1189

https://kb.vmware.com/s/article/1318

So please make sure your VMware vSphere ESXi Hosts are synced to a good time server!

(or disable time sync above!)

and here is a Stratum 1 Time Server based on a Raspberry Pi, using GPS and PPS - I built earlier!


If you want to discuss this post, and have questions about NTP, please post a question to the VMware topic area.