Andy's VMware vSphere tip#3 - Experiments in TIME. Just call me a TIME LORD!
Real life example, a client added 20 new hosts to a large vSphere Cluster, DRS was enabled, and after a few minutes, VMs started to populate the new hosts. (via VMotion/DRS)
A very large outsourcer, then suddenly found issues with Active Directory Replication across the business and also Microsoft Exchange email servers services started to fail.
Because of TIME, TIME had skewed between Active Directory servers and Microsoft Exchange servers by 1.5 hours! and all stopped replicating, and services failed! Active Directory and Microsoft Exchange failed. Logons and email stopped working for many thousands of employees!
The reason, a few of the new ESXi hosts, were syncing time from a rouge time server, which was giving the WRONG time, the time in fact was 1.5 hours ahead.
You may be thinking, ah, they are using VMware Tools to sync to host! - No!
This function has been removed from VMware Tools for many years!
BUT here is a golden nugget, did you know that a vMotion of a VM will sync time with the host unless disabled!
Time is resynchronized when you migrate the virtual machine using vMotion, take a snapshot, restore to a snapshot, shrink the virtual disk, or restart the VMware Tools service in the virtual machine (including rebooting the virtual machine).