Andy's VMware vSphere tip#3 - Experiments in TIME. Just call me a TIME LORD!
Real life example, a client added 20 new hosts to a large vSphere Cluster, DRS was enabled, and after a few minutes, VMs started to populate the new hosts. (via VMotion/DRS)
A very large outsourcer, then suddenly found issues with Active Directory Replication across the business and also Microsoft Exchange email servers services started to fail.
Because of TIME, TIME had skewed between Active Directory servers and Microsoft Exchange servers by 1.5 hours! and all stopped replicating, and services failed! Active Directory and Microsoft Exchange failed. Logons and email stopped working for many thousands of employees!
The reason, a few of the new ESXi hosts, were syncing time from a rouge time server, which was giving the WRONG time, the time in fact was 1.5 hours ahead.
You may be thinking, ah, they are using VMware Tools to sync to host! - No!
This function has been removed from VMware Tools for many years!
BUT here is a golden nugget, did you know that a vMotion of a VM will sync time with the host unless disabled!
Time is resynchronized when you migrate the virtual machine using vMotion, take a snapshot, restore to a snapshot, shrink the virtual disk, or restart the VMware Tools service in the virtual machine (including rebooting the virtual machine).
Microsoft Data Protection Manager loses the disk (disk becomes RAW)
After New Installation and several days of backups, the DPM loses one of the disks with backed-up data. Disk becomes RAW, with all data lost. Configured hardware RAID-10, disks are 7200 rpm, 8TB capacity. What could be the issue?