Urgent: VMware ESX Server Roll Backup on Reboot
Hi All,
One of our customers has physically reset (using the power button) their ESX server. Upon powering it back on, it looks to have reverted to an old snapshot (number of years ago). What is the likelihood of being able to get this back? What steps should I take to mitigate risk of losing the data since the snapshot was taken?
We do have a valid backup, but obviously booting into the valid state would be much better than having to restore the systems (2 VMs) from backup.
Also, can anyone offer any explanation to how this happened given there was no human intervention other than rebooting the server? It seems like a critical flaw in the design if the system boots into an old snapshot if the power is lot.
Thanks in advance.
One of our customers has physically reset (using the power button) their ESX server. Upon powering it back on, it looks to have reverted to an old snapshot (number of years ago). What is the likelihood of being able to get this back? What steps should I take to mitigate risk of losing the data since the snapshot was taken?
We do have a valid backup, but obviously booting into the valid state would be much better than having to restore the systems (2 VMs) from backup.
Also, can anyone offer any explanation to how this happened given there was no human intervention other than rebooting the server? It seems like a critical flaw in the design if the system boots into an old snapshot if the power is lot.
Thanks in advance.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Well that makes sense. Glad you got it figured out :)
~coolsport00
~coolsport00
On that point I've seen a few odd things.
Recently (believe it or not) a path failed on the SAN, I won't go into which SAN but it's a big boys, anyway both filers panicked and restarted..
The next stages were quite amazing.
ESX host which had plenty of memory and CPU available seemed to continue to run the guests in memory..
BUT a few guests were nasty. I had one which came back after a little work which had a different static IP than it had prior to the SAN failure..
Another which appeared to revert to a snapshot as I think it was being VCB'ed at the time.
About 20 which just hung and needed a restart.
1 which corrupted the Notes address book.
But all up that's pretty remarkable out of 400 odd guests any a few issues, and considering we had the entire shop running again in about 2 hours.
Me I was on a trip at the time and worked from the passenger seat via NextG.. recovering systems etc.
Just a short tail of Murphy's Law..
Recently (believe it or not) a path failed on the SAN, I won't go into which SAN but it's a big boys, anyway both filers panicked and restarted..
The next stages were quite amazing.
ESX host which had plenty of memory and CPU available seemed to continue to run the guests in memory..
BUT a few guests were nasty. I had one which came back after a little work which had a different static IP than it had prior to the SAN failure..
Another which appeared to revert to a snapshot as I think it was being VCB'ed at the time.
About 20 which just hung and needed a restart.
1 which corrupted the Notes address book.
But all up that's pretty remarkable out of 400 odd guests any a few issues, and considering we had the entire shop running again in about 2 hours.
Me I was on a trip at the time and worked from the passenger seat via NextG.. recovering systems etc.
Just a short tail of Murphy's Law..
I personally just stay away from snapshots as they can tend to be a nuiscance. If a VM fails...beit from update or whatever, I'll just restore it from a backup. I use Veeam to backup all my VMs. I do so a couple times during the week, so I would lose a day, but all my 'data' is on a SAN...not a VM (or on any of my physical servers either), so there really isn't a problem restoring a VM from a couple days off. Anyway...my recommendation is, if you have a backup copy of the VM (if this is what you meant), to restore that backup to your host.
Regards,
~coolsport00