<

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x

HOW TO: VMware Snapshots :- Be Patient

Published on
196,740 Points
48,240 Views
60 Endorsements
Last Modified:
Approved
Community Pick
Andrew Hancock (VMware vExpert / EE MVE^2)
MVE^2, Expert of the Year 2017-2011, Scribe 2016-2012, Author of the Year 2017-6,2013-2012 VMware vExpert 2018-2011 27 years of experience.
VMware Virtual Machine Snapshots

Please just let me open and state, if you find yourself with a snapshot issue, and are struggling, please post a question to get an Expert to help you resolve the situation, otherwise you risk data loss. Finally, Snapshots are evil, and one of the most popular questions in the VMware Zone on Experts Exchange.

A snapshot is NOT a backup of a VM; that is a gross misconception.

A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes.  When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file.  There is an active link between the original VMDK file and the new delta file.  Anything that gets written to the VM actually gets written to the delta file.   The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM.  If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.  

VMware Snapshots are really designed for patching a vritual machine, testing and then rollback if the patch did not function. It is not recommended to leave a virtual machine running on a snapshot vritual disk

1. performance when using a snapshotted disk is worse than normal
2. if the snapshot delta virtual disk runs out of datastore storage space, the virtual machine will fail.

How do I know if I have a snapshot?

1. Check the Virtual Machine, Right Click the Virtual Machine, Click Snapshot

Virtual Machine Snapshot2. Are they any Snapshots listed

Snapshots Listedin the example above, there is a snapshot listed XenDeploy#1

It is very common, that the snapshot is missing from the snapshot manager window, but the virtual machine is still running on a snapshot, and the sure way to tell, is to look at the contents of the vm folder

Virtual Machine snapshotin the above example, the pink rectangle highlights the parent disk, the red rectangle highlights the current snapshot delta disk.

if you see a virtual machine disk with a -00000x.vmdk it is likely this is a snapshot delta disk. This is the active disk which is being written to, the parent disk is no longer at present being written to. Final confirmation can be confirmed by checking the virtual machine disk properties of the VM.

Virtual Machine Snapshotthe above example, the virtual machine disk can clearly be seen as Windows-7-000001.vmdk, this is a snapshot disk.

I often here from many VMware Admins, but I do not use snapshots, so how did it get there?

BUT, most backup applications, VMware vDR, Veeam Backup and Replication and many more, use the Snapshot API function to backup virtual machines, and can leave a VM running on a snapshot. So check your datastores, setup alerting to advise of snapshots in vCenter Server, use Nagios, vCLI, PowerCLI, many how brew scripts on the market to catch these evil snapshots before its too late.

If you find yourself in a situation with a Snapshot, and you have started the Deletion process, Be Patient, it can take minutes, hours or days to complete, and appear to hang at 95% or 99% with no feedback from vSphere Client.

We have waited 3.5 days for a snapshot to complete, which depends on how large the snapshot, and how fast the datastore the virtual machine is stored on.

So Be Patient, do not fiddle, cancel the process or restart the host, it will complete eventually. Try and Delete snapshots with the virtual machine powered off.

Just to confirm this also checkout this EE Question, which will confirm to be patient!

ESXi snapshot deletion stuck at 99%

I would highly recommend reading these two articles on VMware Virtual Machine Snapshots at the VMware Knowledgebase

Understanding Snapshots  
Snaphot Best Practices

Also check out the following Snapshot Articles by fellow VMware vExpert Eric Siebert

Part 1 : How VMware snapshots work
Part 2 : Deleting virtual machine snapshots without wasting disk space
Part 3 : Troubleshooting VMware snapshots

*******************************************************************************************************************************************
Thank you for reading my article, please leave valuable feedback. If you liked my VMware article and would like to see more VMware Articles from me, please click the Yes button near the: Was this article helpful? at the Top of this article to the right of the Article title. Thank You.
*******************************************************************************************************************************************
60
18 Comments
LVL 38

Expert Comment

by:younghv
Another good example for linking in VM Zone questions.

"Yes" vote above.
0

Expert Comment

by:Comptx
can you turn on a vm while a snapshot is being removed?
0
LVL 130

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
No, this really is a question!
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

LVL 3

Expert Comment

by:Marc Smets
Hit my head against the wall because I too fell in the trap of A snapshot is NOT a backup of a VM; that is a gross misconception.

Very good article, helped me alot in understanding how snapshots work.
0
LVL 9

Expert Comment

by:Zacharia Kurian
Andrew, you should combine your great VM articles into a book and get it published. :)

Great real time solutions from you all the time.

Zac
0
LVL 130

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Zac

Thanks for your kind words!

That is actually a work in progress!!!!

The book is delayed though!

Andrew
2
LVL 5

Expert Comment

by:onlinerack
Andrew, great article and terrific style.

Keep up the good work (voted YES).
0
LVL 130

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
@onlinerack Thanks
0
LVL 11

Expert Comment

by:Senior IT System Engineer
Hi Andrew,

Is it possible to expedite the VM snapshot by turning off the VM ?
0
LVL 58

Expert Comment

by:Pete Long
Great Article - Well Written.

Pete
0
LVL 130

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
@Pete Long

Thanks Pete, this was written a long time ago, and is on my list to update a little bit, now 6.5 is out!

Starting that on all my articles at present.

Andy
0

Expert Comment

by:Denver IT Manager
Hey everyone (ok mostly Andrew, haha).  I've recently come onboard at my current company as their IT Manager and have never once seen Disk Consolidation is needed in my 12+ years as an IT Professional...until now.  

Due to some very poor decisions by the previous IT Manager, we have a VM that has been locking up during Veeam 12-hour backups as the utility attempts to delete the snapshot it created for the backup.  I've been on the phone with VMware a total of 7-8 hours trying to figure out the issue and they finally isolated it to a couple of rogue snapshot vmdk files that were never removed (dating back to October of last year).  

Below are the details, with some modifications to file names and such to protect our identity.  :)
VMware thinks the main issue is the "713.4G Oct 18 20:44 ProblemVM-000001-sesparse.vmdk" on the first datastore listed below.  They said it appears someone manually diverted the delta file so it could not be written to the base disk.

VMware ESXi 6.5.0 build-4564106
VMware ESXi 6.5.0 GA

VM : ProblemVM.domain.com

[CML Datastore 02] ProblemVM.domain.com/ProblemVM.domain.com.vmdk   - 558GB
[Host VMData] ProblemVM.domain.com_1-000001.vmdk                - 3.2TB
[CML Datastore 07_ProblemVM] ProblemVM/ProblemVM-000008.vmdk           - 3.8TB



./vmfs/volumes/5661e1ef-3974d885-1200-.../ProblemVM.domain.com

scsi0:0.fileName = "ProblemVM.domain.com.vmdk"

[root@Host:/vmfs/volumes/5661e1ef-3974d885-1200-.../ProblemVM.domain.com ls -lah   =====>> CML Datastore 02
total 1340888080
drwxr-xr-x    1 root     root        3.4K Feb  5 18:38 .
drwxr-xr-t    1 root     root        1.9K Dec 17 02:19 ..
-rw-------    1 root     root      713.4G Oct 18 20:44 ProblemVM-000001-sesparse.vmdk
-rw-------    1 root     root         336 Oct 18 20:44 ProblemVM-000001.vmdk
-rw-r--r--    1 root     root          93 Apr  9  2017 ProblemVM.domain.com-a91b7947.hlog
-rw-------    1 root     root       16.0G Feb  5 18:38 ProblemVM.domain.com-a91b7947.vswp
-rw-r--r--    1 root     root          13 Jan 30 15:31 ProblemVM.domain.com-aux.xml
-rw-------    1 root     root        4.4M Feb  5 18:38 ProblemVM.domain.com-ctk.vmdk
-rw-------    1 root     root      558.4G Feb  5 18:21 ProblemVM.domain.com-flat.vmdk
-rw-------    1 root     root        8.5K Feb  5 18:21 ProblemVM.domain.com.nvram
-rw-------    1 root     root         734 Feb  5 18:21 ProblemVM.domain.com.vmdk
-rw-r--r--    1 root     root          79 Feb  5 18:21 ProblemVM.domain.com.vmsd
-rwx------    1 root     root        3.5K Feb  5 18:21 ProblemVM.domain.com.vmx
-rw-------    1 root     root           0 Feb  3 23:05 ProblemVM.domain.com.vmx.lck
-rw-------    1 root     root        3.1K Jan 31 00:54 ProblemVM.domain.com.vmxf
-rwx------    1 root     root           0 Feb  5 18:38 ProblemVM.domain.com.vmx~
-rw-r--r--    1 root     root       26.6M Sep  3 20:05 vmware-36.log
-rw-r--r--    1 root     root        6.0M Oct 18 03:16 vmware-37.log
-rw-r--r--    1 root     root      389.9K Oct 18 16:50 vmware-38.log
-rw-r--r--    1 root     root        8.1M Jan 13 22:57 vmware-39.log
-rw-------    1 root     root        2.7M Feb  3 22:22 vmware-40.log
-rw-------    1 root     root      546.4K Feb  5 18:35 vmware-41.log
-rw-------    1 root     root       75.5K Feb  5 18:38 vmware.log
-rw-------    1 root     root      110.0M Feb  5 18:38 vmx-ProblemVM.domain.com-2837150023-1.vswp
-rw-------    1 root     root      110.0M Jun  8  2017 vmx-ProblemVM.domain.com-2837150023-2.vswp


scsi0:2.fileName = "/vmfs/volumes/55786583-839f85d1-6570-.../ProblemVM.domain.com_1-000001.vmdk"  ====>> Host VMData

-rw-------    1 root     root        6.3M Feb  5 18:38 ProblemVM.domain.com_1-000001-ctk.vmdk
-rw-------    1 root     root       12.6G Feb  5 18:36 ProblemVM.domain.com_1-000001-sesparse.vmdk
-rw-------    1 root     root         445 Feb  5 18:21 ProblemVM.domain.com_1-000001.vmdk
-rw-------    1 root     root        6.3M Feb  5 18:20 ProblemVM.domain.com_1-ctk.vmdk
-rw-------    1 root     root        3.1T Feb  5 17:51 ProblemVM.domain.com_1-flat.vmdk
-rw-------    1 root     root         740 Feb  5 17:50 ProblemVM.domain.com_1.vmdk


scsi0:1.fileName = "/vmfs/volumes/5a592580-7f54ceb3-0445-000e1eb52a90/ProblemVM/ProblemVM-000008.vmdk"    ====>> CML Datastore 07_ProblemVM

-rw-------    1 root     root        7.0M Feb  5 18:21 ProblemVM-000002-ctk.vmdk
-rw-------    1 root     root        1.9T Feb  5 18:36 ProblemVM-000002-sesparse.vmdk
-rw-------    1 root     root         433 Jan 30 15:31 ProblemVM-000002.vmdk
-rw-------    1 root     root        7.0M Feb  5 18:21 ProblemVM-000003-ctk.vmdk
-rw-------    1 root     root       14.3G Feb  5 18:36 ProblemVM-000003-sesparse.vmdk
-rw-------    1 root     root         426 Jan 15 06:32 ProblemVM-000003.vmdk
-rw-------    1 root     root        7.0M Feb  5 18:21 ProblemVM-000004-ctk.vmdk
-rw-------    1 root     root       31.3G Feb  5 18:36 ProblemVM-000004-sesparse.vmdk
-rw-------    1 root     root         433 Feb  2 15:05 ProblemVM-000004.vmdk
-rw-------    1 root     root        7.0M Feb  5 18:21 ProblemVM-000005-ctk.vmdk
-rw-------    1 root     root       14.1G Feb  5 18:36 ProblemVM-000005-sesparse.vmdk
-rw-------    1 root     root         433 Feb  5 17:15 ProblemVM-000005.vmdk
-rw-------    1 root     root        7.0M Feb  5 18:21 ProblemVM-000006-ctk.vmdk
-rw-------    1 root     root       27.6G Feb  5 18:36 ProblemVM-000006-sesparse.vmdk
-rw-------    1 root     root         433 Feb  5 17:15 ProblemVM-000006.vmdk
-rw-------    1 root     root        7.0M Feb  5 18:21 ProblemVM-000007-ctk.vmdk
-rw-------    1 root     root       18.5G Feb  5 18:36 ProblemVM-000007-sesparse.vmdk
-rw-------    1 root     root         433 Feb  5 18:21 ProblemVM-000007.vmdk
-rw-------    1 root     root        7.0M Feb  5 18:38 ProblemVM-000008-ctk.vmdk
-rw-------    1 root     root       14.1G Feb  5 18:36 ProblemVM-000008-sesparse.vmdk
-rw-------    1 root     root         410 Feb  5 18:21 ProblemVM-000008.vmdk
-rw-------    1 root     root        7.0M Feb  5 18:21 ProblemVM-ctk.vmdk
-rw-------    1 root     root        3.5T Jan 13 22:15 ProblemVM-flat.vmdk
-rw-------    1 root     root         523 Jan 15 06:32 ProblemVM.vmdk

*************************************************************************************

What the VMware tech instructed me to do was:

1. Power Down the VM out of hours
2. Consolidate the VM disks
3. Storage vMotion to new datastore(s)

Reading this article has given me pause as I am planning on attempting a disk consolidation this saturday during our maintenance window.  This is a production server and affects most of manufacturing/sales when down.  
My question is, should I go with what VMware told me to do or go with what Andrew swears by?  
I've initiated moving all datastore files (at least for the SAN datastores, one was created locally on the host's internal storage - again a poor decision by the previous person in charge...) into Tier 1 (SSD) in hopes of much faster response time but I'm very nervous about this taking the dreaded "3-4 days" as some have seen before, and I'm also nervous about the possibility of data loss/corruption given how messy these files look.

Any help, tips, ideas, suggestions would be greatly appreciated.
0
LVL 130

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Please post a question and myself or other Experts will be glad to assist!
0

Expert Comment

by:Denver IT Manager
Sorry Andrew, I meant to post that on a different thread:  https://www.experts-exchange.com/questions/28986920/VM-Virtual-machine-disks-consolidation-needed.html

That should clear up the comment I made:  My question is, should I go with what VMware told me to do or go with what Andrew swears by?  

You mentioned in the thread I referenced that you should not do a consolidation, but rather do a manual snapshot, wait a couple minutes, and then delete the snapshot.
0
LVL 130

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Please post a NEW question, and we can discuss your current issue.
0
LVL 130

Author Comment

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Answered!
0

Expert Comment

by:piaakit
Yea In the vm datastore I see many xxx0001.ctk.vmdk
Shall I delete them manually ? Will this delta snapshot  
Caused vm OS got stuck in one area can't boot into the os ?
0

Featured Post

IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

Teach the user how to use create log bundles for vCenter Server or ESXi hosts Open vSphere Web Client: Generate vCenter Server and ESXi host log bundle:  Open vCenter Server Appliance Web Management interface and generate log bundle: Open vCenter Se…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

Keep in touch with Experts Exchange

Tech news and trends delivered to your inbox every month