Link to home
Start Free TrialLog in
Avatar of elit2007
elit2007

asked on

Snapshot

This week suddenly two virtual machines on our shop stopped because of to little space on the hosts storage. This is because we didn't think of the extra amount of storage that is required when using snapshots on a Exchange server on this host. The host storage is 270GB. Only 200GB is predefined by the virtual machines, but snapshot files are living there own life's besides and are now nearly reaching 70GB,
Can we safely delete all snapshot's from the snapshot manager in ESXi 4.0, and is it correct that data on the snapshot files will be merged with the "original" storage upon deletions?

snapshotmanager.jpg
storage.jpg
Avatar of coolsport00
coolsport00
Flag of United States of America image

Yes, if you choose to "Delete All", your snapshots will transfer its data to its parent disk in the snap tree. Once the data is transferred, then that orig snap is deleted. This process continues up the "snap tree" until all data is 'commmited' and all snaps are deleted in the tree.

~coolsport00
I highly recommend reading these 2 articles on snaps:
Understanding Snaps - http://kb.vmware.com/kb/1015180
Snap Best Practices - http://kb.vmware.com/kb/1025279

BTW...with the amt of snaps you have, and the size of them all, don't expect this process to end anytime soon. It'll take a day...maybe longer.

~coolsport00
Avatar of Paul Solovyovsky
The snapshots are part of the original vmdk and you will still need the room for them.  When a snapshot is taken the original vmdk is put in read mode and all new data is being written to the snapshot vmdk.  When you delete the snapshot it will write the snapshot to the original vmdk and will make it just as large as it is with snapshots and you will need more space during the process for the snapshots to write the data back to the original.  Snapshots are unlike SAN snapshots, they are a part of the VM.
Avatar of elit2007
elit2007

ASKER

Do we need to shut the vm down first?
And do keep in mind what "pausolov" has added, and what I suggested in my orig post...you will need the space on your datastore equal to the size of all your snaps as the data in all the snaps will be 'commited' to the VMDK.

~coolsport00
Are you sure paulolo?  If you take a look at the screenshot of the data browser you can see the two virtual disks on the top of the list (104GB and 52GB). They are set to be predefined/static.  They cannot grow, and they are just limits and not what actually is stored at the moment. Therefore I don’t understand why I can’t release some storage space when the snapshot files has been merged to the original virtual disk?
when you try to commit a snapshot while a VM is running, it will take another snapshot for the VM to run on while it consolidates the other snapshots.  So, if you don't have any space available or not very much, you will want to power down the VM to prevent another crash.
Good idea :-) Only got 2GB left. So I'm I right about my last post?
You will need more than 2GB left to successfully commit your snaps. See this KB that discusses "consolidating" a bit more:
http://kb.vmware.com/kb/1007849

~coolsport00
You could clone your VM & that removes snapshots during that process, if you have enough space on another datastore to clone it to.

~coolsport00
Either clone or take vmware converter and take another image of it inside the OS, this will just backup the used blockes, I've had issues cloning VMs that have snapshots.
Before you do any of the above are you using vDR? (VMware Data Recovery)
Right now I'm moving the second VM to free up 50GB.
I'm using Veeam backup. But the backup job stopped working when the host ran out of storage. I think this is because Veeam backup don't get enough space for bacakup snapshot. See error message:
Production datastore 'Storage (tiger)' is getting low on free space (2,13 GB left), and may run out of free disk space completely due to open snapshots.

Removing snapshot
RemoveSnapshot failed, snapshotRef "64-snapshot-140", timeout "3600000"
The object has already been deleted or has not been completely created
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually.
Possible causes for snapshot removal failure:
 - Network connectivity issue, or vCenter Server is too busy to serve the request
 - ESX host was unable to process snapshot removal request in a timely manner
 - Snapshot was already removed by another application
BackupDiskFull failed
Client error: Failed to open VDDK disk [[Storage (tiger)] Exchange/Exchange-000016.vmdk] ( is read-only mode - [true] )
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [10.1.254.181];Port: 443;Login: [root];VMX Spec: [moref=64];Snapshot mor: [64-snapshot-140];Transports: [san]] failed because of the following errors:
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [10.1.254.181];Port: 902;Login: [root];VMX Spec: [moref=64];Snapshot mor: [64-snapshot-140];Transports: [nbd]] failed because of the following errors:

Server error: End of file

Open in new window

Veeam does use snapshotting and yes...if there isn't enough space to create its own snap to b/u, it will fail.

~coolsport00
Hello again, this is what I ended up with. First I ran vConverter inside the virtual machine and convertet it to another VM host. Then I shut down the virtual server and pushed "Delete all" in ESXi Snapshot Manager.
Then I went to sleep and when I cheked the server next morning all snapshots where gone from the Snapshot manager, but no snapshotfiles where deleted from the storage. The virtual machine was still running on the last snapshot disk. So I gave up this server and booted up the server that I used vConverter to build. And now it's running on that server. No more use of snapshots for me :0)
ASKER CERTIFIED SOLUTION
Avatar of coolsport00
coolsport00
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Also note that if a VM is powered off, the snapshot consolidation process requires very little free space for functionality (about 16 MB).  

Note that if a VM is powered-off for snapshot consolidation (delete or delete-all), the VM must remain powered off until the consolidation is complete.  Changing the VM power state during snapshot consolidation can potentially break the process.  Conversely, if the VM is left powered-on, it should stay powered-on.  This is why ensuring enough space exists is paramount.

Powering off a VM can also free up space by deleting the VM swap file.  By default, if a VM does not have a memory reservation set, a swap file equal in size to the VM's RAM allocation is created at power on.  i.e. a VM with 8 GB RAM has an 8 GB swap file.  Powering off the VM deletes the 8 GB swap file and will free up that amount of space on the datastore.  This is often used when troubleshooting snapshot issues where the datastore has run out of space.

To confirm that a snapshot consolidation process is complete, command-line monitoring via an ESX host can be performed - see KB article 1007566 - http://kb.vmware.com/kb/1007566.

Kind regards,

-noblehobo