VMware ESXi v4 Snapshot Removal Stuck at 99%

darrell_chapman
darrell_chapman used Ask the Experts™
on
We are running VMware ESXi v4 on a physical server utilitizing iSCSI storage. I realized last night that one of our virtual hosts had some snapshots from a few months ago. Not wanting them to get out of control I decided to delete all snapshots beginning at 9:16 PM last evening. The virtual host was powered down to do this to speed up the process however I was shocked to find this task still running this morning when I arrived at the office. It has been running for over 12 hours now and seems to be stuck at 99%. Can anyone help me to determine if it is still running or if it's locked up please?

Here is the current state of the disk files for this host:
Disk1: disk size=100GB, snapshot delta=608.20MB
Disk2: disk size=40GB, no snapshot
Disk3: disk size=100GB, snapshot delta=6.52GB
Disk4: disk size=80GB, snapshot delta=16MB
Disk5: disk size=500GB, snapshot delta=130.91GB
Disk6: disk size=500GB, no snapshot

Your help is greatly appreciated.
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017
Commented:
How large is the total snapshot, what is the underlying datastore,

THIS IS NORMAL AND WILL COMPLETE BE PATIENT.


We waited minutes, hours and days....longest so far is 3.5 days at 99%!

do not fiddle, cancel, restart etc
some background info on snapshots

A snapshot is NOT a backup of a VM; that is a gross misconception.  

A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes.  When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file.  There is an active link between the original VMDK file and the new delta file.  Anything that gets written to the VM actually gets written to the delta file.   The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM.  If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.  

I highly recommend reading these 2 articles on VMware Virtual Machine Snapshots:

Understanding Snapshots - http://kb.vmware.com/kb/1015180
Snaphot Best Practices - http://kb.vmware.com/kb/1025279

Also check out the following Snapshot Articles by Eric Siebert

Pt.1- http://is.gd/Lajg4p
Pt.2- http://is.gd/NdKQWC
Pt.3- http://is.gd/tp2vEK

Also just to back this up with other EE members experiences, same Question here, with resolution be patient, some snapshots can fail a merge.

http://www.experts-exchange.com/Software/VMWare/Q_27723854.html

Author

Commented:
I know how snapshots work but I've never had it take this long to commit the delta data back to the base disk. I just wish I had some sort of feedback from ESXi to tell me it's working and how long to expect it to finish.

Author

Commented:
Oh, an VMware's knowledgebase is down for maintenance today (of all days).
11/26 Forrester Webinar: Savings for Enterprise

How can your organization benefit from savings just by replacing your legacy backup solutions with Acronis' #CyberProtection? Join Forrester's Joe Branca and Ryan Davis from Acronis live as they explain how you can too.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
Yes, I'm aware this KB is down, we have a local copy here, on a flash drive, we take around with us.

Personally, I would wait for 24 hours.

what's the datastore? RAID x? SATA or SAS, iSCSI, Jumbo Frames?

total size of snapshots?

Author

Commented:
iSCSI on gigabit Ethernet with jumbo frames. Only 138GB snapshots
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
datastore? RAID x? SATA?

have you checked esxtop and any traffic on iSCSI?

Author

Commented:
It's an 8 spindle RAID6 LUN with SAS drives. I can't run exstop as that requires SSH access and this is ESXi (with SSH disabled).

There's plenty of iSCSI traffic however I also have other servers as well that are active.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
Fellow 2018
Expert of the Year 2017

Commented:
okay RAID6 is not particular fast, check the datastore for any changes.

if you have committed 138GB of snapshots before, and it was faster, its always possible its hung. But you would be more familiar with the performance of your rig than I.

check the vmware.log in vm folder, as you have no access to console, you cannot tail the logs for activity, so you would have to download from vSphere Client and vCenter.

Author

Commented:
Two of the delta files (Disk3:6.52GB, Disk4:16MB) have been committed to the base disks so progress is being made; just very, very slowly.  I still have the 608MB and 130GB delta files to commit.

The vmware.log file hasn't been modified since last evening even though the changes above were just committed a few hours ago. The VM's configuration file has been changed recently (as expected) to account for the change above.
Nagendra Pratap SinghDesktop Applications Specialist

Commented:
@hanccocka,

How did you download the KB? Scraping or some less messy way?

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial