?
Solved

"Remove all snapshots" very slow after shutting down VM from the OS

Posted on 2015-01-08
15
Medium Priority
?
1,269 Views
Last Modified: 2015-01-08
We have a VMware host running ESXi v5.5.0.  A (Server 2012) VM has many snapshots.  I ran "remove all snapshots" while the VM was running.  This got to around 50% in about an hour.  I then foolishly shut down the OS from within Windows in the hope that this would speed up the removal.  In fact it appears to have now slowed to a crawl (it's been on 56% for over two hours) with the VM is showing as still powered up, with a blank screen (because you can't make any changes to VMs while snapshots are being consolidated).

I tried to cancel the snapshot removal which didn't appear to make a difference, although the option is now greyed out.

Is there any way to either reboot the VM or speed up / abort the snapshot removal - or is it just a matter of waiting?  If the latter, is the percentage complete figure likely to be consistent and accurate, or might it just suddenly jump to 100%

Thanks in advance for any suggestions.
0
Comment
Question by:David Haycox
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 7
15 Comments
 
LVL 122
ID: 40538753
We have a VMware host running ESXi v5.5.0.  A (Server 2012) VM has many snapshots.  I ran "remove all snapshots" while the VM was running.  This got to around 50% in about an hour.  I then foolishly shut down the OS from within Windows in the hope that this would speed up the removal.  In fact it appears to have now slowed to a crawl (it's been on 56% for over two hours) with the VM is showing as still powered up, with a blank screen (because you can't make any changes to VMs while snapshots are being consolidated).

Oh Dear!

The Answer is Be Patient!

Do not mess, meddle, power off, shutdown, restart or fiddle, when you remove a snapshot, otherwise serious corruption can occur or cancel!

I tried to cancel the snapshot removal which didn't appear to make a difference, although the option is now greyed out.

Worse!

Is there any way to either reboot the VM or speed up / abort the snapshot removal - or is it just a matter of waiting?  If the latter, is the percentage complete figure likely to be consistent and accurate, or might it just suddenly jump to 100%

Now you've been told off, let's see if we can remedy the situation!

Be Patient.....

Can you upload a screenshot of the folder.... for me to examine...

Now that you have fiddled, you have have caused it to HANG....

but lets get a screenshot...

Grab a coffee, or beer, relax chill out, read my EE Article, whilst I look at your snapshot...

HOW TO: VMware Snapshots :- Be Patient
0
 
LVL 2

Author Comment

by:David Haycox
ID: 40538794
Hi Andrew,

I just found your article - a little too late, alas.  I consider myself told off (as if I hadn't chided myself enough already)!

Many thanks for the quick response.  Here are the folder contents:

VM Files part 1VM Files part 2
Before you say anything, this is a replica server (using Veeam) and was never intended to be used to run VMs... until the production server died and we just switched on the replica without first clearing its 25 snapshots...!

I can confirm that removing all snapshots on another (smaller, powered-off) VM on the same host completes successfully in only a few minutes.

Thanks again.
0
 
LVL 122
ID: 40538834
Okay, don't have to study it too much.... the reason it's slow, is because of the many snapshots, there is a magic limit to the max number of snapshot a VM can have, and then the VM would fail, and not power on, I was told by VMware this is 29, but I have seen VMs, with up to 72 snapshots and still running!

anyway....how long has this been running, and can you see any changes in the datastore, e.g. snapshots disappearing ?
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 2

Author Comment

by:David Haycox
ID: 40538864
It has been running for about 4 hours 40 minutes.  The progress bar was moving well until I shut down the VM's OS (about an hour in).  It has moved since then, but only by a couple of percent and not at all in the last 3 hours.

I have been monitoring the datastore files, but only for about the last hour or so.  There were 86 files then, and the same now.  That's 75 for the 25 snapshots of the 3 virtual disks, plus 11 assorted others?  Hmm.
0
 
LVL 2

Author Comment

by:David Haycox
ID: 40538917
The other VMs have files like "VMname-snapshotXXX.vmsn" in them (one for each snapshot by the look of it), but these are  missing from the troublesome one.  Is that expected?  Thanks.
0
 
LVL 122

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 2000 total points
ID: 40538933
Okay, I think this process has crashed!

A few options.... if it has crashed, it's not going to complete now, it's stalled.

and the only options to clear-up the HANG, is to

1. Restart Network Management Agents and/or
2. Restart the vCenter Server service if applicable

(the above will not affect any other running machines!).

3. Sometimes, a HOST Reboot will be required.

4. and then, we will have to "cross fingers" and hope no corruption has occurred, and I cannot guarantee this, stopping any snapshot whilst is started is prone with error.

We will then *NOT* attempt the snapshot again, there are two option functons we can try...

A. CLONE out the current disks and snapshot to a new virtual machine disk.
B. CLONE the VM (same will happen!), but we can only do this with vCenter Server.
C. use VMware Converter to complete a V2V.

So, back to you.... when you want to continue...
0
 
LVL 2

Author Comment

by:David Haycox
ID: 40538944
Sounds like a plan.  If I can just get the SQL database files off, that will be enough!  Will post back when have news.  Thanks again.
0
 
LVL 122
ID: 40538981
That's no problems, "Snapshots are EVIL"
0
 
LVL 2

Author Comment

by:David Haycox
ID: 40538991
Restarted management agents (that was easy, have iLO access so did it from the console).  No sign of snapshot removal but VM still showing powered on.  Controls to stop available but still in progress after several minutes.  May have to reboot host?
0
 
LVL 122
ID: 40539038
What is the state of the VM, on or off ?

can you perform any tasks on the VM ?

or does it state task in progress, or not in this state!
0
 
LVL 2

Author Comment

by:David Haycox
ID: 40539061
VM state was on, tasks can be performed and I tried to power it off.  States task in progress.  Snapshot manager shows no snapshots (all files still present in folder though).

Anyway, I waiting a few minutes then rebooted.  Was able to power up the VM fine!!  Am copying database files off it manually to just put them back on the original VM (host's mainboard failed).  I reckon that'll be faster than a Veeam recovery, especially with the snapshots being all over place.

You are a hero, sir!  I have been (rightly) chastised and learned a great deal.  I certainly won't be making that mistake again.  Thank you very much, indeed.
0
 
LVL 122
ID: 40539101
do you have vCenter Server, I would assume so ?

Do you need to clean up these snapshots ?
0
 
LVL 2

Author Comment

by:David Haycox
ID: 40539124
We have vCenter server but it's not managing that particular host (the Essentials licence allows for only three hosts, so it's full).  The host in question here is at a different site and is used mainly just to keep replicas of the VMs on the three hosts managed by the vCenter.

If I get the database files copied off cleanly (so far so good) then we don't need to clean the snapshots - once confirmed it's okay on the original host again I'll just delete it so Veeam can recreate a replica.  If you have anything you can copy and paste about snapshot clean-up I'd be interested, but don't waste any time on it as it looks (fingers crossed) like we're done here.
0
 
LVL 122
ID: 40539143
Two methods....

1. Take a snapshot
2. Wait 60 seconds
3. DELETE ALL in Snapshot Manager
4. Be patient

However, with your situation, and the number of snapshots, I would not do this...

I would use a CLONE, with no vCenter you will have to do at the console or login via SSH, and perform the following...

vmkfstools -i <most recent snapshot file name> /vmfs/volumes/<temp folder name>/<newfilename.vmdk>

this will create a brand new disk and merge all the snapshots, you can then safely add this disk to the VM, and discard the old disks, and ALL the snapshots!

Done....

(you can also use VMware Converter to create a V2V)
0
 
LVL 2

Author Comment

by:David Haycox
ID: 40539150
Excellent, filed for future reference.  Much obliged.
0

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this step by step tutorial with screenshots, we will show you HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 6.5 (ESXi 6.5). This is important if you need to enable SSH remote access for additional troubleshooting of the ESXi hos…
Ransomware is a malware that is again in the list of security  concerns. Not only for companies, but also for Government security and  even at personal use. IT departments should be aware and have the right  knowledge to how to fight it.
Teach the user how to rename, unmount, delete and upgrade VMFS datastores. Open vSphere Web Client: Rename VMFS and NFS datastores: Upgrade VMFS-3 volume to VMFS-5: Unmount VMFS datastore: Delete a VMFS datastore:
This video shows you how to use a vSphere client to connect to your ESX host as the root user. Demonstrates the basic connection of bypassing certification set up. Demonstrates how to access the traditional view to begin managing your virtual mac…
Suggested Courses

801 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question