Virtual machine disks consolidation needed.

I noticed the other day that all of our VM's that get backed up by Veeam Backup and Recovery had configuration issues and were displaying "Virtual machine disks consolidation needed."

This coincided with poor VM performance accross all of the VM's affected - so i thought that (even though Veeam had competed sucessfully) the snapshots must have failed to be removed after being backed up. This was verified by seeing vmdisk-0000001.vmdk in the datastores for all of the machines affected, however, in snapshot manager it shows no snapshots.

If i try and consolidate whilst the VM is on it fails with "The maximum consolidate retries was exceeded for scsi0:0.

If i switch the VM off and consolidate it goes through and removes the snapshot disks... happy days, however the next time Veeam runs it leaves the VM's in the same "needing consolidation" state.

I tried rebooting the host ESXi 5.5 and recreating a new Veeam backup job but still the snapshots arent properly removed.

So, to try and rule Veeam out of the equasion i tried creating a new snapshot on one of the VMs through snapshot manager and then deleting it. This said it had completed sucessfully but looking at the datastore i am seeeing vmdisk-0000001.vmdk and alarm saying that "Virtual machine disks consolidation needed."

As i didnt want to mess around too much with the production environment - i have cloned one of the VMs, and re-run the Create Snapshot / Remove Snaphot which again completes sucessfully but then leaves the snapshot disk behind and asks for disk consolidation.

Running out of ideas here - has anyone come accross this before?
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Yes,  possibly storage too slow.

Are you sure the snapshot is in use?

The consolidation detection is a bit stupid, and if it detects a file with -000x.vmdk it says, okay, lets put up a Consolidation Warning.

So....can you check if VM is stilling running on a snapshot.
dbarnbrookAuthor Commented:
Hi Andrew, thanks for the quick response...

We have a Dell Equalogic PS6100 SAS iSCSI for the storage and haven't seen any speed issues previously.

So if a snapshot is created and then removed successfully, either by a Veeam Backup or by the snapshot manager, we see the vmdisk-0000001.vmdk left over (indicating that it is still running off a snapshot). The only way to remove this is by switching off the VM and then using consolidation (which removes the 00000x.vmdk).

I've disabled all backups for the timebeing as i dont want to keep having to switch off VM's and consolidating them.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
depends if the snapshot is locked, because it's in use by another process.

Is the VM running on the snapshot, have you confirmed this ?

just because there is a snapshot file, it could be an orphan child.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

dbarnbrookAuthor Commented:
Yes the VM's are definitely running on snapshots... when you go to Edit Settings you can see the Hard Disk is the -00000x.vmdk
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Okay, so this confirms, the VM is actually running the snapshot.

The parent disks are not currently assigned to any other storage backup device, e.g. check Veeam, the parent disks are not still attached.....

and then try the following:-

1. Take a manual snapshot (no need for memory or quieces the VM)
2. wait 60 seconds to 2 minutes

wait and be patient....
dbarnbrookAuthor Commented:
Okay, i checked Veeam and definately no backups currently running (cant see the parent disks attached anywhere).

Taken a manual snapshot - completed
Waited a few mins - completed
Deleted All - completed

The "Remove all snapshots" task completed and then immediately get a warning "Virtual machine consolidation is needed" and "Virtual machine disks consolidation failed". And we are back to running from -000002.vmdk.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
okay, there is a lock on the snapshot (child), or the parent VMDK.

In fact the remove process is not working at all....

The create process, is creating a new snapshot as it should, and it's chaining it to 0001.vmdk which is chained to the parent....

Look closely that the parent disk, is not attach or in use to any other VM, or Task in Progress....

To ensure, the locks are off...

1. Restart vCenter Server services (does not affect VMs)

2. Restart the host ESXi server.

Does this happen to all VMs ?
dbarnbrookAuthor Commented:
After restarting VCenter Server services and ESXi host we were still seeing the issue.

So have managed to resolve by following this VMWare article KB2082886

It talks about how ESXi5.5 (and 6) handle snapshots differently from previous versions so I still find it strange how this issue suddenly appeared as we have been running ESXi5.5 for months now without issues and haven't changed our backup procedure or Veeam Version.

Anyway thanks for the suggestions...
dbarnbrookAuthor Commented:
I've requested that this question be closed as follows:

Accepted answer: 0 points for dbarnbrook's comment #a40947674

for the following reason:

Tried all of the Experts suggestions but was still experiencing issues so tried something else which sorted the problem.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The problem has not been resolved you have applied a "band-aid" - workaround, the issue you are experiencing, was detail in my first post....http:#a40944050

"Yes,  possibly storage too slow."

You have just turn-off the issue!

At the bottom of the KB Article it states:-

Note: To work around this issue, when under heavy IO load, you can alternatively retry snapshot consolidation at a time when the virtual machine is issuing less IO.

e.g. storage too slow, you have overwhelmed your storage!

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
dbarnbrookAuthor Commented:
Ok i'll definately look at the storage speed Andrew - thanks for your timeand input.

I'm just happy for the timebeing that the workaround has worked and we are not running on snapshots on all of our production servers and that we can resume the backup processes.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.