dbarnbrook
asked on
Virtual machine disks consolidation needed.
I noticed the other day that all of our VM's that get backed up by Veeam Backup and Recovery had configuration issues and were displaying "Virtual machine disks consolidation needed."
This coincided with poor VM performance accross all of the VM's affected - so i thought that (even though Veeam had competed sucessfully) the snapshots must have failed to be removed after being backed up. This was verified by seeing vmdisk-0000001.vmdk in the datastores for all of the machines affected, however, in snapshot manager it shows no snapshots.
If i try and consolidate whilst the VM is on it fails with "The maximum consolidate retries was exceeded for scsi0:0.
If i switch the VM off and consolidate it goes through and removes the snapshot disks... happy days, however the next time Veeam runs it leaves the VM's in the same "needing consolidation" state.
I tried rebooting the host ESXi 5.5 and recreating a new Veeam backup job but still the snapshots arent properly removed.
So, to try and rule Veeam out of the equasion i tried creating a new snapshot on one of the VMs through snapshot manager and then deleting it. This said it had completed sucessfully but looking at the datastore i am seeeing vmdisk-0000001.vmdk and alarm saying that "Virtual machine disks consolidation needed."
As i didnt want to mess around too much with the production environment - i have cloned one of the VMs, and re-run the Create Snapshot / Remove Snaphot which again completes sucessfully but then leaves the snapshot disk behind and asks for disk consolidation.
Running out of ideas here - has anyone come accross this before?
This coincided with poor VM performance accross all of the VM's affected - so i thought that (even though Veeam had competed sucessfully) the snapshots must have failed to be removed after being backed up. This was verified by seeing vmdisk-0000001.vmdk in the datastores for all of the machines affected, however, in snapshot manager it shows no snapshots.
If i try and consolidate whilst the VM is on it fails with "The maximum consolidate retries was exceeded for scsi0:0.
If i switch the VM off and consolidate it goes through and removes the snapshot disks... happy days, however the next time Veeam runs it leaves the VM's in the same "needing consolidation" state.
I tried rebooting the host ESXi 5.5 and recreating a new Veeam backup job but still the snapshots arent properly removed.
So, to try and rule Veeam out of the equasion i tried creating a new snapshot on one of the VMs through snapshot manager and then deleting it. This said it had completed sucessfully but looking at the datastore i am seeeing vmdisk-0000001.vmdk and alarm saying that "Virtual machine disks consolidation needed."
As i didnt want to mess around too much with the production environment - i have cloned one of the VMs, and re-run the Create Snapshot / Remove Snaphot which again completes sucessfully but then leaves the snapshot disk behind and asks for disk consolidation.
Running out of ideas here - has anyone come accross this before?
ASKER
Hi Andrew, thanks for the quick response...
We have a Dell Equalogic PS6100 SAS iSCSI for the storage and haven't seen any speed issues previously.
So if a snapshot is created and then removed successfully, either by a Veeam Backup or by the snapshot manager, we see the vmdisk-0000001.vmdk left over (indicating that it is still running off a snapshot). The only way to remove this is by switching off the VM and then using consolidation (which removes the 00000x.vmdk).
I've disabled all backups for the timebeing as i dont want to keep having to switch off VM's and consolidating them.
We have a Dell Equalogic PS6100 SAS iSCSI for the storage and haven't seen any speed issues previously.
So if a snapshot is created and then removed successfully, either by a Veeam Backup or by the snapshot manager, we see the vmdisk-0000001.vmdk left over (indicating that it is still running off a snapshot). The only way to remove this is by switching off the VM and then using consolidation (which removes the 00000x.vmdk).
I've disabled all backups for the timebeing as i dont want to keep having to switch off VM's and consolidating them.
depends if the snapshot is locked, because it's in use by another process.
Is the VM running on the snapshot, have you confirmed this ?
just because there is a snapshot file, it could be an orphan child.
Is the VM running on the snapshot, have you confirmed this ?
just because there is a snapshot file, it could be an orphan child.
ASKER
Yes the VM's are definitely running on snapshots... when you go to Edit Settings you can see the Hard Disk is the -00000x.vmdk
Okay, so this confirms, the VM is actually running the snapshot.
The parent disks are not currently assigned to any other storage backup device, e.g. check Veeam, the parent disks are not still attached.....
and then try the following:-
1. Take a manual snapshot (no need for memory or quieces the VM)
2. wait 60 seconds to 2 minutes
3. DELETE ALL...
wait and be patient....
The parent disks are not currently assigned to any other storage backup device, e.g. check Veeam, the parent disks are not still attached.....
and then try the following:-
1. Take a manual snapshot (no need for memory or quieces the VM)
2. wait 60 seconds to 2 minutes
3. DELETE ALL...
wait and be patient....
ASKER
Okay, i checked Veeam and definately no backups currently running (cant see the parent disks attached anywhere).
Taken a manual snapshot - completed
Waited a few mins - completed
Deleted All - completed
The "Remove all snapshots" task completed and then immediately get a warning "Virtual machine consolidation is needed" and "Virtual machine disks consolidation failed". And we are back to running from -000002.vmdk.
Taken a manual snapshot - completed
Waited a few mins - completed
Deleted All - completed
The "Remove all snapshots" task completed and then immediately get a warning "Virtual machine consolidation is needed" and "Virtual machine disks consolidation failed". And we are back to running from -000002.vmdk.
okay, there is a lock on the snapshot (child), or the parent VMDK.
In fact the remove process is not working at all....
The create process, is creating a new snapshot as it should, and it's chaining it to 0001.vmdk which is chained to the parent....
Look closely that the parent disk, is not attach or in use to any other VM, or Task in Progress....
To ensure, the locks are off...
1. Restart vCenter Server services (does not affect VMs)
2. Restart the host ESXi server.
Does this happen to all VMs ?
In fact the remove process is not working at all....
The create process, is creating a new snapshot as it should, and it's chaining it to 0001.vmdk which is chained to the parent....
Look closely that the parent disk, is not attach or in use to any other VM, or Task in Progress....
To ensure, the locks are off...
1. Restart vCenter Server services (does not affect VMs)
2. Restart the host ESXi server.
Does this happen to all VMs ?
ASKER
After restarting VCenter Server services and ESXi host we were still seeing the issue.
So have managed to resolve by following this VMWare article KB2082886
It talks about how ESXi5.5 (and 6) handle snapshots differently from previous versions so I still find it strange how this issue suddenly appeared as we have been running ESXi5.5 for months now without issues and haven't changed our backup procedure or Veeam Version.
Anyway thanks for the suggestions...
So have managed to resolve by following this VMWare article KB2082886
It talks about how ESXi5.5 (and 6) handle snapshots differently from previous versions so I still find it strange how this issue suddenly appeared as we have been running ESXi5.5 for months now without issues and haven't changed our backup procedure or Veeam Version.
Anyway thanks for the suggestions...
ASKER
I've requested that this question be closed as follows:
Accepted answer: 0 points for dbarnbrook's comment #a40947674
for the following reason:
Tried all of the Experts suggestions but was still experiencing issues so tried something else which sorted the problem.
Accepted answer: 0 points for dbarnbrook's comment #a40947674
for the following reason:
Tried all of the Experts suggestions but was still experiencing issues so tried something else which sorted the problem.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Ok i'll definately look at the storage speed Andrew - thanks for your timeand input.
I'm just happy for the timebeing that the workaround has worked and we are not running on snapshots on all of our production servers and that we can resume the backup processes.
I'm just happy for the timebeing that the workaround has worked and we are not running on snapshots on all of our production servers and that we can resume the backup processes.
Are you sure the snapshot is in use?
The consolidation detection is a bit stupid, and if it detects a file with -000x.vmdk it says, okay, lets put up a Consolidation Warning.
So....can you check if VM is stilling running on a snapshot.