Veeam Proxy issue: Removing Veeam ghost snapshots

Luciano PatrãoICT Senior Infraestructure  Engineer  
CERTIFIED EXPERT
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.
Published:
When using Veeam, or other Backup Appliance, sometimes we get some issues with Backups Snapthos that the Backup creates and cannot remove. This is an example from Veeam Proxy.
This is an example of a vCenter DB with a warning with the following message: "Virtual Disk Consolidation is required". This is normal in a VMware farm and happens sometimes (this could be caused by too many snapshot, very old snapshots or snapshot that were not properly removed). All we need to do is click in the VM and in the "Snapshot - Consolidate" option we can fix this issue. Sometimes is needed to power off the VM so that all vmdk locks are free and VM can consolidate all snapshots.
 
When trying to consolidate (with VM power down) the disks I get: "An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED".

Snapshots01.jpgWith this message, this issue needs further troubleshooting.
 
First, need to check the vmdk files.
 
Note: Since this issue happen in my vCenter DB and vCenter was down, I had to connect with vSphere Client to the ESXi host directly where this VM was allocated ( if this issue is in any of your vCenter VMs, you should check which ESXi host is the VMS using, before power off the vCenter).
 
Browsing the Datastore where the VM is located and in the VM folder, I found 18 Snapshots files and 18 ctk vmdk files. This is not a normal number of snapshots and not a normal number of ctk files.
 
What are ctk files? CTK files are used by Changed Block Tracking (CBT) to track disk sectors that have been changed since the last backup. So that Backup tools only backup the altered disk sectors since the last changed ID (or backup). So after this, I suspected that was an issue originated by Veeam Backup tool.
 
Sometimes all that is necessary is to remove all these ctk files and delete all Snapshots or Consolidate the disks, this will fix the problem.
 
A faster and safe way is to move (not delete) all ctk files to a temp folder and try to delete all Snapshots.
First, you need to power off the VM. Then go to ESXi host shell console and do the following tasks:
 
First, let's list all ctk files.
 
[root@ESXihost-001:/vmfs/volumes/datastore-01/db-003] ls -l *ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000001-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000004-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000005-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000006-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000007-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000008-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000009-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000010-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000011-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000012-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000013-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000014-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000015-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000016-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:48 db-003-000017-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 06:16 db-003-000018-ctk.vmdk
                      -rw-------    1 root     root       5243392 Mar  4 04:49 db-003-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000001-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000004-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000005-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000006-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000007-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000008-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000009-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000010-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000011-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000012-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000013-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000014-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000015-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000016-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:49 db-003_1-000017-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 06:16 db-003_1-000018-ctk.vmdk
                      -rw-------    1 root     root       4260352 Mar  4 04:50 db-003_1-ctk.vmdk
                      -rw-------    1 root     root       6554112 Mar  4 06:16 db-003_2-ctk.vmdk
                      [root@ESXihost-001:/vmfs/volumes/datastore-01/db-003]

Open in new window


Then create a temporally folder inside of the VM folder
 

[root@ESXihost-001:/vmfs/volumes/datastore-01/db-003] mkdir cbt-temp

Open in new window


Then move all the files into the folder
 

[root@ESXihost-001:/vmfs/volumes/datastore-01/db-003] mv *-ctk.vmdk /vmfs/volumes/datastore-01/db-003/cbt-temp/

Open in new window


Then try to delete all Snapshots, or just consolidate the Disks.
 

In my case, when I tried that and I still got: "An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED"
 

Something is locking these vmdk disks. I then focused my attention on Veeam side.
 

In this environment, Veeam Backup Server is a physical machine, but Veeam Proxy is a VM. When using any Backup as a VM (or a Backup VM appliance), Virtual Disks from VMs are added to the appliance so that the appliance can create a backup. When the backup is finished, the Virtual Disks must be released/removed from the VM appliance. Sometimes this is not removed.
 

Investigating the Veeam Proxy VM I notice that 2 disks from the vCenter DB were added into the Veeam Proxy VM.
 

Snapshots02.jpg


Snapshots03.jpg


As we can see, those 2 Virtual Disks are in the Veeam Proxy VMs, that is why they are locked. In this case, we need to remove these virtual disks from this Veeam Proxy in order to delete the snapshots or consolidate the disks.
 

Remove the Virtual Disks from this VM.
 

Snapshots04.jpg


Important Note: Choose the option "Remove from Virtual Machine", not the "Remove from Virtual Machine and delete files from disk". As is shown in the above images. If you choose the second option, Virtual Disks will be deleted from your original VM (in our case is the vCenter DB).
 

After this, now try again to delete all snapshots, or consolidate the disks. For this case, I chose the consolidate option.
 

The consolidation of the Virtual Disks then started. This is a procedure that can take a while (in this case took 45m), so please be patience.
 

Snapshots05.jpg
For the final test, I started the Backup Job for this vCenter and double checked that it did finish properly and no Virtual Disks were locked in the Veeam Proxy or any snapshot were left in the fault VM.
 

Don't forget to delete the temp folder and the files, when the CBT files were moved.
 

Hope this information was useful.
 

This article is the part of my "TIP Articles". So, please vote "Helpful" on this Article. And I encourage your comments and feedback.
2
1,405 Views
Luciano PatrãoICT Senior Infraestructure  Engineer  
CERTIFIED EXPERT
vExpert vSAN, NSX, Cloud Provider, Veeam Vanguard, Virtual Backups, and Storage design, and an active blogger.

Comments (1)

DP230Network Administrator
CERTIFIED EXPERT

Commented:
Hi, in my environment, Veeam backup and Veeam proxy is on the same virtual machine. There is a problematic VM and it's running on 33th snapshot. Can you suggest solutions? Many thanks!

Have a question about something in this article? You can receive help directly from the article author. Sign up for a free trial to get started.