asked on
Vmware consolidation is needed but fails / no snapshot remaining
I've a Vmware VM server, with 4 disks (Vsphere 7). Messaging server witch some big disks.
A consolidation is needed for this machine, but it always fails.
The machine is dayly backuped witch Acronis Cyber Backup.
For the backup (hot backup) the job do a snapshot of the machine, and delete it after the backup. The backup files are on a filer and a cloud storage.
It seems, that after the backup, a new vmdk file is created with a name like:
server-0000xx.vmdk for the first disk (300 GB).
The vmx file is refers to the this new vmdk file for the first disk.
There is no remaining snapshot and no delta file in the datastore.
There are now dozens of files, with this server-0000xx.vmdk (and ...-ctk.vmdk).
What's the good solution?
Stop the VM, modifiy .vmx file and change the reference to the server-0000xx.vmdk to server-0.vmdk, rename this file with the same name, remove the other files server-0000xx.vmdk. And retart the VMs?
Is that the good way? No risk to loss data ?
May be a better solution ?
I thank you for your advices.
J.
ASKER
Hello Andrew,
Indeed, I made a mistake, by not carefully reading an Internet article presenting the method I mentioned.
Yes, in the VM seetings of the VM, the file server-000058.vmx.
And in the .vmx there is this line:
scsi0:0.fileName = "server-000058.vmdk"
J.
This could be because the parent disk is currently attached to the backup server so please check the backup server and check the patent is not attached if it’s not attached the parent disk has a lock on it!
The fact that the file states 58 could mean the snapshot chain is 58 disks which technically VMware support stated is impossible and cannot happen and possibly cannot consolidated
So please check back server and get back to me and we will proceed
ASKER
Indeed, when a consolidation task is launched, the result is "Virtual machine disks consolidation succeeded".
But consolidation is always needed after that.
Well, in the datastore I don't see the parent disk. It should be server-01.vmdk.
But I only find from server-000001.vmdk to server-000058.vmdk
Regarding your question about the backup server. I don't know how to check if there is a parent disk attached to it.
The backup task was set up by a former colleague and I'm not sure where to find the information. I will look tomorrow.
Good night.
Indeed, when a consolidation task is launched, the result is "Virtual machine disks consolidation succeeded".
vSphere and Acronis backup is dumb! It's just going through the motions, but cannot merge the snapshots.
Edit the VM settings of the Backup Server (assuming's it's a VM), manually check the parent vmdk is not attached.
there must be a server-01.flat-vmdk, otherwise the chain would be broken, and the VM would not be on.
A quick test
1, Create/Take a new snapshot.
2. Wait 120 seconds.
3. select DELETE ALL
Do you end up with another server-000059.vmdk
ASKER
Hello Andrew,
I don't believe Acronis Backup creates VMs but files with a proprietary format, .tibx files.
I can make your quick test .
J.
Anyway did you check the Acronis backup server Disk Properties ?
Did you do the quick snapshot test ?
ASKER
The snapshot test have been done.
A new disk server-000060.vmdk have been created and remains the disk used after snaphsot deletion in the VM settings. That's for the first disk. No problem for the other disks.
J.
Is the Backup Server with Acronis, a VM ?
ASKER
correction:
I said I don't see the parent file.
Indeed, it is present. That's probably server.vmdk
It's weird, with the command " find /vmfs/volumes/ -name *-delta*;find /vmfs/volumes/ -name *-0000* " I've found a file
"server-000002.vmdk.REDO_6RTx2E" in the folder corresponding to an old Acronis virtual machine ?????
The same day as for the other file server.vmdk (the supposed parent file) !!!!
It's a real tangled web !!!
NB: command "vmfsfilelockinfo -p server.vmdk " does not indicate a lock on this file (is that command correct ?).
About Acronis, the Acronis Backup system is a bit complex. There are several servers on which there is an Acronis agent.
I believe the server that executes the task is never the same (unless one has been forced in particular).
J.
1. Is the Acronis Backup server a VMware Virtual Machine ? - Yes / No ?
2. If so, check the virtual machine disk properties, now you have found the server.vmdk is it attached to the Acronis backup server - if not a virtual machine - not applicable.
Once you have found the parent vmdk file attached to a server, remove it from the server, do not delete from disk.
Either there is a lock on the file, or the chain is broken, hence why the snapshot cannot me merged, and creates a new snapshot just need to establish what has the parent vmdk.
Otherwise your only option, because the snapshot chain cannot be merged is to try
1. CLONE the machine to a new machine. The resulting CLONE will not have any snapshots, then use this CLONE VM. This can be done at the GUI or shell. But the VM will need to be powered off to complete this action to ensure there is no loss of data, and databases are consistent.
However this does very much depend on the application/service of the VM, because you will need to shutdown the VM.
So to have 59/60 snapshots, based on 1 a day being created, it's been like this for days before being spotted.
What is the VM ? Exchange server ?
Even if we could consolidate the VM, it's possible it may cause the VM to have issues in realtime with message and delivery.
ASKER
There are several Acronis backup servers (twelve !!!).
I've just found the server involved in the backup process. Its' a VMs and have his second disk attached to one of the "faulty" server (yes, It's an Exchange server).
server-000002.vmdk.
So, i delete the attachment in the vm settings ?
And should I try a new consolidation after that ?
ASKER
I haven't done anything yet.
So let's summarize:
I detach the disk corresponding to the VM to backup from the Acronis backup VM (click on the "cross", behind disk on the VM settings).
I Create/Take a new snapshot.
Wait 120 seconds.
select DELETE ALL
Is that ok ? Did I understand well ?
ASKER
Well, i've launched the snapshot deletetion (delete all).
I will be patient.
I'll let you know.
J.
ASKER
There was a server-000061.vmdk
The task seems have been ended successfully. 1 big hour.
14 TB has been recovered !!!!
Many thanks, for your help, Andrew.
I will be monitoring these backups more closely.
J.
Patience with snapshots is the key!
Snapshots are evil, and you need to really check after each backup none our left.
TIP! Enable vCenter Server ALARMS!
ASKER
The disks are correct now.
server.vmdk
server_1.vmdk
server_2.vmdk
server_3.vmdk
Remove me from doubt, do you think - on the backup server - that the attachment to the disk of the machine to be backed up is done at the time of the Acronis backup?
Or is it a requirement for the Acronis backup server that the disk is permanently attached. I imagine that we are rather in the 1st case.
But I'm afraid this is a common issue with all third party backup solutions it's not just Acronis.
I'm afraid it comes down to VMware Administration, and one of your daily checks to check for such things.
ASKER
From my point of view, Acronis Cyber Backup generates too many errors that it is difficult to follow, especially when you are short of time.
In your opinion, are backup products based on snapshots good solutions? But are there only other solutions?
Are Veeam solutions good solutions?
Sadly it comes down to VMware Administration and monitoring the health of the environment
ASKER
ok.
Thanks, Andrew.
I can work with you to obtain a solution
Those files you describe are snapshots!
Can you check disk properties in the VM settings is the current disk a file with 00000xx.vmdk