Link to home
Start Free TrialLog in
Avatar of JM D.
JM D.

asked on

Vmware consolidation is needed but fails / no snapshot remaining

I've a Vmware VM server, with 4 disks (Vsphere 7). Messaging server witch some big disks.
A consolidation is needed for this machine, but it always fails.

The machine is dayly backuped witch Acronis Cyber Backup.

For the backup (hot backup) the job do a snapshot of the machine, and delete it after the backup. The backup files are on a filer and a cloud storage.

It seems, that after the backup, a new vmdk file is created with a name like:

server-0000xx.vmdk for the first disk (300 GB).

The vmx file is refers to the this new vmdk file for the first disk.

There is no remaining snapshot  and no delta file in the datastore.

There are now dozens of files, with this server-0000xx.vmdk (and ...-ctk.vmdk).


What's the good solution?

Stop the VM, modifiy .vmx file and change the reference to the server-0000xx.vmdk to server-0.vmdk, rename this file with the same name, remove the other files server-0000xx.vmdk. And retart the VMs?

Is that the good way? No risk to loss data ?

May be a better solution ?


I thank you for your advices.


J.




Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Your method is prone to failure! Do not mess!

I can work with you to obtain a solution

Those files you describe are snapshots!

Can you check disk properties in the VM settings is the current disk a file with 00000xx.vmdk
Avatar of JM D.
JM D.

ASKER

Hello Andrew,


Indeed, I made a mistake, by not carefully reading an Internet article presenting the method I mentioned.


Yes, in the VM seetings of the VM, the file server-000058.vmx.
And in the .vmx there is this line:
scsi0:0.fileName = "server-000058.vmdk"


J.




Okay so you definitely have a VM running on a snapshot.

This could be because the parent disk is currently attached to the backup server so please check the backup server and check the patent is not attached if it’s not attached the parent disk has a lock on it!

The fact that the file states 58 could mean the snapshot chain is 58 disks which technically VMware support stated is impossible and cannot happen and possibly cannot consolidated

So please check back server and get back to me and we will proceed
Avatar of JM D.

ASKER

Indeed, when a consolidation task is launched, the result is "Virtual machine disks consolidation succeeded".

But consolidation is always needed after that.


Well, in the datastore I don't see the parent disk. It should be server-01.vmdk.

But I only find from server-000001.vmdk to server-000058.vmdk


Regarding your question about the backup server. I don't know how to check if there is a parent disk attached to it.

The backup task was set up by a former colleague and I'm not sure where to find the information. I will look tomorrow.


Good night.

Indeed, when a consolidation task is launched, the result is "Virtual machine disks consolidation succeeded".

vSphere and Acronis backup is dumb! It's just going through the motions, but cannot merge the snapshots.

Edit the VM settings of the Backup Server (assuming's it's a VM), manually check the parent vmdk is not attached.

there must be a server-01.flat-vmdk, otherwise the chain would be broken, and the VM would not be on.

A quick test

1, Create/Take a new snapshot.
2. Wait 120 seconds.
3. select DELETE ALL

Do you end up with another server-000059.vmdk

Avatar of JM D.

ASKER

Hello Andrew,

I don't believe Acronis Backup creates VMs but files with a proprietary format, .tibx files.

I can make your quick test .

J.

I'm not quite sure what the last comment refers to.

Anyway did you check the Acronis backup server Disk Properties ?

Did you do the quick snapshot test ?
Avatar of JM D.

ASKER

The snapshot test have been done.

A new disk server-000060.vmdk have been created and remains the disk used after snaphsot deletion in the VM settings. That's for the first disk. No problem for the other disks.


J.

Okay, so the parent disk is locked or in use.

Is the Backup Server with Acronis, a VM ?
Avatar of JM D.

ASKER

correction:

I said I don't see the parent file.

Indeed, it is present. That's probably server.vmdk


It's weird, with the command " find /vmfs/volumes/ -name *-delta*;find /vmfs/volumes/ -name *-0000* "  I've found a file 

"server-000002.vmdk.REDO_6RTx2E" in the folder corresponding to an old  Acronis virtual machine ?????

The same day as for the other file server.vmdk (the supposed parent file)  !!!!

It's a real tangled web !!!


NB: command "vmfsfilelockinfo -p server.vmdk " does not indicate a lock on this file (is that command correct ?).


About Acronis,  the Acronis Backup system is a bit complex. There are several servers on which there is an Acronis agent.

I believe the server that executes the task is never the same (unless one has been forced in particular).


J.


All I really need to know is

1. Is the Acronis Backup server a VMware Virtual Machine ? - Yes / No ?

2. If so, check the virtual machine disk properties, now you have found the server.vmdk is it attached to the Acronis backup server - if not a virtual machine - not applicable.

Once you have found the parent vmdk file attached to a server, remove it from the server, do not delete from disk.

Either there is a lock on the file, or the chain is broken, hence why the snapshot cannot me merged, and creates a new snapshot just need to establish what has the parent vmdk.

Otherwise your only option, because the snapshot chain cannot be merged is to try

1. CLONE the machine to a new machine. The resulting CLONE will not have any snapshots, then use this CLONE VM. This can be done at the GUI or shell. But the VM will need to be powered off to complete this action to ensure there is no loss of data, and databases are consistent.

However this does very much depend on the application/service of the VM, because you will need to shutdown the VM.

So to have 59/60  snapshots, based on 1 a day being created, it's been like this for  days before being spotted.

What is the VM ? Exchange server ?

Even if we could consolidate the VM, it's possible it may cause the VM to have issues in realtime with message and delivery.
Avatar of JM D.

ASKER

There are several Acronis backup servers (twelve !!!).

I've just found the server involved in the backup process. Its' a VMs and have his second disk attached to one of the "faulty" server (yes, It's an Exchange server).

server-000002.vmdk.

So, i delete the attachment in the vm settings ?

And should I try a new consolidation after that ?

ASKER CERTIFIED SOLUTION
Avatar of Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Andrew Hancock (VMware vExpert PRO / EE Fellow/British Beekeeper)
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of JM D.

ASKER

I haven't done anything yet.

So let's summarize:

I detach the disk corresponding to the VM to backup from the Acronis backup VM (click on the "cross", behind disk on the VM settings).


I Create/Take a new snapshot.
Wait 120 seconds.
select DELETE ALL


Is that ok ? Did I understand well ?

Avatar of JM D.

ASKER

Well, i've launched the snapshot deletetion (delete all).

I will be patient.

I'll let you know. 


J.


if it didn't create another 000061.vmdk, then it will start the merge of all the snapshots into the parent.
Avatar of JM D.

ASKER

There was a server-000061.vmdk

The task seems have been ended successfully. 1 big hour.

14 TB has been recovered !!!!

Many thanks, for your help,  Andrew.

I will be monitoring these backups more closely.


J.


Glad it all worked out fine for you. Check all -0000xx have gone, and also the parent disk, is the current disk the server is writing to.

Patience with snapshots is the key!

Snapshots are evil, and you need to really check after each backup none our left.

TIP! Enable vCenter Server ALARMS!
Avatar of JM D.

ASKER

The disks are correct now.

server.vmdk

server_1.vmdk

server_2.vmdk

server_3.vmdk


Remove me from doubt, do you think - on the backup server - that the attachment to the disk of the machine to be backed up is done at the time of the Acronis backup?

Or is it a requirement for the Acronis backup server that the disk is permanently attached. I imagine that we are rather in the 1st case.

The disk should only be hot-add during backup and released.

But I'm afraid this is a common issue with all third party backup solutions it's not just Acronis.

I'm afraid it comes down to VMware Administration, and one of your daily checks to check for such things.
Avatar of JM D.

ASKER

From my point of view, Acronis Cyber Backup generates too many errors that it is difficult to follow, especially when you are short of time.

In your opinion, are backup products based on snapshots good solutions? But are there only other solutions?

Are Veeam solutions good solutions?

I’m afraid there are no other solutions all third party backup solutions are based on the Storage API which issues a snapshot!

Sadly it comes down to VMware Administration and monitoring the health of the environment
Avatar of JM D.

ASKER

ok.

Thanks, Andrew.