We help IT Professionals succeed at work.

Issues with VEAAM Replication

BarepAssets
BarepAssets asked
on
Hi,
   I am using veaam replication to replicates vms to a standalone host in our DR site.
On replication I was doing testing and the virtual machine was ok turning on etc in the DR site but there
was a certain level of corruption in some of the folders mostly was ok but some folders i could not even get into.
I deleted down the virtual machine in the DR site and started a new veaam job replicating using the vss option as
from searching around the net this appears to be good way of avoiding corruption. Replication started yesterday it is
a 1.5 terrabyte machine it is in a data store of 1.82 terrabyte. When i came in this morning the virtual machine in production
would not switch on as the data store was out of space. I managed to get the virtual machine on by deleting a *.vbk ( i backed up to my local pc first)
which i know  is veaam related file but this was only 256 mb in size. So at the moment i have a virtual machine with should be 1.5 tb and i only have 223mb
free in a datastore 1.82. What files are veaam files that i can delete??

Thks

Eoghan

Comment
Watch Question

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
All Backup products that backup a VMware virtual machine, create a snapshot before backup starts, the virtual machine then writes changes to a snapsot file. This snapshot file has filled up your datastore, and hence there is no space for the virtual machine to power on.

A snapshot is NOT a backup of a VM; that is a gross misconception.  

A snap shot is a way to preserve a point in time when the VM was running OK before making changes. A snapshot is NOT a way to get a static copy of a VM before making changes.  When you take a snapshot of a VM what happens is that a delta file gets created and the original VMDK file gets converted to a Read-Only file.  There is an active link between the original VMDK file and the new delta file.  Anything that gets written to the VM actually gets written to the delta file.   The correct way to use a snapshot is when you want to make some change to a VM like adding a new app or a patch; something that might damage the guest OS. After you apply the patch or make the change and it’s stable, you should really go into snapshot manager and delete the snapshot which will commit the changes to the original VM, delete the snap, and make the VMDK file RW. The official stance is that you really shouldn’t have more than one snap at a time and that you should not leave them out there for long periods of time. Adding more snaps and leaving them there a long time degrades the performance of the VM.  If the patch or whatever goes badly or for some reason you need to get back to the original unmodified VM, that’s possible as well.  

I highly recommend reading these 2 articles on VMware Virtual Machine Snapshots:

Understanding Snapshots - http://kb.vmware.com/kb/1015180
Snaphot Best Practices - http://kb.vmware.com/kb/1025279

Also check out the following Snapshot Articles by Eric Siebert

Pt.1- http://is.gd/Lajg4p
Pt.2- http://is.gd/NdKQWC
Pt.3- http://is.gd/tp2vEK

What you need to do is

1. Increase space on the datastore, and then commit/delete the snapshot. Please make sure you have enough space to delete the snapshot, Snapshot deletion could take MANY HOURS to complete.

2. Decrease the amount of memory that the VM requires, this will reduce the vswap file required, and you will be able to power on the VM temporary.

3. Use VMware Converter to Backup/Export the VM to a workstation, and re-import.

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
A 1.5TB Virtual Disk, stored on a 1.8TB datastore is not enough space, to use Snapshots.

Try reducing the memory in the VM.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
Veeam Backup has caused this issue, but there are NO Veeam files that you need to delete.

You need to resolve the snapshot, datastore FULL issue.
BarepAssetsSys Admin

Author

Commented:
i got the server changed memory this gave enough space i migrated one of the storage to another datastore but still i have 2.95 gb users are using i can see it slow dropping in space

In snapshot manager there is no snapshots to delete???
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
This can happen that the snapshot is not listed in snapshot manager.

Check the folder datastore for files that end -00000x.vmdk, where x is 1,2,3 etc

You will not slow down the drop in datastore usage as the VM writes to the delta file, snapshot file, and the rate of change of the VM, is caused by user access.
BarepAssetsSys Admin

Author

Commented:
this is true i have whole load of files with x.vmdk  we now have 20 gb on the drive so i have a bit of time i need to apply the snapshots i tried creating a new snapshot and applying but would not let me
i am looking at maybe vmware convertor to another vmware machine in the  DR as believe it applys snapshot but any time i try convert it will not pick up my 1.2 tb drive it picks up the other 2 smaller drives ok does convertor have 1 tb limit ?? any other ideas on what i could do
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
You will not have enough space, to delete and committ/merge the snapshots.

You will need to use VMware Converter to Backup and Restore. VMware Converter does not use snapshots.

What type of drive is the 1.2TB drive?
BarepAssetsSys Admin

Author

Commented:
it is just windows 2008 r 2  file drive.. on raid 6 on sas array  
BarepAssetsSys Admin

Author

Commented:
could i possible plug a usb drive directly into the server add as a lun and do an extend on the current data store?? But i am already close on my 2 tb limit how much space do i need tfor the snaps applyinh
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
you cannot add datastores using USB external disks.
BarepAssetsSys Admin

Author

Commented:
is there a limit with convertor with disk sizes ??
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
No, but what format is this disk?

simple or dynamic, and you are using v5?
BarepAssetsSys Admin

Author

Commented:
it is basic and v4 thanks for all your help on this...at the moment i rang vmware they are getting me to try shutting down the vm and then try creating a snapshot then delete all i will give it go i tried it switched on but no joy i need to want till out of hours running out of space fairly fast

Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
if you have no space on the datastore, the Snapshot delete will fail.
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
How much memory is allocated to this VM?

How much space is left on the datastore, when the VM is off?
BarepAssetsSys Admin

Author

Commented:
i have 14 gb free  we managed to move a disk to another storage but the other disks i cant move as i made oversight did not set up the block size to support the size of the disk at the moment free and i have 3 gb allocated memory
Andrew Hancock (VMware vExpert / EE Fellow)VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017

Commented:
and how large are the snapshots?
BarepAssetsSys Admin

Author

Commented:
hi we got the issue resolved we do clone of the server to another storage location.  I am still unsure how the snaps go so big they were 500 gb in size we are small enough here less than 100 users
typical change is couple gig a night it is strange??

What is your recommended space for snapshots ???
VMware and Virtualization Consultant
CERTIFIED EXPERT
Fellow
Expert of the Year 2017
Commented:
we work on +20% of the Virtual Disk.

It would depend how long the virtual machine was writing to a snapshot.
BarepAssetsSys Admin

Author

Commented:
Excellent assistance

Explore More ContentExplore courses, solutions, and other research materials related to this topic.