VMWare: How to prevent 'out of space' error making a backup with veeam?

Hello everybody.

I have a VM of 300GB in a Datastore with a capacity of 305 GB, so there are only 5 GB free. I am using Veeam as Backup software.Its working directory is in another datastore with 300+ GB free.

I thought that having 5GB of free space in the VM's datastore was enough for the configuration files and that stuff, but maybe I was wrong. The problem happened when I found the following error last Monday:


[msg.hbacommon.outofspace] There is no more space for virtual disk myvm-000001.vmdk.  You
Nov 21 01:00:34.073: vcpu-0| may be able to continue this session by freeing disk space on
Nov 21 01:00:34.073: vcpu-0| the relevant partition, and clicking Retry. Otherwise,
Nov 21 01:00:34.073: vcpu-0| click Abort to terminate this session.

Taking a look at the logs, it seems that Veeam had just made a snapshot, but in the very same datastore of the VM, not in the working directory condfigured in the VM, where there is more than enough space free.

Why was that snapshot created in the VM's datastored when its working directory was configured to be in other datastore?


Nov 20 22:00:06.912: vmx| SnapshotVMX done with snapshot 'VEEAM BACKUP TEMPORARY SNAPSHOT': 27
Nov 20 22:00:06.912: vmx| Vix: [105074 vmxCommands.c:2353]: VMAutomationCreateSnapshotCallback: Got CreateSnapshot callback, snapshotErr = 0, UID = 27
Nov 20 22:00:07.241: vcpu-1| HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/4beada54-cca8b737-8c6c-f4ce46bcaa39/myvm_1/myvm-000001.vmdk'
Nov 20 22:00:07.347: vcpu-1| DISKLIB-CHAIN : UpdateContentID: old = 0x1c0a9282, new = 0x7c25aa6d
Nov 20 22:00:07.347: vcpu-1| DISKLIB-DDB   : "longContentID" = "e605c5b301ba16c949cc573c7c25aa6d" (was "15037ad78fd1b288bd4bceba1c0a9282")
Nov 20 22:00:07.437: vcpu-1| DISKLIB-CHAIN : UpdateContentID: updated long content id
Nov 21 01:00:34.070: vcpu-0| Msg_Question:
Nov 21 01:00:34.073: vcpu-0| [msg.hbacommon.outofspace] There is no more space for virtual disk myvm-000001.vmdk.  You
Nov 21 01:00:34.073: vcpu-0| may be able to continue this session by freeing disk space on
Nov 21 01:00:34.073: vcpu-0| the relevant partition, and clicking Retry. Otherwise,
Nov 21 01:00:34.073: vcpu-0| click Abort to terminate this session.
Nov 21 01:00:34.073: vcpu-0| ----------------------------------------

How can we prevent this situation from happening again?

Thanks in advance,
    Gregorio Diaz
McGregor09Asked:
Who is Participating?
 
TolomirAdministratorCommented:
Just as reference in this question 15% of free space was accepted.

http://www.experts-exchange.com/Software/VMWare/Q_26636453.html?cid=1576

0
 
Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:
Hi

Veeam always create a temporarily snaphot for the backups. After the backups will delete the snapshots.

So yes you need to have Storage space to hold this temporarily snapshots for your backups.

And if you get out of storage space, the VM stop working.

Jail
0
 
Paul SolovyovskySenior IT AdvisorCommented:
Also keep in mind that the amount of memory in your VM is the amount that your VM will create a swap file if you don't reserver the memory. If you have plenty of memory try to reserve 4GB RAM for example, this will reduce your swap file by 4GB and provide this space for the snaphsot.  You should always leave at least 15GB of space on the datastores.
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
McGregor09Author Commented:
Hi Paulsolov,

The VM has always had 4GB of RAM assigned, no problem with that.
Now, what I have done is changed the configuration of the VM. Let's say that I have the VM in Datastore A. I have set the working directory to another datastore, B.  And, just in case this was not enough, I set the swapfile directory in another datastore C. Now, B and C have  about 300GB free space, and A just 5.

The thing is that shrinking the size of my VM will take me some days, and meanwhile I want to make sure there is plenty of space for everything.

As I have done this, do you think I have to worry about this error taking place again?
Are those 15GB really necessary? Which VMWare processes do take up that space?

Thanks again
0
 
Paul SolovyovskySenior IT AdvisorCommented:
15GB is best practice.  You lose 5% of storage for the vmdk wrapper around the OS volumes, if you thin provision it will kill you.  

If you're using snapshots you need min. of 10GB free space but it all depends on how much in changes will be done during the backup.  

0
 
Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:
Hi

Yes paulsolov is correct, but snapshots with the Veeam incremental will not take much Storage, but sill you will always need the extra space for snapshots backup and also for normal VM management(swap etc)

Jail
0
 
TolomirAdministratorCommented:
As said in the soon  to be deleted question. A snapshot can grow to the original size of the harddisk.

Of cause you do the backup off hours still you should reserve some space paulsolov's 5% would be around 50 GB to be reserved.

We could discuss the point if it is possible to keep working files and / or snapshots in a separate datastore or if all those file have to be kept in one datastore.

Tolomir
0
 
McGregor09Author Commented:
Hi all,

thanks for the answers!

I see, so around 15% is necessary. Just to make it 110% clear, even in my case, I have set the working and swap directories in another two datastores, should I have that 15% free space in the VM's datastore?

0
 
Paul SolovyovskySenior IT AdvisorCommented:
With the swap file on a different partition it gives you more space when you start up the VM (since it doesn't create the swap file on the same volume).  You still need space for the snapshots so 10-15 percent would be min. depending on how much your snapshots grow during backups.  Also ensure that you're not thin provisioning.
0
 
Gerald ConnollyCommented:
An old rule of thumb for snapshot sizing is that 80% of writes go to 20% of the disk, so you should plan on at least 20% but YMMV
0
 
McGregor09Author Commented:
Thanks for the detailed information. I see I have to invest a little bit more on my system  :)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.