Solved

VMWare: How to prevent 'out of space' error making a backup with veeam?

Posted on 2010-11-24
11
3,661 Views
Last Modified: 2012-05-10
Hello everybody.

I have a VM of 300GB in a Datastore with a capacity of 305 GB, so there are only 5 GB free. I am using Veeam as Backup software.Its working directory is in another datastore with 300+ GB free.

I thought that having 5GB of free space in the VM's datastore was enough for the configuration files and that stuff, but maybe I was wrong. The problem happened when I found the following error last Monday:


[msg.hbacommon.outofspace] There is no more space for virtual disk myvm-000001.vmdk.  You
Nov 21 01:00:34.073: vcpu-0| may be able to continue this session by freeing disk space on
Nov 21 01:00:34.073: vcpu-0| the relevant partition, and clicking Retry. Otherwise,
Nov 21 01:00:34.073: vcpu-0| click Abort to terminate this session.

Taking a look at the logs, it seems that Veeam had just made a snapshot, but in the very same datastore of the VM, not in the working directory condfigured in the VM, where there is more than enough space free.

Why was that snapshot created in the VM's datastored when its working directory was configured to be in other datastore?


Nov 20 22:00:06.912: vmx| SnapshotVMX done with snapshot 'VEEAM BACKUP TEMPORARY SNAPSHOT': 27
Nov 20 22:00:06.912: vmx| Vix: [105074 vmxCommands.c:2353]: VMAutomationCreateSnapshotCallback: Got CreateSnapshot callback, snapshotErr = 0, UID = 27
Nov 20 22:00:07.241: vcpu-1| HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/4beada54-cca8b737-8c6c-f4ce46bcaa39/myvm_1/myvm-000001.vmdk'
Nov 20 22:00:07.347: vcpu-1| DISKLIB-CHAIN : UpdateContentID: old = 0x1c0a9282, new = 0x7c25aa6d
Nov 20 22:00:07.347: vcpu-1| DISKLIB-DDB   : "longContentID" = "e605c5b301ba16c949cc573c7c25aa6d" (was "15037ad78fd1b288bd4bceba1c0a9282")
Nov 20 22:00:07.437: vcpu-1| DISKLIB-CHAIN : UpdateContentID: updated long content id
Nov 21 01:00:34.070: vcpu-0| Msg_Question:
Nov 21 01:00:34.073: vcpu-0| [msg.hbacommon.outofspace] There is no more space for virtual disk myvm-000001.vmdk.  You
Nov 21 01:00:34.073: vcpu-0| may be able to continue this session by freeing disk space on
Nov 21 01:00:34.073: vcpu-0| the relevant partition, and clicking Retry. Otherwise,
Nov 21 01:00:34.073: vcpu-0| click Abort to terminate this session.
Nov 21 01:00:34.073: vcpu-0| ----------------------------------------

How can we prevent this situation from happening again?

Thanks in advance,
    Gregorio Diaz
0
Comment
Question by:McGregor09
  • 3
  • 3
  • 2
  • +2
11 Comments
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 34205031
Hi

Veeam always create a temporarily snaphot for the backups. After the backups will delete the snapshots.

So yes you need to have Storage space to hold this temporarily snapshots for your backups.

And if you get out of storage space, the VM stop working.

Jail
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 34205293
Also keep in mind that the amount of memory in your VM is the amount that your VM will create a swap file if you don't reserver the memory. If you have plenty of memory try to reserve 4GB RAM for example, this will reduce your swap file by 4GB and provide this space for the snaphsot.  You should always leave at least 15GB of space on the datastores.
0
 

Author Comment

by:McGregor09
ID: 34205649
Hi Paulsolov,

The VM has always had 4GB of RAM assigned, no problem with that.
Now, what I have done is changed the configuration of the VM. Let's say that I have the VM in Datastore A. I have set the working directory to another datastore, B.  And, just in case this was not enough, I set the swapfile directory in another datastore C. Now, B and C have  about 300GB free space, and A just 5.

The thing is that shrinking the size of my VM will take me some days, and meanwhile I want to make sure there is plenty of space for everything.

As I have done this, do you think I have to worry about this error taking place again?
Are those 15GB really necessary? Which VMWare processes do take up that space?

Thanks again
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 34206252
15GB is best practice.  You lose 5% of storage for the vmdk wrapper around the OS volumes, if you thin provision it will kill you.  

If you're using snapshots you need min. of 10GB free space but it all depends on how much in changes will be done during the backup.  

0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 34207596
Hi

Yes paulsolov is correct, but snapshots with the Veeam incremental will not take much Storage, but sill you will always need the extra space for snapshots backup and also for normal VM management(swap etc)

Jail
0
Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

 
LVL 27

Expert Comment

by:Tolomir
ID: 34208191
As said in the soon  to be deleted question. A snapshot can grow to the original size of the harddisk.

Of cause you do the backup off hours still you should reserve some space paulsolov's 5% would be around 50 GB to be reserved.

We could discuss the point if it is possible to keep working files and / or snapshots in a separate datastore or if all those file have to be kept in one datastore.

Tolomir
0
 
LVL 27

Accepted Solution

by:
Tolomir earned 63 total points
ID: 34208316
Just as reference in this question 15% of free space was accepted.

http://www.experts-exchange.com/Software/VMWare/Q_26636453.html?cid=1576

0
 

Author Comment

by:McGregor09
ID: 34211009
Hi all,

thanks for the answers!

I see, so around 15% is necessary. Just to make it 110% clear, even in my case, I have set the working and swap directories in another two datastores, should I have that 15% free space in the VM's datastore?

0
 
LVL 42

Assisted Solution

by:paulsolov
paulsolov earned 62 total points
ID: 34212831
With the swap file on a different partition it gives you more space when you start up the VM (since it doesn't create the swap file on the same volume).  You still need space for the snapshots so 10-15 percent would be min. depending on how much your snapshots grow during backups.  Also ensure that you're not thin provisioning.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 34213897
An old rule of thumb for snapshot sizing is that 80% of writes go to 20% of the disk, so you should plan on at least 20% but YMMV
0
 

Author Closing Comment

by:McGregor09
ID: 34217517
Thanks for the detailed information. I see I have to invest a little bit more on my system  :)
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Will try to explain how to use the VMware feature TAGs in the VMs and create Veeam Backup Jobs using TAGs. Since this article is too long, I will create second article for the Veeam tasks.
In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
In this Micro Tutorial viewers will learn how they can get their files copied out from their unbootable system without need to use recovery services. As an example non-bootable Windows 2012R2 installation is used which has boot problems.
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now