Solved

VMWare: How to prevent 'out of space' error making a backup with veeam?

Posted on 2010-11-24
11
4,231 Views
Last Modified: 2012-05-10
Hello everybody.

I have a VM of 300GB in a Datastore with a capacity of 305 GB, so there are only 5 GB free. I am using Veeam as Backup software.Its working directory is in another datastore with 300+ GB free.

I thought that having 5GB of free space in the VM's datastore was enough for the configuration files and that stuff, but maybe I was wrong. The problem happened when I found the following error last Monday:


[msg.hbacommon.outofspace] There is no more space for virtual disk myvm-000001.vmdk.  You
Nov 21 01:00:34.073: vcpu-0| may be able to continue this session by freeing disk space on
Nov 21 01:00:34.073: vcpu-0| the relevant partition, and clicking Retry. Otherwise,
Nov 21 01:00:34.073: vcpu-0| click Abort to terminate this session.

Taking a look at the logs, it seems that Veeam had just made a snapshot, but in the very same datastore of the VM, not in the working directory condfigured in the VM, where there is more than enough space free.

Why was that snapshot created in the VM's datastored when its working directory was configured to be in other datastore?


Nov 20 22:00:06.912: vmx| SnapshotVMX done with snapshot 'VEEAM BACKUP TEMPORARY SNAPSHOT': 27
Nov 20 22:00:06.912: vmx| Vix: [105074 vmxCommands.c:2353]: VMAutomationCreateSnapshotCallback: Got CreateSnapshot callback, snapshotErr = 0, UID = 27
Nov 20 22:00:07.241: vcpu-1| HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/4beada54-cca8b737-8c6c-f4ce46bcaa39/myvm_1/myvm-000001.vmdk'
Nov 20 22:00:07.347: vcpu-1| DISKLIB-CHAIN : UpdateContentID: old = 0x1c0a9282, new = 0x7c25aa6d
Nov 20 22:00:07.347: vcpu-1| DISKLIB-DDB   : "longContentID" = "e605c5b301ba16c949cc573c7c25aa6d" (was "15037ad78fd1b288bd4bceba1c0a9282")
Nov 20 22:00:07.437: vcpu-1| DISKLIB-CHAIN : UpdateContentID: updated long content id
Nov 21 01:00:34.070: vcpu-0| Msg_Question:
Nov 21 01:00:34.073: vcpu-0| [msg.hbacommon.outofspace] There is no more space for virtual disk myvm-000001.vmdk.  You
Nov 21 01:00:34.073: vcpu-0| may be able to continue this session by freeing disk space on
Nov 21 01:00:34.073: vcpu-0| the relevant partition, and clicking Retry. Otherwise,
Nov 21 01:00:34.073: vcpu-0| click Abort to terminate this session.
Nov 21 01:00:34.073: vcpu-0| ----------------------------------------

How can we prevent this situation from happening again?

Thanks in advance,
    Gregorio Diaz
0
Comment
Question by:McGregor09
  • 3
  • 3
  • 2
  • +2
11 Comments
 
LVL 23

Expert Comment

by:Luciano Patrão
ID: 34205031
Hi

Veeam always create a temporarily snaphot for the backups. After the backups will delete the snapshots.

So yes you need to have Storage space to hold this temporarily snapshots for your backups.

And if you get out of storage space, the VM stop working.

Jail
0
 
LVL 42

Expert Comment

by:paulsolov
ID: 34205293
Also keep in mind that the amount of memory in your VM is the amount that your VM will create a swap file if you don't reserver the memory. If you have plenty of memory try to reserve 4GB RAM for example, this will reduce your swap file by 4GB and provide this space for the snaphsot.  You should always leave at least 15GB of space on the datastores.
0
 

Author Comment

by:McGregor09
ID: 34205649
Hi Paulsolov,

The VM has always had 4GB of RAM assigned, no problem with that.
Now, what I have done is changed the configuration of the VM. Let's say that I have the VM in Datastore A. I have set the working directory to another datastore, B.  And, just in case this was not enough, I set the swapfile directory in another datastore C. Now, B and C have  about 300GB free space, and A just 5.

The thing is that shrinking the size of my VM will take me some days, and meanwhile I want to make sure there is plenty of space for everything.

As I have done this, do you think I have to worry about this error taking place again?
Are those 15GB really necessary? Which VMWare processes do take up that space?

Thanks again
0
Guide to Performance: Optimization & Monitoring

Nowadays, monitoring is a mixture of tools, systems, and codes—making it a very complex process. And with this complexity, comes variables for failure. Get DZone’s new Guide to Performance to learn how to proactively find these variables and solve them before a disruption occurs.

 
LVL 42

Expert Comment

by:paulsolov
ID: 34206252
15GB is best practice.  You lose 5% of storage for the vmdk wrapper around the OS volumes, if you thin provision it will kill you.  

If you're using snapshots you need min. of 10GB free space but it all depends on how much in changes will be done during the backup.  

0
 
LVL 23

Expert Comment

by:Luciano Patrão
ID: 34207596
Hi

Yes paulsolov is correct, but snapshots with the Veeam incremental will not take much Storage, but sill you will always need the extra space for snapshots backup and also for normal VM management(swap etc)

Jail
0
 
LVL 27

Expert Comment

by:Tolomir
ID: 34208191
As said in the soon  to be deleted question. A snapshot can grow to the original size of the harddisk.

Of cause you do the backup off hours still you should reserve some space paulsolov's 5% would be around 50 GB to be reserved.

We could discuss the point if it is possible to keep working files and / or snapshots in a separate datastore or if all those file have to be kept in one datastore.

Tolomir
0
 
LVL 27

Accepted Solution

by:
Tolomir earned 63 total points
ID: 34208316
Just as reference in this question 15% of free space was accepted.

http://www.experts-exchange.com/Software/VMWare/Q_26636453.html?cid=1576

0
 

Author Comment

by:McGregor09
ID: 34211009
Hi all,

thanks for the answers!

I see, so around 15% is necessary. Just to make it 110% clear, even in my case, I have set the working and swap directories in another two datastores, should I have that 15% free space in the VM's datastore?

0
 
LVL 42

Assisted Solution

by:paulsolov
paulsolov earned 62 total points
ID: 34212831
With the swap file on a different partition it gives you more space when you start up the VM (since it doesn't create the swap file on the same volume).  You still need space for the snapshots so 10-15 percent would be min. depending on how much your snapshots grow during backups.  Also ensure that you're not thin provisioning.
0
 
LVL 17

Expert Comment

by:Gerald Connolly
ID: 34213897
An old rule of thumb for snapshot sizing is that 80% of writes go to 20% of the disk, so you should plan on at least 20% but YMMV
0
 

Author Closing Comment

by:McGregor09
ID: 34217517
Thanks for the detailed information. I see I have to invest a little bit more on my system  :)
0

Featured Post

DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

If we need to check who deleted a Virtual Machine from our vCenter. Looking this task in logs can be painful and spend lot of time, so the best way to check this is in the vCenter DB. Just connect to vCenter DB(default DB should be VCDB and using…
In this article, I show you step by step with screenshots to assist you - HOW TO: Deploy and Install the VMware vCenter Server Appliance 6.5 (VCSA 6.5), with some helpful tips along the way.
This tutorial will walk an individual through the process of installing of Data Protection Manager on a server running Windows Server 2012 R2, including the prerequisites. Microsoft .Net 3.5 is required. To install this feature, go to Server Manager…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial

685 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question