Solved

How can I script/automate VMWare backups to disk (NFS)?

Posted on 2009-04-09
10
3,023 Views
Last Modified: 2013-12-19
Ok, I am trying to get a full handle on the VMWare thing and have 3 VM Host servers up and running and using Virtual Center etc.  It's quite nice.

I want to backup to disk cheaply, so I thought FreeNAS would be perfect for this, so I setup a cheap FreeNAS box, from parts test with.  I have attached the FreeNAS as a DataStore on each of my VM Host servers, now...here's the question.

How can I automate VMDK backups to the NFS datastore?  I've been reading so much VCB documentation that my head hurts.  Any ideas?

Also, I'm excellent with MS products, my *NIX skills are just so-so - mainly rusty from the old days.

Thanks!!

Chris
0
Comment
Question by:ShopLiftin
10 Comments
 
LVL 7

Expert Comment

by:kumarnirmal
ID: 24110807
I suggest that you take a look at this product - http://www.trilead.com/Products/VM_Explorer/
0
 
LVL 22

Expert Comment

by:Luciano Patrão
ID: 24113547
Hi

You can use VCB, or other scripts to backup your VMware

For VCB checj this article

http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1277118_mem1,00.html?track=sy421

Examples of scripts backup, and very easy to work with.

http://engineering.xtravirt.com/products/phd-technologies/visbu.html

and

http://www.vmts.net/vmbk.htm

Jail
0
 
LVL 8

Expert Comment

by:markzz
ID: 24116987
Before suggesting a backup method or what you are asking for is a scripted routine could you explain what type of VM licensing you have.
Quite simply put, if you are licensed for VCB you should leverage of it.
Its a very powerful tool. Of course you can backup to OPenfiler or OPenNAS.
In VM there is much more to consider than just backing up..I guess where most go wrong is they can't successfully recover.
0
 

Author Comment

by:ShopLiftin
ID: 24120365
Ok,  I guess more detail is needed here.

I have 3 VM Hosts using a Macromedia Flex license server on a Virtual Center Server.  I also do have VCB licensing on each host server.  All 3 VM Hosts attach via 2 GB Ethernet connections to an EMC Celerra NS20 on a separate VLAN using NFS.  I only have 5 VM's per host for load balancing and DR purposes.

I am currently using Symantec Backup Exec (I'm still on 11d) also running on the VC Server attached to a LTO3 Tape Library.

I backup the data using Symantex BE via agents on the member servers.

What I am asking here, is can I ALSO backup the VMDK's to disk, possibly once or twice a week to a NAS server, ie, a FreeNAS setup, with 5 X 1.5TB SATA Drives so that in the event of a disaster on the EMC Celerra end, I could run critical servers also via NFS off of the FreeNAS box while repairing the Celerra.

So the idea would be that I could recover servers quickly on an alternate NFS (because ideally I would have separate copies of the VMDK's that I could start from)...then I could recover up to date data files from the LTO3's.  Make sense so far?

I have created a test FreeNAS box with the 5 X 1.5TB drives and I'm getting 5.45TB useable space.  I have the FreeNAS on the VLAN and added to each VM host as another datastore called "nas1".  (Also, I built this 7TB total - 5.45TB in RAID5 FreeNAS box for less than $800 using an ASUS mobo with 5 SATA ports, a ThermalTake Case + PS, a LV Sempron Processor, 2GB Patriot RAM and an Intel NIC, the SeaGates were only $76 each, I installed FreeNAS to a $6 USB Stick)

I can't believe that there aren't just straightforward ways to do snapshots to an alternate datastore.

Markzz:  I hope I've provided enough info, because I was reading a 28 page VMWare communities board on the topic and it was making me ill, and it was also unfinished and recently posted to...

Kumar and BestWay, I'm looking into those links now.

-Chris
0
 
LVL 8

Expert Comment

by:markzz
ID: 24120891
There are a number of methods to copy a VM, but to copy them while they are running is not such a simple task.
I guess one of my concerns is if you head down a particular direction like I did then realise you just don't have the comprehensive skills to support a routine written in a Lunix envronment, it can be such an enormous waste of time and resources.
I would encourage you to firstly leverage your VCB lincenses to ensure you have complete and recoverable backups.
Regarding the question of getting a copy to another VMFS store I'd suggest the simplest routine would be to perform a scheduled Clone.
It's very simple to setup a Schedule via the VC, one of the options is to performa a clone. These can be done live if you are on 3.5u2 or higher.
There's always a catch. If you don't want multiple version on disk you will beed to kick of a script to delete you previous clone. Pretty straight forward realy, I would do this from your VC via the remote CLI.

As I say this is still quite an untidy routine by my way of thinking. There are other alternatives but they all cost $$
I'd suggest you also look at vRanger (esx Ranger) it's a very easy and comprehensive backup and restore routine and can perform backups at a block level so you are only moving changed blocks, it also has VCB type backup functionallity but again effectively giving you incremental backups. Lets not get into the incremental verse full backup methods, I'm just highlighting the functions.
To start with.
Have a fiddle with the VC's Scheduling functions if your happy with that then have a read about the remote CLI.
There are esx specific CLI commands to remove and manipulate your vmdk's but as these systems are not powered on I expect the conventional Linux Commands will be fine
eg. cd to your VMFS
rm -rf /foldername
 
0
Don't lose your head updating email signatures!

Do your end users still have the wrong email signature? Do email signature updates bore you or fill you with a sense of dread? You can make this a whole lot easier on yourself by trusting an Exclaimer email signature management solution. Over 50 million users do...so should you!

 
LVL 8

Expert Comment

by:markzz
ID: 24122093
To add a little more
I have tested some of the alternative solutions. eg, VM Explorer, FastSCP, WinSCP.
I use WinSCP and FastSCP for moving small amounts of data to and from the ESX servers.
An example of the expected performance.
I can backup a running guest of 150GB with VCB in about 75 minutes. I tried the same thing with VM Explorer it took 48 hours. Or FastSCP took 12 hours. It's far to add that VCB's in my case are over 4Gb FC where as the other mentioned routines were of course over the network at 100Mb.
It's best to do your own testing but the point here is although there are many possible backup or replication routines few are robust enough or fast enough to implement into a production environment.
0
 

Author Comment

by:ShopLiftin
ID: 24129769
Ok, I'm looking Markzz here, and I'll give it a go.

I'm not completely useless with Linux, I'm just rusty.  I was only making comment that backing up files on VMWare was not finalized yet.  No open file option just exists in the system.  Storing changes in a delta file while the copy takes place, that sort of thing.

Mark, let me try scheduling the deletion of clones using the RCLI and scheduling the cloning of VMs.

This sounds like a great option, I'll post back here later today.

Chris
0
 

Author Comment

by:ShopLiftin
ID: 24140376
Markzz,

I get the error when I run the scheduled clone, "The attempted operation cannot be performed in the current state (Powered On)."

Ok, I have looked into Perl a little bit and using the RCLI scripts isn't looking so difficult.

I have actually tried someone elses script but they (the automated ones) don't account for a couple of things, such as:
  1. VM Names with spaces in them.
  2. VM Configurations with VMDK's on different datastores.

I am creating/tweaking a script not that will backup a SINGLE VM - with or without spaces
- You can specify the name of each VMDK and the datastore that it resides in.

It will create a snap shot, delete the exisiting copy, create a new copy, and then delete the snap.

The modifications are done in variables at the top.  I will post the script tomorrow or Thursday after I've tested.

Chris
0
 

Accepted Solution

by:
ShopLiftin earned 0 total points
ID: 24144883
Ok, after some research and complete dissatisfaction with the VCB, here was my solution.

I modified a script, I created 1 version for each VM that I have (18) and I created 18 Scheduled tasks that recur to make full VMDK backups of my environment.  The script requires an installation of VMWare RCLI, I installed it on my VI Server.

Here' the Perl script for my backup routine - borrowed & modified before I got it an made some changes of my own.

Chris
#  This was posted on VMWare Communities originally and the idea and actual work part of the code was written/modified and now modified again

#  I think I am the second modifier of this code, but I can't be certain.  All the original credits are still left below.

# - Chris Schimanski - modifier #2+

#This Script is designed to create a backup of just 1 Virtual Machine.  I modified this one because I have VM's with VMDK's on separate datastores

#and because of the renaming on some VMs in my environment, I needed to have a few extra variables to handle these needs.
 

#You must set some variables below and pass your root (or backup user account) password into the script as an argument.

#  I M P O R T A N T : You are using linux here, folders, files, etc are CASE sensitive.

#    F I N D   T H E   ' CHANGE HERE ' tags below

#

# ESXi 3.5 Backup Script

#

# Autor: Christian Gruetzner

#

#

#############################################################################

#

#2008-08-22  V 1.0 Basic Script

#            Functions:     - create snapshots of all local VM's

#                           - get files from ESX-Datastore to local Store

#                           - exclude unneccessary files from backup

#                           - remove all created snapshots

#                           - actual time in log

#

#############################################################################

#

# Modified by Stephen Maquiso (tipoytm@hotmail.com) 11/6/08
 

#Array declarations...

my @DSNames;

my @Folders;

my @DestPaths;
 
 

# ---------CHANGE HERE------------ ESX HOST IP! Not your VI Server

my $url = "https://192.168.##.###:443/sdk/vimService";  #URL to your ESX Host, default: <https://IP-Address:Port/sdk/vimService>

# ---------CHANGE HERE------------ or use root

my $username = "root";                                  #Username

# You can just assign the password here or pass it as an argument.  I don't like storing the password in a script text file.

$argument = shift;

my $password = $argument;                               #User password

# ---------CHANGE HERE------------ or not, it's just a name.

my $snapshotname = "BackupSnap";                        #Name of your Snapshot

# ---------CHANGE HERE------------ or not, check to make sure it's valid on your VI Server

my $RCLIPath = "C:/Progra~1/VMware/VMWare~2";           #VI Remote CLI Path (Windows: Use ONLY Short Folder Names!!!!)

# ---------CHANGE HERE------------ Enter the name of the VM as it appears in your VI Client

my $VirtualMachineName = "\"VMNameHere\"";              #If the name has spaces, enclose the name in quotes /"name/"

# ---------CHANGE HERE------------ Enter the Data Store Name in the brackets

$DSNames[0] = "[DataStoreNameHere]";

# ---------CHANGE HERE------------ This is USUALLY the same as the VM Name, but if the folder on the datastore is different, enter it here

$Folders[0] = "VMFolderNameHere";

# ---------CHANGE HERE------------ Enter the TARGET DATASTORE in the Brackets and the TARGET FOLDER after the / below.  This folder must exist but this can be altered to use the vifs.pl --mkdir to create it.

$DestPaths[0] = "[TargetDataStoreNameHere]/TargetFolderNameHere";
 

# Uncomment to specify a second Datastore/VMDK Folder if the VM has disks on separate datastores.

# Copy it to handle more datastore/folders.

# ---------CHANGE HERE------------ Enter the Data Store Name in the brackets

#$DSNames[0] = "[DataStoreNameHere]";

# ---------CHANGE HERE------------ This is USUALLY the same as the VM Name, but if the folder on the datastore is different, enter it here

#$Folders[0] = "VMFolderNameHere";

# ---------CHANGE HERE------------ Enter the TARGET DATASTORE in the Brackets and the TARGET FOLDER after the / below.  This folder must exist but this can be altered to use the vifs.pl --mkdir to create it.

#$DestPaths[0] = "[TargetDataStoreNameHere]/TargetFolderNameHere";
 

#******************************************************************************************************************************************************

#********************************** F O R   G E N E R A L   U S E   Y O U   S H O U L D N ' T   M O D I F Y   B E L O W **********************************************

#************************************************** E N D   S E C T I O N *********************************************************************************
 
 

#************************************************** N E W   S E C T I O N ********************************************************************************

#**************** This Section Creates the snapshot for this Virtual Machine ******** ***************************************************************************

#******************************************************************************************************************************************************

&actualtime();

print " ***** Script Start *************************\n\n";

&actualtime();

print " ----- Create Snapshot of the VM - must be running -----";

print "\n\n";

system("perl $RCLIPath/Perl/apps/vm/snapshotmanager.pl --url $url --username $username --password $password --operation create --powerstatus poweredOn --snapshotname $snapshotname --vmname $VirtualMachineName");

print "\n\n";

#******************************************************************************************************************************************************

#************************************************** E N D   S E C T I O N *********************************************************************************
 
 

#************************************************** N E W   S E C T I O N ********************************************************************************

#**************** This Section Copies the relevant VM Files to the specified datastore ***************************************************************************

#******************************************************************************************************************************************************

&actualtime();

print " ----- Copy VM files for $VirtualMachineName -----";

print "\n\n";
 

#run as long the cache array has data and save the value everytime in $filename

my $i = 0;

foreach (@DSNames)

{

    #read all available files and save filenames in the cache-array

    my @cache = `perl $RCLIPath/bin/vifs.pl --url $url --username $username --password $password --dir \"$DSNames[$i] $Folders[$i]\"`;

    foreach my $filename (@cache)

    {

        #exclude uninterresting files from backup to save backup space

        if($filename !~ /.log/ && $filename !~ /.vswp/ && $filename !~ /.vmsn/ && $filename !~ /-delta/)

        {

            #remove the "\n" at the end of $filename to prevent a error massage in log

            chomp($filename);

            

            #For readability, combine the datastore, folders, filenames and wrap in quotes.

            my $sourcefile = "\"$DSNames[$i]/$Folders[$i]/$filename\"";

            my $targetfile = "\"$DestPaths[$i]/$filename\"";

            

            &actualtime();

            print " ----- Start Copy File: $filename \n\n";

            print ": Source: $sourcefile\n\n";

            print ": Target: $targetfile\n\n";

            print "Look for errors below here:\n\n";

            system("perl $RCLIPath/bin/vifs.pl --url $url --username $username --password $password --force --copy $sourcefile $targetfile");

            print "\n\n Moving to next file.\n\n";

        }

    }

    print "No more files, moving to next DataStore.\n\n";

    $i++;

}

print "No more DataStores.\n\n";

print "\n\n";

#******************************************************************************************************************************************************

#************************************************** E N D   S E C T I O N *********************************************************************************
 
 

#******************************************************************************************************************************************************

&actualtime();

#************************************************** N E W   S E C T I O N ********************************************************************************

#**************** This Section Removes the Snapshot and applies the delta. ************************************************************************************

#**************** You may want to change the removeall option here to just remove ******************************************************************************

#******************************************************************************************************************************************************

print " ----- Remove ALL Snapshots of the VM -----";

print "\n\n";

system("perl $RCLIPath/Perl/apps/vm/snapshotmanager.pl --url $url --username $username --password $password --operation removeall --powerstatus poweredOn --snapshotname $snapshotname --children 1 --vmname $VirtualMachineName");

$i++;

print "\n\n";

&actualtime();

print " ***** Script End ***************************";

#******************************************************************************************************************************************************

#************************************************** E N D   S E C T I O N *********************************************************************************
 
 

#sub function to print the actual time in the log

sub actualtime

{

	my ($Sekunden, $Minuten, $Stunden, $Monatstag, $Monat,

	    $Jahr, $Wochentag, $Jahrestag, $Sommerzeit) = localtime(time);

	my $CTIME_String = localtime(time);

	$Monat+=1;

	$Jahrestag+=1;

	$Monat = $Monat < 10 ? $Monat = "0".$Monat : $Monat;

	$Monatstag = $Monatstag < 10 ? $Monatstag = "0".$Monatstag : $Monatstag;

	$Stunden = $Stunden < 10 ? $Stunden = "0".$Stunden : $Stunden;

	$Minuten = $Minuten < 10 ? $Minuten = "0".$Minuten : $Minuten;

	$Sekunden = $Sekunden < 10 ? $Sekunden = "0".$Sekunden : $Sekunden;

	$Jahr+=1900;

	

	print "$Jahr-$Monat-$Monatstag $Stunden:$Minuten:$Sekunden";

}

Open in new window

0
 

Author Comment

by:ShopLiftin
ID: 24144917
Also, the script that BestWay pointed out did not run and backup my VM's, mainly because of the issues that I wrote into the solution script, spaced in folder names, VMDKs spread across datastores, etc.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
In this step by step tutorial with screenshots, we will show you HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 6.5 (ESXi 6.5). This is important if you need to enable SSH remote access for additional troubleshooting of the ESXi hos…
Teach the user how to join ESXi hosts to Active Directory domains Open vSphere Client: Join ESXi host to AD domain: Verify ESXi computer account in AD: Configure permissions for domain user in ESXi: Test domain user login to ESXi host:
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now