Solved

How can I script/automate VMWare backups to disk (NFS)?

Posted on 2009-04-09
10
3,064 Views
Last Modified: 2013-12-19
Ok, I am trying to get a full handle on the VMWare thing and have 3 VM Host servers up and running and using Virtual Center etc.  It's quite nice.

I want to backup to disk cheaply, so I thought FreeNAS would be perfect for this, so I setup a cheap FreeNAS box, from parts test with.  I have attached the FreeNAS as a DataStore on each of my VM Host servers, now...here's the question.

How can I automate VMDK backups to the NFS datastore?  I've been reading so much VCB documentation that my head hurts.  Any ideas?

Also, I'm excellent with MS products, my *NIX skills are just so-so - mainly rusty from the old days.

Thanks!!

Chris
0
Comment
Question by:ShopLiftin
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
10 Comments
 
LVL 7

Expert Comment

by:kumarnirmal
ID: 24110807
I suggest that you take a look at this product - http://www.trilead.com/Products/VM_Explorer/
0
 
LVL 23

Expert Comment

by:Luciano Patrão
ID: 24113547
Hi

You can use VCB, or other scripts to backup your VMware

For VCB checj this article

http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1277118_mem1,00.html?track=sy421

Examples of scripts backup, and very easy to work with.

http://engineering.xtravirt.com/products/phd-technologies/visbu.html

and

http://www.vmts.net/vmbk.htm

Jail
0
 
LVL 8

Expert Comment

by:markzz
ID: 24116987
Before suggesting a backup method or what you are asking for is a scripted routine could you explain what type of VM licensing you have.
Quite simply put, if you are licensed for VCB you should leverage of it.
Its a very powerful tool. Of course you can backup to OPenfiler or OPenNAS.
In VM there is much more to consider than just backing up..I guess where most go wrong is they can't successfully recover.
0
Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

 

Author Comment

by:ShopLiftin
ID: 24120365
Ok,  I guess more detail is needed here.

I have 3 VM Hosts using a Macromedia Flex license server on a Virtual Center Server.  I also do have VCB licensing on each host server.  All 3 VM Hosts attach via 2 GB Ethernet connections to an EMC Celerra NS20 on a separate VLAN using NFS.  I only have 5 VM's per host for load balancing and DR purposes.

I am currently using Symantec Backup Exec (I'm still on 11d) also running on the VC Server attached to a LTO3 Tape Library.

I backup the data using Symantex BE via agents on the member servers.

What I am asking here, is can I ALSO backup the VMDK's to disk, possibly once or twice a week to a NAS server, ie, a FreeNAS setup, with 5 X 1.5TB SATA Drives so that in the event of a disaster on the EMC Celerra end, I could run critical servers also via NFS off of the FreeNAS box while repairing the Celerra.

So the idea would be that I could recover servers quickly on an alternate NFS (because ideally I would have separate copies of the VMDK's that I could start from)...then I could recover up to date data files from the LTO3's.  Make sense so far?

I have created a test FreeNAS box with the 5 X 1.5TB drives and I'm getting 5.45TB useable space.  I have the FreeNAS on the VLAN and added to each VM host as another datastore called "nas1".  (Also, I built this 7TB total - 5.45TB in RAID5 FreeNAS box for less than $800 using an ASUS mobo with 5 SATA ports, a ThermalTake Case + PS, a LV Sempron Processor, 2GB Patriot RAM and an Intel NIC, the SeaGates were only $76 each, I installed FreeNAS to a $6 USB Stick)

I can't believe that there aren't just straightforward ways to do snapshots to an alternate datastore.

Markzz:  I hope I've provided enough info, because I was reading a 28 page VMWare communities board on the topic and it was making me ill, and it was also unfinished and recently posted to...

Kumar and BestWay, I'm looking into those links now.

-Chris
0
 
LVL 8

Expert Comment

by:markzz
ID: 24120891
There are a number of methods to copy a VM, but to copy them while they are running is not such a simple task.
I guess one of my concerns is if you head down a particular direction like I did then realise you just don't have the comprehensive skills to support a routine written in a Lunix envronment, it can be such an enormous waste of time and resources.
I would encourage you to firstly leverage your VCB lincenses to ensure you have complete and recoverable backups.
Regarding the question of getting a copy to another VMFS store I'd suggest the simplest routine would be to perform a scheduled Clone.
It's very simple to setup a Schedule via the VC, one of the options is to performa a clone. These can be done live if you are on 3.5u2 or higher.
There's always a catch. If you don't want multiple version on disk you will beed to kick of a script to delete you previous clone. Pretty straight forward realy, I would do this from your VC via the remote CLI.

As I say this is still quite an untidy routine by my way of thinking. There are other alternatives but they all cost $$
I'd suggest you also look at vRanger (esx Ranger) it's a very easy and comprehensive backup and restore routine and can perform backups at a block level so you are only moving changed blocks, it also has VCB type backup functionallity but again effectively giving you incremental backups. Lets not get into the incremental verse full backup methods, I'm just highlighting the functions.
To start with.
Have a fiddle with the VC's Scheduling functions if your happy with that then have a read about the remote CLI.
There are esx specific CLI commands to remove and manipulate your vmdk's but as these systems are not powered on I expect the conventional Linux Commands will be fine
eg. cd to your VMFS
rm -rf /foldername
 
0
 
LVL 8

Expert Comment

by:markzz
ID: 24122093
To add a little more
I have tested some of the alternative solutions. eg, VM Explorer, FastSCP, WinSCP.
I use WinSCP and FastSCP for moving small amounts of data to and from the ESX servers.
An example of the expected performance.
I can backup a running guest of 150GB with VCB in about 75 minutes. I tried the same thing with VM Explorer it took 48 hours. Or FastSCP took 12 hours. It's far to add that VCB's in my case are over 4Gb FC where as the other mentioned routines were of course over the network at 100Mb.
It's best to do your own testing but the point here is although there are many possible backup or replication routines few are robust enough or fast enough to implement into a production environment.
0
 

Author Comment

by:ShopLiftin
ID: 24129769
Ok, I'm looking Markzz here, and I'll give it a go.

I'm not completely useless with Linux, I'm just rusty.  I was only making comment that backing up files on VMWare was not finalized yet.  No open file option just exists in the system.  Storing changes in a delta file while the copy takes place, that sort of thing.

Mark, let me try scheduling the deletion of clones using the RCLI and scheduling the cloning of VMs.

This sounds like a great option, I'll post back here later today.

Chris
0
 

Author Comment

by:ShopLiftin
ID: 24140376
Markzz,

I get the error when I run the scheduled clone, "The attempted operation cannot be performed in the current state (Powered On)."

Ok, I have looked into Perl a little bit and using the RCLI scripts isn't looking so difficult.

I have actually tried someone elses script but they (the automated ones) don't account for a couple of things, such as:
  1. VM Names with spaces in them.
  2. VM Configurations with VMDK's on different datastores.

I am creating/tweaking a script not that will backup a SINGLE VM - with or without spaces
- You can specify the name of each VMDK and the datastore that it resides in.

It will create a snap shot, delete the exisiting copy, create a new copy, and then delete the snap.

The modifications are done in variables at the top.  I will post the script tomorrow or Thursday after I've tested.

Chris
0
 

Accepted Solution

by:
ShopLiftin earned 0 total points
ID: 24144883
Ok, after some research and complete dissatisfaction with the VCB, here was my solution.

I modified a script, I created 1 version for each VM that I have (18) and I created 18 Scheduled tasks that recur to make full VMDK backups of my environment.  The script requires an installation of VMWare RCLI, I installed it on my VI Server.

Here' the Perl script for my backup routine - borrowed & modified before I got it an made some changes of my own.

Chris
#  This was posted on VMWare Communities originally and the idea and actual work part of the code was written/modified and now modified again
#  I think I am the second modifier of this code, but I can't be certain.  All the original credits are still left below.
# - Chris Schimanski - modifier #2+
#This Script is designed to create a backup of just 1 Virtual Machine.  I modified this one because I have VM's with VMDK's on separate datastores
#and because of the renaming on some VMs in my environment, I needed to have a few extra variables to handle these needs.
 
#You must set some variables below and pass your root (or backup user account) password into the script as an argument.
#  I M P O R T A N T : You are using linux here, folders, files, etc are CASE sensitive.
#    F I N D   T H E   ' CHANGE HERE ' tags below
#
# ESXi 3.5 Backup Script
#
# Autor: Christian Gruetzner
#
#
#############################################################################
#
#2008-08-22  V 1.0 Basic Script
#            Functions:     - create snapshots of all local VM's
#                           - get files from ESX-Datastore to local Store
#                           - exclude unneccessary files from backup
#                           - remove all created snapshots
#                           - actual time in log
#
#############################################################################
#
# Modified by Stephen Maquiso (tipoytm@hotmail.com) 11/6/08
 
#Array declarations...
my @DSNames;
my @Folders;
my @DestPaths;
 
 
# ---------CHANGE HERE------------ ESX HOST IP! Not your VI Server
my $url = "https://192.168.##.###:443/sdk/vimService";  #URL to your ESX Host, default: <https://IP-Address:Port/sdk/vimService>
# ---------CHANGE HERE------------ or use root
my $username = "root";                                  #Username
# You can just assign the password here or pass it as an argument.  I don't like storing the password in a script text file.
$argument = shift;
my $password = $argument;                               #User password
# ---------CHANGE HERE------------ or not, it's just a name.
my $snapshotname = "BackupSnap";                        #Name of your Snapshot
# ---------CHANGE HERE------------ or not, check to make sure it's valid on your VI Server
my $RCLIPath = "C:/Progra~1/VMware/VMWare~2";           #VI Remote CLI Path (Windows: Use ONLY Short Folder Names!!!!)
# ---------CHANGE HERE------------ Enter the name of the VM as it appears in your VI Client
my $VirtualMachineName = "\"VMNameHere\"";              #If the name has spaces, enclose the name in quotes /"name/"
# ---------CHANGE HERE------------ Enter the Data Store Name in the brackets
$DSNames[0] = "[DataStoreNameHere]";
# ---------CHANGE HERE------------ This is USUALLY the same as the VM Name, but if the folder on the datastore is different, enter it here
$Folders[0] = "VMFolderNameHere";
# ---------CHANGE HERE------------ Enter the TARGET DATASTORE in the Brackets and the TARGET FOLDER after the / below.  This folder must exist but this can be altered to use the vifs.pl --mkdir to create it.
$DestPaths[0] = "[TargetDataStoreNameHere]/TargetFolderNameHere";
 
# Uncomment to specify a second Datastore/VMDK Folder if the VM has disks on separate datastores.
# Copy it to handle more datastore/folders.
# ---------CHANGE HERE------------ Enter the Data Store Name in the brackets
#$DSNames[0] = "[DataStoreNameHere]";
# ---------CHANGE HERE------------ This is USUALLY the same as the VM Name, but if the folder on the datastore is different, enter it here
#$Folders[0] = "VMFolderNameHere";
# ---------CHANGE HERE------------ Enter the TARGET DATASTORE in the Brackets and the TARGET FOLDER after the / below.  This folder must exist but this can be altered to use the vifs.pl --mkdir to create it.
#$DestPaths[0] = "[TargetDataStoreNameHere]/TargetFolderNameHere";
 
#******************************************************************************************************************************************************
#********************************** F O R   G E N E R A L   U S E   Y O U   S H O U L D N ' T   M O D I F Y   B E L O W **********************************************
#************************************************** E N D   S E C T I O N *********************************************************************************
 
 
#************************************************** N E W   S E C T I O N ********************************************************************************
#**************** This Section Creates the snapshot for this Virtual Machine ******** ***************************************************************************
#******************************************************************************************************************************************************
&actualtime();
print " ***** Script Start *************************\n\n";
&actualtime();
print " ----- Create Snapshot of the VM - must be running -----";
print "\n\n";
system("perl $RCLIPath/Perl/apps/vm/snapshotmanager.pl --url $url --username $username --password $password --operation create --powerstatus poweredOn --snapshotname $snapshotname --vmname $VirtualMachineName");
print "\n\n";
#******************************************************************************************************************************************************
#************************************************** E N D   S E C T I O N *********************************************************************************
 
 
#************************************************** N E W   S E C T I O N ********************************************************************************
#**************** This Section Copies the relevant VM Files to the specified datastore ***************************************************************************
#******************************************************************************************************************************************************
&actualtime();
print " ----- Copy VM files for $VirtualMachineName -----";
print "\n\n";
 
#run as long the cache array has data and save the value everytime in $filename
my $i = 0;
foreach (@DSNames)
{
    #read all available files and save filenames in the cache-array
    my @cache = `perl $RCLIPath/bin/vifs.pl --url $url --username $username --password $password --dir \"$DSNames[$i] $Folders[$i]\"`;
    foreach my $filename (@cache)
    {
        #exclude uninterresting files from backup to save backup space
        if($filename !~ /.log/ && $filename !~ /.vswp/ && $filename !~ /.vmsn/ && $filename !~ /-delta/)
        {
            #remove the "\n" at the end of $filename to prevent a error massage in log
            chomp($filename);
            
            #For readability, combine the datastore, folders, filenames and wrap in quotes.
            my $sourcefile = "\"$DSNames[$i]/$Folders[$i]/$filename\"";
            my $targetfile = "\"$DestPaths[$i]/$filename\"";
            
            &actualtime();
            print " ----- Start Copy File: $filename \n\n";
            print ": Source: $sourcefile\n\n";
            print ": Target: $targetfile\n\n";
            print "Look for errors below here:\n\n";
            system("perl $RCLIPath/bin/vifs.pl --url $url --username $username --password $password --force --copy $sourcefile $targetfile");
            print "\n\n Moving to next file.\n\n";
        }
    }
    print "No more files, moving to next DataStore.\n\n";
    $i++;
}
print "No more DataStores.\n\n";
print "\n\n";
#******************************************************************************************************************************************************
#************************************************** E N D   S E C T I O N *********************************************************************************
 
 
#******************************************************************************************************************************************************
&actualtime();
#************************************************** N E W   S E C T I O N ********************************************************************************
#**************** This Section Removes the Snapshot and applies the delta. ************************************************************************************
#**************** You may want to change the removeall option here to just remove ******************************************************************************
#******************************************************************************************************************************************************
print " ----- Remove ALL Snapshots of the VM -----";
print "\n\n";
system("perl $RCLIPath/Perl/apps/vm/snapshotmanager.pl --url $url --username $username --password $password --operation removeall --powerstatus poweredOn --snapshotname $snapshotname --children 1 --vmname $VirtualMachineName");
$i++;
print "\n\n";
&actualtime();
print " ***** Script End ***************************";
#******************************************************************************************************************************************************
#************************************************** E N D   S E C T I O N *********************************************************************************
 
 
#sub function to print the actual time in the log
sub actualtime
{
	my ($Sekunden, $Minuten, $Stunden, $Monatstag, $Monat,
	    $Jahr, $Wochentag, $Jahrestag, $Sommerzeit) = localtime(time);
	my $CTIME_String = localtime(time);
	$Monat+=1;
	$Jahrestag+=1;
	$Monat = $Monat < 10 ? $Monat = "0".$Monat : $Monat;
	$Monatstag = $Monatstag < 10 ? $Monatstag = "0".$Monatstag : $Monatstag;
	$Stunden = $Stunden < 10 ? $Stunden = "0".$Stunden : $Stunden;
	$Minuten = $Minuten < 10 ? $Minuten = "0".$Minuten : $Minuten;
	$Sekunden = $Sekunden < 10 ? $Sekunden = "0".$Sekunden : $Sekunden;
	$Jahr+=1900;
	
	print "$Jahr-$Monat-$Monatstag $Stunden:$Minuten:$Sekunden";
}

Open in new window

0
 

Author Comment

by:ShopLiftin
ID: 24144917
Also, the script that BestWay pointed out did not run and backup my VM's, mainly because of the issues that I wrote into the solution script, spaced in folder names, VMDKs spread across datastores, etc.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If we need to check who deleted a Virtual Machine from our vCenter. Looking this task in logs can be painful and spend lot of time, so the best way to check this is in the vCenter DB. Just connect to vCenter DB(default DB should be VCDB and using…
In this article, I will show you HOW TO: Install VMware Tools for Windows on a VMware Windows virtual machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, using the VMware Host Client. The virtual machine has Windows Server 2016 instal…
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

739 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question