Need advice on online backup service for VMs

I have four remote sites running two vSphere Essentials 4 servers each.  Each site uses Veeam Backup & Replication software to both replicate the VMs back and forth on a daily basis.  Also, I backup each VM to a local Windows disk nightly.  Each site has combination of about 80-100 GB of VMs each.  So, I have about 400 GB of VM's to backup.  The problem is we have no tape backup system in place at any of these sites.  Therefore, a natural disaster would wipe us out since we have no offsite backups.

Two options I am considering are installing a tape drive at each site or going with an online backup solution.  I would like to avoid having to install tape drives at these remote sites due to the overhead and work in the actual installation, swapping tapes, staying tapes offsite, etc.  I have not used an online storage service, but it sounds intriguing.  A few concerns with that are the costs associated with storing up to 400 GB online and also the amount of time it will take to backup over the Internet.  Some of these sites have slow upload speeds.

I would appreciate any guidance/thoughts on this.
Who is Participating?
dosdet2Connect With a Mentor Commented:
That is basically what I do.  I have USB drive Docks at both ends.
The problem you will need to address is handling and Transport.  I handle mine personally so if you have someone else do it, make sure you train them.

Even though they are enterprise class, they still need to be handled like eggs.  Very gently. Insert them and extract them from the docks slowly and gently - turning off the power to the dock first.

Transporting the is also critical.  I acquired an old double CD case, removed the plastic CD holder from the cloth exterior and cut a block of soft foam to fit into the cloth container.  
(Approx 11" x 9" x 6")  
I cut the foam in half and cut out a rectangle the size of the hard drive in the bottom part.  The drives transport in a static bag in this case.  It works great.  If you want to use regular external drives, you need to transport them similarly - I have lost 2 of the external drives (Free Agents) because the were just in my brief case (wedged in foam if course) and that was still too much jarring.

Another option would be to use the Laptop size portable drives.  They are lighter and handle shocks and transporting much better (That is direct from a Seagate tech).  Especially if they are kept in their carrying cases.

The deal with those is that they aren't quite as fast and usually less capacity - However I did see a 1.5TB Seagate 2.5" portable drive at Costco the other day, so capacities are increasing on those units too and I assume speed is/ will be increasing also.  If I had your situation, I think this is how I would go.

As far as backup speed goes, I think if you connect directly to the server with USB 2.0 or better, you will be surprised at the backup speed as compared to a 100Mb Ethernet.  Not quite Gbit but very doable especially with some sort of changed block tracking technology.

I hope this helps.  I'd like to save you from some of the problems I had.

Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:

With the Veeam Backup & Replication you will get very low kb to copy between the site.

It only need on full backup(the first), then after this, using compressed and deduplication Storage backup, and also incremental backup it will copy only the few changes between the last backup and the current backup.

This is for backup, but also for any restore that you need to do using InstantRestore(Instant VM Recovery)

Have a look at this 2 videos from Veeam.

Last week I have design a similar architecture.

5 sites.
Site 1 and 2(big sites) using a Physical Server with Veeam Backup, and backup to Storage compressed and deduplication
Sites 3, 4 and 5(smaller sites) using Veeam Backup Virtual Appliance and backup to Storage(local storage).

Whith we have local backups for all sites, but if I need I can create a DR site, where all VM be replicated into this site using the replications features from Veeam, and using the options that I have added above.

Hope this can help you.

Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:

Forget to mention that I am referring to Veeam vPower v5

Also take a look at this White Paper

Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

Do you need to backup the complete set of VMs every night?
That might affect your decision.
To make the numbers easy, assume you wanted to back up 90GB per night.
To do that between 6PM and 9AM, you would need 90GB in 15 hours which is 6GB per hour.
To get 6GB/hour you would need to guarantee 14mBits/second throughput on average all night!

You could reduce that a lot by doing a complete backup of each VM once a week and then incrementals.  You could stagger the full VM backups.  But even then you could be talking about quite a lot of data.

Really you need to calculate more accurately how much data you would need to back up per night and what time window you have to see if it's feasible.

Why not do tape backups and get each site to send their tapes to another site?
You would have your own secure offsite backups and easy access to a remote tape if you needed it.

Ultrium 3 tape drives and tapes are very good - you get 400GB uncompressed per tape and the tapes are very robust.  Plenty of room for future expansion too.  You could get away with using DAT but the Ultrium tapes have been more reliable than DAT for me.
Just a warning.

In doing research on disaster preparedness, I studied several cases of  firms which were across the street from the twin towers during the 911 attack.  They lost their complete systems, everything was destroyed.  

One in particular had an online, real-time, cloud backup service where they were supposed to be able to setup workstations and internet access at another location and instantly log-in to their data as they rebuild their own systems.  They relocated and there was no data!  It took them weeks to get some of their data back and, because of the way the incremental backup worked, much had to be re-created   - which was very costly.

Here is the kicker - Their contract did not "guarantee" that all their data would be available and there was a stipulated maximum financial recovery which was very limited (about 1 months storage costs).  

The point is if you decide to do this, Read the Contract very carefully before you sign it.

In a more constructive light, I use 3 enterprise class, 2TB drives to backup on and swap them out every day so there is always one off-site at all times.  Then at the off-site location they are replicated back to duplicate NAS drives and an ESX server (to match the live site).

Note: I had to build a special transport container to transport them in, or they will go bad with so much transporting.

I have a VPN link to our off-site location and I'm working on doing the VM replications across the link.  I don't know if this would work in your situation but so far so good for us. (I also have local replication and local file system backup.)
For what it's worth.
Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:

@dosdet2 always depends on the budged to implement this backup and DR site. Also replication at Storage level with CDP is a good choice, but this is more expensive and many companys do not have the budged to implement this.

@Martin_J_Parker I have some company with 600Gb to 1TB VMs to backup, and with incremental backup we can backup all in 1h(or less), and with full backup in less than 2h with 70-100 MB/s.

Also using compressed and deduplication Storage backup the amount of data to backup will be less.

That's impressive BestWay - but is that dependent on getting decent compression?
Compression rates are going to depend on data type.  If it's all video or picture data that won't compress much at all whereas program source code will scrunch down nicely.  

Also, incremental sizes will depend heavily on how much the data is changing from day to day - and that could vary a lot between different sorts of applications.

70-100MB/s isn't the data transmission speed between sites is it?  Do you mean 70-100Mb/s?
100 Megabytes/second sounds pretty high speed to me!
We are on a very tight budget also, but it can be done if you work at it.  We also have 4 vm servers, about 200gb each, running on 2 ESX servers.

The most expensive part was the Backup ESX server (2-4xCPUs, 64g ram, 8-1tb drives) which was under $7K.  
The NAS drives were around $2k each w/6 1tb drives. Software, xfer drives &  incidentals were about another 3-4K.  We used enterprise class drives for all.
This was built piece by piece and spread out over a year.

We are using a differential block replication which only transfer's a VM's changed blocks.  Locally all four backup in about 1.5 hours (dedicated g-bit).  Across the internet is a different story.  We have one replication started and it completes in about 2 hours - down from 28 hours for the first time.
The least used one, of course.  

The others I'm still working with.  They were initially replicated locally but when they went off-site, the initial replication is still time consuming. I believe it is because of the change, the software see it as a different server and even though the replicated VM is there and referenced, it still wants to do an initial block by block comparison.  Remote site is DSL, being upgraded next week.  That should help.

Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:

Let me give you some examples from last night:

Veeam v4.1.1

VM Domain Controller Backup
Total VM size: 76,00 GB
Processed size: 76,00 GB
Processing rate: 129 MB/s
Backup mode: SAN/NBD without changed block tracking
Start time: 16-03-2011 02:17:40
End time: 16-03-2011 02:27:45
Duration: 0:10:05

VM Exchange Server Backup
8 of 8 files processed

Total VM size: 190,00 GB
Processed size: 190,00 GB
Processing rate: 22 MB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 16-03-2011 02:27:45
End time: 16-03-2011 04:55:00
Duration: 2:27:14

VM SQL Server Backup
8 of 8 files processed

Total VM size: 71,00 GB
Processed size: 71,00 GB
Processing rate: 123 MB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 16-03-2011 01:00:32
End time: 16-03-2011 01:10:25
Duration: 0:09:52

Total Backup
Total size of VMs to backup: 939,00 GB
Processed size: 939,00 GB
Processing rate: 66 MB/s
Start time: 16-03-2011 01:00:05
End time: 16-03-2011 05:02:54
Duration: 4:02:48

Compressed and deduplication is a feature from VMware API Storage, this doesn't look at file level.

Luciano PatrãoICT Senior Infraestructure  Engineer  Commented:

@dosdet2 yes you are correct, for this type of "small" DR is not very expensive, but I was referring to Storage replication trough WAN like EMC Avamar.

Also the full backup/replication on my last example was about 20h.

shannon_adamsAuthor Commented:
@dosdet2 - what is the model of "enterprise class, 2TB drives" you use for offisite storage?
Is that across an internet connection, or local?
If internet, what kind of connection do you have at each end?
It says processing rate, does that mean actual transfer rate?

It doesn't show the total amount of data transferred.

Our 200GB servers backup locally in 25 - 29 minutes each, 5 min for the less used one.
It's the internet transfer that is the killer.
They are Seagates - ST32000644NS  
The NS designates the Enterprise class, the standard drives are usually AS
shannon_adamsAuthor Commented:
@dosdet2: I thought you meant you were using external hard drives.  But, you gave me another idea.  Since I am talking 100 GB of VMs max per site, I am thinking about two good 1 TB USB drives and having an onsite person alternate those every week.  One stays at their house and the other stays hooked to the admin server that will perform the backups.  I will have one weekly full backup and daily incrementals.  Our backups are compressing by about 75%, but even 25 GB over our WAN or the Internet is going to be tough with the bandwidth we have at these small, remote sites.  Does that sound reasonable?
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.