Want to win a PS4? Go Premium and enter to win our High-Tech Treats giveaway. Enter to Win


Considerations for backup in Cloud : handing over tapes to tenants / customers

Posted on 2014-01-29
Medium Priority
Last Modified: 2016-10-27
When operating a private Cloud that hosts multiple tenants / customers,
we have a contractual agreement that in the event a tenant wants to
exit from our Cloud (decommissioning or moving to another Cloud
service provider or ...), we have to hand over the backup tapes to the
tenant (VMs & data backups).

a) So we have to cater / reserve tapes used for specific tenant & not
    share the tapes?  If the tenant or customer has multiple systems/
    services hosted in our Cloud, they may even require that the tapes
    are segregated by per project.  How is this practiced out there in
    the Cloud industry?  We're a private cloud provider (not public)
    but can have up to 50 tenants (with multiple systems/services
    per tenant).  Any consideration in terms of SaaS, IaaS to take
    into account?

b) Suppose we use a specific software to take backup (Data Protector,
    Netbackup, Commvault), how is the tenant whom we've handed the
    tapes to going to read the tapes if they're using a different backup

c) If we're practising encryption in backup, is hardware or software
    encryption more portable or practical so that after handing over
    the tapes (& assuming the tenant use a different brand of tape
    drive, eg: Quantum vs IBM;  I'm not talking about media types'
    differences like DDS/DAT vs LTO vs optical drive), the tenant will
    not into issue with decrypting it?  Something like just have to
    key in the password & the customer can decrypt it : so is this
    hardware or software encryption

d) I've heard that in Netbackup & Data Protector we have to do
     daily dumping of backup catalogs: if backups to SATA disks
     using a 'universally portable' method (say create a Truecrypt
     container on SATA & just backup into SATA drives), I guess
     we won't need to do this cumbersome daily backup cataloging?

e) In one bank & a critical financial information service provider,
    I've seen older tapes (more than 5 years old) became unreadable
    though the offsite storage provider stores the tape in the right
    environmental temperature/moisture: any way we can prevent
    such issue?  We fear it may become a litigation issue if our backup
    tapes become unreadable.  To call back tons of tapes to do regular
    test restoration is not feasible.  Can we just sign a back to back
    agreement/SLA with our offsite storage provider that in the event
    a tape under their care become unreadable, liability is on them?
    What about technical prevention/mitigation measures?

    Is backing up to multiple SATA hard disks feasible?  Then call bac
    SATA disks from time to time & just mount them: if there's error,
    amber LED will show (ie we don't need to do the tedious tasks
    of test restoring)?
Question by:sunhux
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 4

Author Comment

ID: 39817228
Typo correction:

d)  is backups to SATA disks using a 'universally portable' method
     (say create a Truecrypt container on SATA & write into SATA
      drives) a feasible method?

I've heard of Acronis writing to an image that's password protected.
Is this encryption?  Would rather use something Opensource so that
it's highly portable / universally readable when a tenant moves out
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1500 total points
ID: 39817254
We also do the same, but we Ask our Clients, what Exit plan they require and tailor, accordingly.  As part of our Exit plan, we have a number of days chargeable, which we help the Client Move their data to new system, either Cloud or on-premise.

We have a Bronze, Silver and Gold (it's included with Gold for FREE!), but is optional with Bronze and Silver. This is up to 14 man days, included to transfer to Amazon, other Cloud Provider, Rackspace, or on-premise.

Using tapes is very old, (we abandoned in 2004!) even if you use tapes, how do you know the software and tape devices will be compatible with the clients software and tape devices.

Tape devices are also not supported by ESXi.

a) I would recommend and use either Unitrends, AppAssure or NetBackup and Backup Tenats software, and create Tenant Jobs.

b) As per my initial statement, you will need to use a common Backup Software application, or they will need to use the same software and devices as you.

c) They will need to use the same hardware and software.

d) You will always need a catalog first to restore the data.

e) tapes are very difficult, not only does the tape degrade, but we've found that most 7 year archives are rubbish, because you cannot find a tape drive with the correct head alignment to restore the data from tape.

I would recommend backups to Disk! Use disk storage, and Archive VMs, in VMDK format.

Author Comment

ID: 39818375
Just to elaborate on item (c) :
Suppose we use Netbackup NBU & our tenant also uses NBU at their
premises (but different brand of drive, say IBM while we use Quantum):
 so in this case, software encryption will ensure tenant can read the
tapes (assuming both our cloud & tenant are on LTO5 media) while
hardware encryption will make the tapes not portable over to the
tenant's drive?
When ransomware hits your clients, what do you do?

MSPs: Endpoint security isn’t enough to prevent ransomware.
As the impact and severity of crypto ransomware attacks has grown, Webroot fought back, not just by building a next-gen endpoint solution capable of preventing ransomware attacks but also by being a thought leader.


Author Comment

ID: 39818389
> would recommend backups to Disk! Use disk storage, and Archive VMs, in VMDK format
But if there's a regulatory need to rotate backup media offsite (in case the current
site is hit by disaster), do you rotate the disks offsite?

How's VM's archived in VMDK format?  Scp out the vmdk files from the
ESXi hosts?
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1500 total points
ID: 39818568
We mirror data to several different datacenters, in the UK, EMEA, and US - so that covers us with our clients.

3 Different Datacenters, and we also archive to Gigasoft Data Backup


So that puts a tick in the box as offsite backups for our clients.

Are backups, are Snapshot Based on LUNs, so they exist in plain VMware format, so they will always be available and compatible with VMware, we have backups for clients as far back as ESX 1.0, ESX 2.0 and ESX 2.54, which if they wanted us to retrieve tomorrow we could for them, and mount up.

Author Comment

ID: 39819709
How's the VMDK files being copied out to disks for
backups in your environment?  Do you scp/sftp them
out from the ESXi host or the LUNs are mounted as
readonly & then copy out to disks?

One other question that needs your clarification:
Suppose we use Netbackup NBU & our tenant also uses NBU at their
premises (but different brand of drive, say IBM while we use Quantum):
 so in this case, software encryption will ensure tenant can read the
tapes (assuming both our cloud & tenant are on LTO5 media) while
hardware encryption will make the tapes not portable over to the
tenant's drive?
LVL 124
ID: 39820043
The VMDKs are not copied, they are stored on the LUNs, the LUNs are mirrored around the datacentres.

In the event of a restore, it depends on the clients request, and what technology they have available, in it's simplest form, scp/sftp could be used.

Even if you use the same software, they may not be able to read the tapes, if they have an alignment issue with your tape drives.

but if using hardware encryption the make and model would need to be the same.

You would need to send out a test tape to ensure compatibility.

Author Comment

ID: 39821147
>  same software, they may not be able to read the tapes, if they have
> an alignment issue with your tape drives.
Have not heard of alignment issue before.  Is this due to differences
in the firmware version of the tape drives or ?
I suppose if the backup software version (eg: NBU) differs by a lot,
may not be supported.
LVL 124

Assisted Solution

by:Andrew Hancock (VMware vExpert / EE MVE^2)
Andrew Hancock (VMware vExpert / EE MVE^2) earned 1500 total points
ID: 39821979
It's a difference in the tape drive heads, the same as a the heads on a video recorder.

did you have a video recorder, with a tracking button or wheel, you used to adjust when you inserted another video, because it did not play well.

Veeam Backup and Replication v7.0 now includes TAPE SUPPORT!

So you can one...

1. Backup to Disk.
2. Backup to Off-site
3. Backup to Tape

I think that ticks all your boxes!
LVL 21

Accepted Solution

SelfGovern earned 500 total points
ID: 39823285
In answer to your questions about encryption.  There are several ways to encrypt data on a modern LTO tape.
1) Software: The backup application manages encryption keys, and encrypts the data before it is sent ot the tape drive.  You lose any compression that the tape drive would normally give you (encrypted data is not compressible).  Encryption is a pretty processor-intensive task, and you might see a drop in performance.  How much depends on your CPU and what else is going on on the system.  You can also compress data before it is sent to the tape drive, but this puts additional load on the backup server, and may slow performance further.
2) Encrypt in SW with backup application managed keys.  Your Backup Exec or TSM or Data Protector or Veeam manages the encryption keys, but it sends the key to the tape drive and the tape drive encrypts the data *after* compressing it.  Encryption happens at wire speed, so there is no loss of performance or compression.  ANY LTO drive that supports encryption and  that can read a particular tape if unencrypted, will be able to read this tape encrypted, provided it has the key.
There are two potential drawbacks here.
a) People don't back up their encryption keys rigorously, don't have a copy offsite, and don't test their restore processes.  Without the key, you are without the data.  (This is the same as in 1), above)
b) It's as easy as un-checking the "encrypt" box before a job runs, and re-checking it after the job runs, for a bad apple to generate a tape that only he knows is unencrypted.  That fact is probably stored in the backup logs somewhere, but I doubt anyone ever looks at reports on jobs that complete successfully.  Or do any backup vendors support automatic email to multiple people in the organization when any policy around encryption changes?
3) You can get some sort of hardware key management system.  This can be as simple as HP's MSL Encryption Kit, to some of the expensive enterprise encryption key managers.  These devices -- if set up properly -- make it difficult to impossible for one person to change encryption policies on his own.  They do enable HW compression on the tape drive, so there is no loss of compression or encryption or performance.
However -- they do require you use that solution (combination of encryption solution, and usually vendor tape library) to decrypt a tape.

Long story short -- you can use the backup application to generate and manage your encryption keys, but still perform encryption *on the tape drive*, with no loss of performance or compression,.  Your customers will be able to decrypt an encrypted tape on their tape drive as long as it is the same generation or one ahead, regardless of who manufactured the tape drive,, so long as you give them the correct key.

Z) If the customer has a tape library with some HW key management solution, they will need to get a tape drive that is outside the library, or partition the library so as to have one tape drive not managed by their HW key management solution.  The reason is that you can only have one key manager for a tape drive.  When the backup application is managing keys, you can't have a HW key manager also managing encryption for drives in a library it manages.
Y) Remember that you will hae to give the customer the key to the tape.  If you're using one key for all backups, you've given away the key for all your tapes... that might not be acceptable to the rest of your clients.  I don't know if or which backup applications allow different key per job -- but I'd look in to this to avoid any possible issue.
X) Store your keys in multiple places.  One of the good ways is in an encrypted file on a USB memory stick.  This prevents casual users from getting access to the keys... but does mean you have to guard the password to the encrypted keyfile.

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Ever wonder what it's like to get hit by ransomware? "Tom" gives you all the dirty details first-hand – and conveys the hard lessons his company learned in the aftermath.
What's worse than having your data encrypted by ransomware? Getting attacked by a so-called "wiper," which simply destroys the data and offers you no hope of ever seeing it again.
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…
Two types of users will appreciate AOMEI Backupper Pro: 1 - Those with PCIe drives (and haven't found cloning software that works on them). 2 - Those who want a fast clone of their boot drive (no re-boots needed) and it can clone your drive wh…

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question