• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 788
  • Last Modified:

Considerations for backup in Cloud : handing over tapes to tenants / customers

When operating a private Cloud that hosts multiple tenants / customers,
we have a contractual agreement that in the event a tenant wants to
exit from our Cloud (decommissioning or moving to another Cloud
service provider or ...), we have to hand over the backup tapes to the
tenant (VMs & data backups).

a) So we have to cater / reserve tapes used for specific tenant & not
    share the tapes?  If the tenant or customer has multiple systems/
    services hosted in our Cloud, they may even require that the tapes
    are segregated by per project.  How is this practiced out there in
    the Cloud industry?  We're a private cloud provider (not public)
    but can have up to 50 tenants (with multiple systems/services
    per tenant).  Any consideration in terms of SaaS, IaaS to take
    into account?

b) Suppose we use a specific software to take backup (Data Protector,
    Netbackup, Commvault), how is the tenant whom we've handed the
    tapes to going to read the tapes if they're using a different backup

c) If we're practising encryption in backup, is hardware or software
    encryption more portable or practical so that after handing over
    the tapes (& assuming the tenant use a different brand of tape
    drive, eg: Quantum vs IBM;  I'm not talking about media types'
    differences like DDS/DAT vs LTO vs optical drive), the tenant will
    not into issue with decrypting it?  Something like just have to
    key in the password & the customer can decrypt it : so is this
    hardware or software encryption

d) I've heard that in Netbackup & Data Protector we have to do
     daily dumping of backup catalogs: if backups to SATA disks
     using a 'universally portable' method (say create a Truecrypt
     container on SATA & just backup into SATA drives), I guess
     we won't need to do this cumbersome daily backup cataloging?

e) In one bank & a critical financial information service provider,
    I've seen older tapes (more than 5 years old) became unreadable
    though the offsite storage provider stores the tape in the right
    environmental temperature/moisture: any way we can prevent
    such issue?  We fear it may become a litigation issue if our backup
    tapes become unreadable.  To call back tons of tapes to do regular
    test restoration is not feasible.  Can we just sign a back to back
    agreement/SLA with our offsite storage provider that in the event
    a tape under their care become unreadable, liability is on them?
    What about technical prevention/mitigation measures?

    Is backing up to multiple SATA hard disks feasible?  Then call bac
    SATA disks from time to time & just mount them: if there's error,
    amber LED will show (ie we don't need to do the tedious tasks
    of test restoring)?
  • 5
  • 4
4 Solutions
sunhuxAuthor Commented:
Typo correction:

d)  is backups to SATA disks using a 'universally portable' method
     (say create a Truecrypt container on SATA & write into SATA
      drives) a feasible method?

I've heard of Acronis writing to an image that's password protected.
Is this encryption?  Would rather use something Opensource so that
it's highly portable / universally readable when a tenant moves out
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We also do the same, but we Ask our Clients, what Exit plan they require and tailor, accordingly.  As part of our Exit plan, we have a number of days chargeable, which we help the Client Move their data to new system, either Cloud or on-premise.

We have a Bronze, Silver and Gold (it's included with Gold for FREE!), but is optional with Bronze and Silver. This is up to 14 man days, included to transfer to Amazon, other Cloud Provider, Rackspace, or on-premise.

Using tapes is very old, (we abandoned in 2004!) even if you use tapes, how do you know the software and tape devices will be compatible with the clients software and tape devices.

Tape devices are also not supported by ESXi.

a) I would recommend and use either Unitrends, AppAssure or NetBackup and Backup Tenats software, and create Tenant Jobs.

b) As per my initial statement, you will need to use a common Backup Software application, or they will need to use the same software and devices as you.

c) They will need to use the same hardware and software.

d) You will always need a catalog first to restore the data.

e) tapes are very difficult, not only does the tape degrade, but we've found that most 7 year archives are rubbish, because you cannot find a tape drive with the correct head alignment to restore the data from tape.

I would recommend backups to Disk! Use disk storage, and Archive VMs, in VMDK format.
sunhuxAuthor Commented:
Just to elaborate on item (c) :
Suppose we use Netbackup NBU & our tenant also uses NBU at their
premises (but different brand of drive, say IBM while we use Quantum):
 so in this case, software encryption will ensure tenant can read the
tapes (assuming both our cloud & tenant are on LTO5 media) while
hardware encryption will make the tapes not portable over to the
tenant's drive?
sunhuxAuthor Commented:
> would recommend backups to Disk! Use disk storage, and Archive VMs, in VMDK format
But if there's a regulatory need to rotate backup media offsite (in case the current
site is hit by disaster), do you rotate the disks offsite?

How's VM's archived in VMDK format?  Scp out the vmdk files from the
ESXi hosts?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
We mirror data to several different datacenters, in the UK, EMEA, and US - so that covers us with our clients.

3 Different Datacenters, and we also archive to Gigasoft Data Backup


So that puts a tick in the box as offsite backups for our clients.

Are backups, are Snapshot Based on LUNs, so they exist in plain VMware format, so they will always be available and compatible with VMware, we have backups for clients as far back as ESX 1.0, ESX 2.0 and ESX 2.54, which if they wanted us to retrieve tomorrow we could for them, and mount up.
sunhuxAuthor Commented:
How's the VMDK files being copied out to disks for
backups in your environment?  Do you scp/sftp them
out from the ESXi host or the LUNs are mounted as
readonly & then copy out to disks?

One other question that needs your clarification:
Suppose we use Netbackup NBU & our tenant also uses NBU at their
premises (but different brand of drive, say IBM while we use Quantum):
 so in this case, software encryption will ensure tenant can read the
tapes (assuming both our cloud & tenant are on LTO5 media) while
hardware encryption will make the tapes not portable over to the
tenant's drive?
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
The VMDKs are not copied, they are stored on the LUNs, the LUNs are mirrored around the datacentres.

In the event of a restore, it depends on the clients request, and what technology they have available, in it's simplest form, scp/sftp could be used.

Even if you use the same software, they may not be able to read the tapes, if they have an alignment issue with your tape drives.

but if using hardware encryption the make and model would need to be the same.

You would need to send out a test tape to ensure compatibility.
sunhuxAuthor Commented:
>  same software, they may not be able to read the tapes, if they have
> an alignment issue with your tape drives.
Have not heard of alignment issue before.  Is this due to differences
in the firmware version of the tape drives or ?
I suppose if the backup software version (eg: NBU) differs by a lot,
may not be supported.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
It's a difference in the tape drive heads, the same as a the heads on a video recorder.

did you have a video recorder, with a tracking button or wheel, you used to adjust when you inserted another video, because it did not play well.

Veeam Backup and Replication v7.0 now includes TAPE SUPPORT!

So you can one...

1. Backup to Disk.
2. Backup to Off-site
3. Backup to Tape

I think that ticks all your boxes!
In answer to your questions about encryption.  There are several ways to encrypt data on a modern LTO tape.
1) Software: The backup application manages encryption keys, and encrypts the data before it is sent ot the tape drive.  You lose any compression that the tape drive would normally give you (encrypted data is not compressible).  Encryption is a pretty processor-intensive task, and you might see a drop in performance.  How much depends on your CPU and what else is going on on the system.  You can also compress data before it is sent to the tape drive, but this puts additional load on the backup server, and may slow performance further.
2) Encrypt in SW with backup application managed keys.  Your Backup Exec or TSM or Data Protector or Veeam manages the encryption keys, but it sends the key to the tape drive and the tape drive encrypts the data *after* compressing it.  Encryption happens at wire speed, so there is no loss of performance or compression.  ANY LTO drive that supports encryption and  that can read a particular tape if unencrypted, will be able to read this tape encrypted, provided it has the key.
There are two potential drawbacks here.
a) People don't back up their encryption keys rigorously, don't have a copy offsite, and don't test their restore processes.  Without the key, you are without the data.  (This is the same as in 1), above)
b) It's as easy as un-checking the "encrypt" box before a job runs, and re-checking it after the job runs, for a bad apple to generate a tape that only he knows is unencrypted.  That fact is probably stored in the backup logs somewhere, but I doubt anyone ever looks at reports on jobs that complete successfully.  Or do any backup vendors support automatic email to multiple people in the organization when any policy around encryption changes?
3) You can get some sort of hardware key management system.  This can be as simple as HP's MSL Encryption Kit, to some of the expensive enterprise encryption key managers.  These devices -- if set up properly -- make it difficult to impossible for one person to change encryption policies on his own.  They do enable HW compression on the tape drive, so there is no loss of compression or encryption or performance.
However -- they do require you use that solution (combination of encryption solution, and usually vendor tape library) to decrypt a tape.

Long story short -- you can use the backup application to generate and manage your encryption keys, but still perform encryption *on the tape drive*, with no loss of performance or compression,.  Your customers will be able to decrypt an encrypted tape on their tape drive as long as it is the same generation or one ahead, regardless of who manufactured the tape drive,, so long as you give them the correct key.

Z) If the customer has a tape library with some HW key management solution, they will need to get a tape drive that is outside the library, or partition the library so as to have one tape drive not managed by their HW key management solution.  The reason is that you can only have one key manager for a tape drive.  When the backup application is managing keys, you can't have a HW key manager also managing encryption for drives in a library it manages.
Y) Remember that you will hae to give the customer the key to the tape.  If you're using one key for all backups, you've given away the key for all your tapes... that might not be acceptable to the rest of your clients.  I don't know if or which backup applications allow different key per job -- but I'd look in to this to avoid any possible issue.
X) Store your keys in multiple places.  One of the good ways is in an encrypted file on a USB memory stick.  This prevents casual users from getting access to the keys... but does mean you have to guard the password to the encrypted keyfile.
  • 5
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now