Backup setup unreliable and taking to long to run

Posted on 2014-08-19
Last Modified: 2016-02-25
Hi Experts,

I have inherited a messy backup system. We have about 8TB of data to backup (6.5 GB for physical machines and 1.5GB for virtual machines). At the moment it’s backed up onto LTO Ultrium 4 data cartridge tapes (800 GB native tapes) using a 16 drive bay tape feeder drive and Symantec backup exec 2010 R3 for 7 physical servers and Veeam backup and replication 6.5 to back up 4 Virtual servers  (Vmware VM's).

Backup Exec 2010 R3 - Runs a full backup on Friday evening and the backup takes some time to complete running well into Tuesday afternoon to complete and uses 9 tapes (possible will need 10 tapes soon). Incremental backups are run 3 times during the week on Tuesday, Wednesday and Thursday evening and use 3 tapes for each incremental backup.
Veeam backup and replication 6.5 – Runs a synthetic full backup on Saturday and incremental backups Sun/Mon/Tues/Wed/Thurs/Fri and stores the files on a ISCSI SAN

Also after the Veeam backups are run, Backup Exec is used to Backup some of the Veeam backup files onto the LTO Tapes (not all of Veeam data so bit of a messy setup)

The problem is the backups take to long to run, are difficult to manage, the tapes take to long to change over and restores are unreliable We would like to implement a new backup solution that ensure full backups have enough time to complete over the weekend without running into any working weekdays but not sure on what backup media & solution is best to use. We need a backup solution and media that is reliable and as quick as possible to backup and restore. We would prefer backup media that can be stored in a safe and taken offsite relatively easily and not sure if tape is best option.

Any suggestions on what we should do?


Question by:Kevin Slater
    LVL 11

    Expert Comment

    by:Old User
    Hi do you have a budget to upgrade any hardware.
    You could get a 16 slot LTO-6 for around £2000
    Backup Exec 2014 with the ESX agent would allow you to use one solution for both VM and physical servers
    LVL 11

    Expert Comment

    Hi Kevin,
    well, yes, your full backup seems to be very long. What is the tape library model and - most important - its attachment to the backup exec server? (scsi, SAS, FC) Also I did not understand : you have several drives or only one in your actual tape library?
    You could still keep a tape system but buying a new one, with SAS or FC attachment, and with LTO6 drives (2.5 TB native capacity and better speed).
    Or you could switch for a disk based system, supporting vm and physical server. But oftenly I would say this is not to be taken offsite. I don't think I have a good knowledge of all disk based solution in order to give you a good advice on a hardware.

    If I were you I would look upon backup settings and backup selections seriously, and check jobs logs, because your incremental backup seem quite big : 3 LTO4 tapes for one incremental, so an incremental is minimum a 2,4 TB size backup, that sound abnormal for me, but it depends of the type and use of data.
    LVL 11

    Expert Comment

    Veeam 7 supports tape backup too. HCL and options are limited, and Veeam's price goes up lately, but I think you should have a look.

    Author Comment

    by:Kevin Slater
    Thanks for replies. Yes we have budget to buy new hardware and software if required. The tape drive is as single unit that supports loading 16 tapes at the same time. Its a HP Ultrium 4-SCSI and uses LTO 64k

    We were thinking maybe we should move away from tape and use another media such as External USB drives or something else?, we are concerned if we have a tape drive hardware failure its more difficult to do restores.

    What advantage would using a 6 slot LTO-6 have instead?
    LVL 20

    Accepted Solution

    LTO-6 has the advantage of much higher capacity per cartridge -- 2.5TB native vs. the 800GB of your LTO-4.  It also can -- theoretically -- run faster (160MB/sec vs. LTO-4's 120MB/sec) -- but only if you can get the data to it fast enough... and with your setup I don't think that the tape drives are the performance problem.  120MB/sec is over 400GB/hour; at that speed (even with uncompressible data), your 8TB should be done in under 20 hours.

    So your primary challenge is to look at how to get your data off the disks faster.  Things to investigate:
    1) What are the interconnects between your backup server and the other servers?  Are all connections at 1Gb or faster?  Any link that is 100Mbit will  limit you to 12MB/sec, which is painfully slow.  If Gb or faster, are the pipes congested, so that the backup data can't get to the backup server fast enough?
    2) Are the client disks fragmented?  This will slow backup down, and can be cured with periodic defragmentation.  Do they contain a significant number of small and tiny files in deeply nested directories?  This is harder to fix, but often a cause of slow backups.  
    3) When you look at your backup logs, how fast does it say that the backup jobs ran?  At the speeds you're relating, you're probably getting an average of under 100GB/hour, or about 15GB/minute or 25MB/second.  At these speeds, you're getting excessive buffer underrun in your tape drives, so the tape has to stop frequently, rewind, and then start up again so it can start writing near where it left off.  This contributes to premature wear on both the tape drive and the media.

    4) As other posters have pointed out, it's very unusual for your incremental backups to be so large; you might want to make sure you're only backing up what you think you are in these incrementals.

    5) Make sure that you don't have software encryption or software compression turned on for your backup jobs.  Both of these are CPU intensive, and either can significantly slow down your backups!  LTO-tape drives are pretty good at compression, and they do it in HW with no performance penalty.  LTO-4 and later tape drives do HW encryption exceptionally well if that's a requirement, and do it without a performance or space penalty.   You'll have to check these for each backup job in the job properties.

    So how to make things better?
    a) First, check all your interconnects.  Make sure you're on Gb networking (or faster).  Take a look at the client disks.  Can they be upgraded for speed (faster disks, RAID 10 instead of RAID 5, defragmentation, etc.)?
    b) Make sure your Backup Exec and its catalog are on fast disks.  Every file you backup has to be catalogued, and if those writes are going to slow disk, then the whole backup environment will be affected.
    c) Make sure your client systems aren't doing disk intensive work while the backup is happening.  No defrags, no virus scans, etc.
    d) Consider interleaving your backup jobs if your version of BE allows it,  This is where you send multiple servers' data to the tape drive at once.  So where now you're getting 25MB/sec/server, if you send four servers simultaneously to one tape as part of one job, you'll be getting an effective speed of 100MB/second, and keeping the tape drive happier too.  The disadvantage is that your restores will take longer, as each one will have to comb through 4x the blocks to get the data for a particular server... but to get your backup jobs done in a reasonable time, it might be worth it.

    e) You can also consider implementing a disk-to-disk intermediate tier in your backups.  So you backup to disk, with many more simultaneous streams than you could get than when backing up to a tape drive, and then you copy the jobs you need to keep on tape off to tape; since it's an optimized mostly unfragmented backup on disk, you can typically pull each one off to tape much quicker than you could from the original systems.

    If doing this, consider getting what they call a "purpose built backup appliance" like HP's StoreOnce or EMC's Data Domain, both of which will perform deduplication, which performs single instance storage at the block level.   With these systems (Let's say the HP StoreOnce 4500), you can probably keep 3+ months of backups on the appliance, copying the full backups off to tape at your convenience without affecting your production systems or users.  See

    f) You can also get another autoloader or tape drive, say an LTO-6.  An LTO-6 will vastly increase the amount of data you can fit on a single tape (over 3x vs. LTO-4), but it won't work any faster than your LTO-4, because the tape drive is not your current bottleneck.  *That said*, having a second tape drive will mean you can partition your backups, half going to the old drive, half to the new drive, and overall your backups will get done faster because you have two targets.   **Do** make sure you send enough data to the LTO-6 drive to keep it streaming (steady state of 60MB/sec or more, assuming uncompressible data; better 100MB/second since data does compress, and if yours is 1.5:1 compressible, that means you're only getting 65MB/sec of writable data (after compression) to the tape write head).

    g) as for your last point about something that can be taken offsite -- tape tends to be the best for this.  The tape cartridges are rugged, high capacity, and inexpensive.  LTO-

    Author Comment

    by:Kevin Slater
    SelfGovern - thanks a lot for that very useful info. At the moment we are getting 1,200mb per minute, 120mb per 6 seconds, 10mb per second on the weekly incremental backup to the LTO 6 tape using Backup Exec and on the full backup that starts friday evening 20mb per second.

    Im checking through your info and will aim to post some further info soon.

    Author Closing Comment

    by:Kevin Slater
    Thanks a lot for your very useful help on this, we are considering getting a EMC data domain as per one of your suggestions on point E to store the backups on, just not sure on which backup software we will use, we may use Veeam still for VM's and maybe an EMC solution to backup server data.

    We they then offload the backup data to External USB 3 hard drives or tape.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Top 6 Sources for Identifying Threat Actor TTPs

    Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

    First I will try to share a design of a Veeam Backup Infrastructure without Direct NFS Access backup.  ( Note: Direct NFS Access backup …
    Storage devices are generally used to save the data or sometime transfer the data from one computer system to another system. However, sometimes user accidentally erased their important data from the Storage devices. Users have to know how data reco…
    This tutorial will walk an individual through setting the global and backup job media overwrite and protection periods in Backup Exec 2012. Log onto the Backup Exec Central Administration Server. Examine the services. If all or most of them are stop…
    The viewer will learn how to start File History, a MACINTOSH like backup utility built into windows 8, on your Windows 8 computer. To open the File History control panel swipe from the right  side to get the search menu or position the cursor in the…

    760 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    8 Experts available now in Live!

    Get 1:1 Help Now