Backup setup unreliable and taking to long to run

Hi Experts,

I have inherited a messy backup system. We have about 8TB of data to backup (6.5 GB for physical machines and 1.5GB for virtual machines). At the moment it’s backed up onto LTO Ultrium 4 data cartridge tapes (800 GB native tapes) using a 16 drive bay tape feeder drive and Symantec backup exec 2010 R3 for 7 physical servers and Veeam backup and replication 6.5 to back up 4 Virtual servers  (Vmware VM's).

Backup Exec 2010 R3 - Runs a full backup on Friday evening and the backup takes some time to complete running well into Tuesday afternoon to complete and uses 9 tapes (possible will need 10 tapes soon). Incremental backups are run 3 times during the week on Tuesday, Wednesday and Thursday evening and use 3 tapes for each incremental backup.
Veeam backup and replication 6.5 – Runs a synthetic full backup on Saturday and incremental backups Sun/Mon/Tues/Wed/Thurs/Fri and stores the files on a ISCSI SAN

Also after the Veeam backups are run, Backup Exec is used to Backup some of the Veeam backup files onto the LTO Tapes (not all of Veeam data so bit of a messy setup)

The problem is the backups take to long to run, are difficult to manage, the tapes take to long to change over and restores are unreliable We would like to implement a new backup solution that ensure full backups have enough time to complete over the weekend without running into any working weekdays but not sure on what backup media & solution is best to use. We need a backup solution and media that is reliable and as quick as possible to backup and restore. We would prefer backup media that can be stored in a safe and taken offsite relatively easily and not sure if tape is best option.

Any suggestions on what we should do?


Kevin SlaterIT ManagerAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Old UserCommented:
Hi do you have a budget to upgrade any hardware.
You could get a 16 slot LTO-6 for around £2000
Backup Exec 2014 with the ESX agent would allow you to use one solution for both VM and physical servers
Mr TorturSystem EngineerCommented:
Hi Kevin,
well, yes, your full backup seems to be very long. What is the tape library model and - most important - its attachment to the backup exec server? (scsi, SAS, FC) Also I did not understand : you have several drives or only one in your actual tape library?
You could still keep a tape system but buying a new one, with SAS or FC attachment, and with LTO6 drives (2.5 TB native capacity and better speed).
Or you could switch for a disk based system, supporting vm and physical server. But oftenly I would say this is not to be taken offsite. I don't think I have a good knowledge of all disk based solution in order to give you a good advice on a hardware.

If I were you I would look upon backup settings and backup selections seriously, and check jobs logs, because your incremental backup seem quite big : 3 LTO4 tapes for one incremental, so an incremental is minimum a 2,4 TB size backup, that sound abnormal for me, but it depends of the type and use of data.
Mr TorturSystem EngineerCommented:
Veeam 7 supports tape backup too. HCL and options are limited, and Veeam's price goes up lately, but I think you should have a look.
Kevin SlaterIT ManagerAuthor Commented:
Thanks for replies. Yes we have budget to buy new hardware and software if required. The tape drive is as single unit that supports loading 16 tapes at the same time. Its a HP Ultrium 4-SCSI and uses LTO 64k

We were thinking maybe we should move away from tape and use another media such as External USB drives or something else?, we are concerned if we have a tape drive hardware failure its more difficult to do restores.

What advantage would using a 6 slot LTO-6 have instead?
Thomas RushCommented:
LTO-6 has the advantage of much higher capacity per cartridge -- 2.5TB native vs. the 800GB of your LTO-4.  It also can -- theoretically -- run faster (160MB/sec vs. LTO-4's 120MB/sec) -- but only if you can get the data to it fast enough... and with your setup I don't think that the tape drives are the performance problem.  120MB/sec is over 400GB/hour; at that speed (even with uncompressible data), your 8TB should be done in under 20 hours.

So your primary challenge is to look at how to get your data off the disks faster.  Things to investigate:
1) What are the interconnects between your backup server and the other servers?  Are all connections at 1Gb or faster?  Any link that is 100Mbit will  limit you to 12MB/sec, which is painfully slow.  If Gb or faster, are the pipes congested, so that the backup data can't get to the backup server fast enough?
2) Are the client disks fragmented?  This will slow backup down, and can be cured with periodic defragmentation.  Do they contain a significant number of small and tiny files in deeply nested directories?  This is harder to fix, but often a cause of slow backups.  
3) When you look at your backup logs, how fast does it say that the backup jobs ran?  At the speeds you're relating, you're probably getting an average of under 100GB/hour, or about 15GB/minute or 25MB/second.  At these speeds, you're getting excessive buffer underrun in your tape drives, so the tape has to stop frequently, rewind, and then start up again so it can start writing near where it left off.  This contributes to premature wear on both the tape drive and the media.

4) As other posters have pointed out, it's very unusual for your incremental backups to be so large; you might want to make sure you're only backing up what you think you are in these incrementals.

5) Make sure that you don't have software encryption or software compression turned on for your backup jobs.  Both of these are CPU intensive, and either can significantly slow down your backups!  LTO-tape drives are pretty good at compression, and they do it in HW with no performance penalty.  LTO-4 and later tape drives do HW encryption exceptionally well if that's a requirement, and do it without a performance or space penalty.   You'll have to check these for each backup job in the job properties.

So how to make things better?
a) First, check all your interconnects.  Make sure you're on Gb networking (or faster).  Take a look at the client disks.  Can they be upgraded for speed (faster disks, RAID 10 instead of RAID 5, defragmentation, etc.)?
b) Make sure your Backup Exec and its catalog are on fast disks.  Every file you backup has to be catalogued, and if those writes are going to slow disk, then the whole backup environment will be affected.
c) Make sure your client systems aren't doing disk intensive work while the backup is happening.  No defrags, no virus scans, etc.
d) Consider interleaving your backup jobs if your version of BE allows it,  This is where you send multiple servers' data to the tape drive at once.  So where now you're getting 25MB/sec/server, if you send four servers simultaneously to one tape as part of one job, you'll be getting an effective speed of 100MB/second, and keeping the tape drive happier too.  The disadvantage is that your restores will take longer, as each one will have to comb through 4x the blocks to get the data for a particular server... but to get your backup jobs done in a reasonable time, it might be worth it.

e) You can also consider implementing a disk-to-disk intermediate tier in your backups.  So you backup to disk, with many more simultaneous streams than you could get than when backing up to a tape drive, and then you copy the jobs you need to keep on tape off to tape; since it's an optimized mostly unfragmented backup on disk, you can typically pull each one off to tape much quicker than you could from the original systems.

If doing this, consider getting what they call a "purpose built backup appliance" like HP's StoreOnce or EMC's Data Domain, both of which will perform deduplication, which performs single instance storage at the block level.   With these systems (Let's say the HP StoreOnce 4500), you can probably keep 3+ months of backups on the appliance, copying the full backups off to tape at your convenience without affecting your production systems or users.  See

f) You can also get another autoloader or tape drive, say an LTO-6.  An LTO-6 will vastly increase the amount of data you can fit on a single tape (over 3x vs. LTO-4), but it won't work any faster than your LTO-4, because the tape drive is not your current bottleneck.  *That said*, having a second tape drive will mean you can partition your backups, half going to the old drive, half to the new drive, and overall your backups will get done faster because you have two targets.   **Do** make sure you send enough data to the LTO-6 drive to keep it streaming (steady state of 60MB/sec or more, assuming uncompressible data; better 100MB/second since data does compress, and if yours is 1.5:1 compressible, that means you're only getting 65MB/sec of writable data (after compression) to the tape write head).

g) as for your last point about something that can be taken offsite -- tape tends to be the best for this.  The tape cartridges are rugged, high capacity, and inexpensive.  LTO-

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Kevin SlaterIT ManagerAuthor Commented:
SelfGovern - thanks a lot for that very useful info. At the moment we are getting 1,200mb per minute, 120mb per 6 seconds, 10mb per second on the weekly incremental backup to the LTO 6 tape using Backup Exec and on the full backup that starts friday evening 20mb per second.

Im checking through your info and will aim to post some further info soon.
Kevin SlaterIT ManagerAuthor Commented:
Thanks a lot for your very useful help on this, we are considering getting a EMC data domain as per one of your suggestions on point E to store the backups on, just not sure on which backup software we will use, we may use Veeam still for VM's and maybe an EMC solution to backup server data.

We they then offload the backup data to External USB 3 hard drives or tape.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.