Things I can do to speed up the backup

My backup is almost half a terabyte per night.  
I'm looking for ways to increase the speed of the backup.
All Servers and the backup system are on Gigabit switches.
Using Hardware compression not software.
I have defragged all of the servers drives and the backup system also.
Antivirus is stopped on both the backup system and all servers while the backup is ran.
I'm getting on average a speed of 650 MB per minute.
I know that's pretty fast however I would like to see what can be done to speed that up.
Suggestions/solutions welcome!
LVL 10
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

What backup software are you using?  Is it agent-based or are you just backing up off of shares?
mcrosslandAuthor Commented:
Backup Exec 2010 R2.  All patches and service pack up to date.
Agent based.
Andrew Hancock (VMware vExpert / EE MVE^2)VMware and Virtualization ConsultantCommented:
Add another 1GBbe Network Cards to servers and Backup Servers, and Ethernet-Channel or Bond them to give additional throughput.
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

hanccocka is right.  About the only thing you could do would be to create a separate VLAN or subnet that backups ran over.  Otherwise 650Mb/minute is good.
mcrosslandAuthor Commented:
How can I ensure that would actually give me a faster backup?  
I need to prove this in order to justify the cost.

FYI:  I just ran a test backup of 54 GB of data from a local hard drive of the backup server to TAPE.
It ran at 5,699 MB per minute!   So now I know my range of current speed VS. Fastest local speed.
I'm sure some of it has to do with the processing power of each individual server also.
In order to prove it you would have to set up a test environment with a single server on a separate subnet across the switch.  Pull a large backup across and compare your speeds.  Theoretically you should get more throughput but that is just theoretical.
mcrosslandAuthor Commented:
Should I go with separate identical NICS or dual port nics?
Separate NICs would guarantee a true test but if you have an unused NIC on a dual card you can test with it.  
How to speed up your backup? You can setup incremental backups so it only has to backup files that changed since last backup. That would speed it up also unless your talking about a half TB of new data every day which is alot of data ....
Two things that you need to understand that haven't been mentioned.

1. Bonding NICs could actually decrease overall throughput.  It depends on the architecture of NIC and bus type.  If using older PCI-X then every packet competes with every other packet on the bus.  Better go PCI-e NICs with onboard processors  .. but CHECK perfmon to see if network I/O is causing CPU overhead.

2. The networking most likely isn't the bottleneck anyway.   It is probably your target disks.  What are you using, cheap SATA drives and a RAID5 config that doesn't have onboard battery-backed up cache?   I've seen such configurations write a whole 20MB/sec, especially on an unaligned NTFS partition that was built with the default settings and a grossly incorrect RAID block size
Thomas RushCommented:
560 MB/minute is not good... that's only 10MB/second, which is old, old DLT tape-class speeds.
LTO from the first generation has been better.

Networking could be an issue if you've got a weak link somewhere, and the actual connection isn't the gigabit you believe you have.  That 10MB/second is much more in line with 100Mbit networking infrastructure.

I agree with dlethe that your target disks could be an issue... if it's a cheap RAID controller with consumer-grade disks, you're not going to see anywhere near the performance you hope to get.  Add the problems of misconfigured settings, and things go bad fast.

One other option that might be workable for you is to get rid of Backup Exec and move to an incremental forever/synthetic full backup with either HP Data Protector or IBM TSM.   With these products, you can perform a full backup once, then always incremental backups.   Periodically (weekly or monthly) you tell the backup application to create a synthetic full backup; it will pull the files already backed up to disk to create a backup exactly as if you'd run an actual full backup at that instant.   Of course, this won't solve your initial slow backup issue... you need to get that sorted out... but it is a very attractive option for some.
Ron MalmsteadInformation Services ManagerCommented:
Backup-to-disk (B2D), runs much faster... mainly because you can run multiple backup jobs simulataneously.

Lots of people use backup to disk, then send disk based backup files to tape... to keep the backup jobs from running over into the next work day.

Yeah I just calculated my speed and I don't even have anything setup for performing backups. From my desktop PC to a 7 year old dell laptop direct ethernet cable from desktop card to laptop card I get 11 MB/sec transfer speed by just using windows explorer and copying to mapped drive. So that about the same speed you have currently without even doing anything fancy.
mcrosslandAuthor Commented:
After reviewing all of this, I have determined the following.
My Antivirus was not stopping on the servers correctly.  Once I did this, the backup completed in 8 hours instead of 11.5 hours at an average rate of 850.  My disk backup runs at almost exactly the same rate.
My tape drive is a SAS high end drive that just inhales data at a local rate of 5,600!
I monitored each server as it was being backed up one night and found that bandwidth is not an issue except for 1 small server that still has a 100MB NIC in it.  That one was spiking at around 80%.
Two things I haven't done yet is try incremental backups as suggested or a VLan for the backup system.
Note: all of my servers are high end hardware based Smart Raid hp servers.
So, my problem was with my antivirus.  

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mcrosslandAuthor Commented:
Problem was with antivirus software not stopping properly on each server prior to backup.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.