Incremental Backups EXTREMELY Slow

I have a file server that contains about 500GB of data. We are doing synthetic full backups. The current configuration has an incremental backup running every night. The server in question is on a gigabit switch and has a Gb NIC. Right now, incremental backups are taking 36-48 hrs and fulll backups are taking 80 hours to complete. This is completely unacceptable as we are required to rotate tapes once a week. I need some help in figuring out what the issue is.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Duncan MeyersCommented:
What tape drive are you using? It's quite possible that you're not getting data to the drive quickly enough. For example, an LTO-3's minimum data rate is 27MB/s. If you can't feed it data at that rate, the drive will write out a block, then wait for its buffers to fill before writing out the next block. The performance hit comes in when the tape drive has to reposition the tape to the end of the last block written (the tape will always over-run a little after it completes a block) the effect is called shoe-shining because the tape is constantly moving backwards and forwards. An LTO-3 on a Windows backup server can spend most of its life shoe-shining and it results in extremely poor performance, high error rates, and woeful tape drive reliability. LTO-4 minimum tape speed is much lower, so shoe-shining is less of a problem, but you still have to get data to the drive fast enough.

Enough of the theory. You're backing up 500GB in 80 hours. That's 6.25GB/hour, 104MB/minute or about 1.7MB/sec. That isn't flash... If you could write out at 30MB/sec then your a full backup should complete in about four and a half hours. Commvault can multiplex and multistream so you should be able to get decent performance. Here are some things to look at:

- How is the performance if you copy a large file (say, 500MB) from the file server to the backup server?
- What is the file server and what sort of disc is the 500GB data store on? Is it running Windows?
- What brand/model are the NICs on both the file server and backup server?
- Is it on a SAN or a local RAID controller?
- Do you have any file server performance issues during the day?
- What are the tape devices? Are they SCSI or SAN attached?
- Do you have appropriate licenses for the clients you're trying to back up?

Duncan MeyersCommented:
One more test...

- What sort of performance do you get if you backup some large files (>10MB each, about 10GB worth) from the local disc on the backup server itself?
jasonwebb2006Author Commented:
We are using an HP Storageworks SSL1016 Autoloader for the tape drives. However, we are running Disk to Disk to Tape.

- FIle Server is an HP Netserver with a 1TB IDE disk  (I Know....its not scsi, which could be a potential issue)
- It is running server 2003 r2 sp2
-Right now we are averaging about 1.2 Mbs for a 1.3GB folder.
-The tape drives are on a scsi device
-The nic on the backup server is an Broadcom NetExtreme Gb and the one on the file server is a Netgear Gb
- The licensing is not an issue.
The Ultimate Tool Kit for Technolgy Solution Provi

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy for valuable how-to assets including sample agreements, checklists, flowcharts, and more!

jasonwebb2006Author Commented:
The tape system is an lto2
Duncan MeyersCommented:
OK. The single ATA disk is a problem, but you should still get better performance than what you're seeing.

The 1.2MB/s average that you mention - is that the throughput during a backup or is that the result of the test I described? You can monitor network performance via Tas k Manager -> Networking tab.

You mention that you're using Disk to Disk to tape. Where is the disk that you're backing up to and how is it configured? Can you create a backup job that backs up direct to tape and check the performance for client direct to tape? Also test the performance when you're backing up a large file from the backup server itself direct to tape (as described in my post).

It is also worth checking that you've got the latest Broadcom drivers. Uninstall BASP software (the BRoadcom teaming software) if it's installed. Manually set the port speed and duplex of both the file server and backup server NICs to 1Gb full duplex. Also set the corresponding ports on your network switches. If you've got Cisco switches the config commands are:

conf t
int g0/<port number>
speed 1000
duplex full
spanning tree portfast
CTRL-Z to exit
write mem

Finally, what sort of performance are you getting as a user of the file server? How fast can you copy a large file from the server to your desktop?

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Duncan MeyersCommented:
Works for me!
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.