Link to home
Start Free TrialLog in
Avatar of txarli33
txarli33Flag for United Kingdom of Great Britain and Northern Ireland

asked on

Backups are very slow

backing up a file server with backup exec i get very slow throughput 300 - 400MB/min
initially i thought the network would be the bottleneck but then i did a ntbackup locally on the server and found the same speed.
I'm getting 1.5 to 2 GB/min on other servers.
I'm quite new in this job and I can't tell where the storage resides,
but this server stores network drives for the company, around 200GB of storage in total
How can i improve the speed of the backup?

thanks for any help
Avatar of txarli33
txarli33
Flag of United Kingdom of Great Britain and Northern Ireland image

ASKER

i forgot to mention
servers (backup and file server) are 2003 standard
backup exec v11d
You will get slower backup speeds if you are backing up many small files, which is typical with a file server.  For example, a 70GB flat database dump will back up faster than 70GB worth of individual documents (Word, Excel, etc).  Are you doing full backups every night or incremental?  Incremental can help as you'll only back up whatever files have changed since the last backup job.
Avatar of Francois_IT
Francois_IT

A common problem for backup speed is a fragmented drive. It also depend if you are doing backup for millions of small files, indexing take long time to generate.
Fragmentation you have a decent amount of control over (run a background defragmenter like DiskKeepeer).
Small files, deeply nested directories, and long file names you have little control over.
Hosting data on a slow NAS device or a crappy (SW?) RAID controller you may or may not have control over (have you got budget for a better storage system?)

Before you try to solve a problem you're guessing at, run a tool like HP's free Library and Tape Tools and test the speed of the disk you're trying to back up.  If it reports a speed of 5-6MB/second, then it probably is your disk.

If you're stuck with the problems above, you have some of options:
1) If you rarely perform single-file restores, try backing up an image of your drive, instead of file-by-file.  An image backup reads the whole disk sequentially by sector, so it avoids the problems of small files or fragmentation.  But single-file restores will typically take a lot longer than they would from a filesystem backup.
2) Backup to disk first, then to tape.  This has some added cost, and possibly complexity, but the disk target won't care how slow the data comes in, and will organize it in to big blocks that can then be read fast and written to tape at a good clip.
2a) Use your backup software to create a backup-to-disk partition.  Cheapest option, but makes the server do additional work, makes the admin do additional work to manage the partition, is a one-server-at-a-time solution, and any data on the filesystem is subject to corruption from virus, accidental deletes, etc., so it's got challenges.  But it's low cost (usually just disk space, possibly the backup application license fee, and some admin time), and might be good enough.
2b) Or, buy a dedicated D2D target device that acts like a tape library, allowing you to use it for several or many servers.  Disadvantages: Costs more than server D2D.  Benefits: Looks just like a tape target so no new processes; space is shared among several servers if you choose (so scales); Good D2D systems can do tape offload directly; so the data doesn't have to go back through the backup server or over the network; Most D2D targets can do deduplication, so you can keep months of backups in the space that would only hold a couple of weeks without deduplication -- so single-file restores can be really fast; Probably the best-optimized for getting data off of your server quickest, and getting it to tape the quickest.

HP makes a D2D appliance called the D2D Backup System that would do all you need if you choose to go that route.  http://www.hp.com/go/d2d

Note: I do work for HP, but until that last line, the above is about as vendor-neutral as you'll find anywhere.  <smile>

guys I fully apreciate your response as my question didnt give a lot of information out
(well that's the state I am now regarding this server)

to answer some of your points I'll say:
- I am doing full backups every night as tapes get taken off site and incremental backups could make difficult restore jobs although I am considering doing differentials so i'd only need normal and last tape for restoring.
- I have bad experiences with diskkeeper where some harddrives end up dying (due to too much work/indexing maybe?) but it is an option to be looked into
- backup to disk, I've done tests backing up to disk on the same server im backing up, on the local disk, and I get the same speed so doing a b2d don't think will help much speed wise although I could fit more backups during the same time window as I could run b2 tape and b2 disk simultaneously
- I've also found out no SCSI or FC cable coming out of the back of the server to a disk shelf so I assume the way it connects to the SAN/NAS is via iSCSI through the 2 NICs which are teamed up, how could I check this? if it has software or hardware iSCSI initiator? would that affect the speed of the backups as well?, also, could the slow speed be due to wrong iSCSI drivers installed?
The iSCSI initiator configuration program on your server will tell you the IP address of the iSCSI storage you're connecting to.   Many storage boxes will have browser interfaces you can open by pointing your web browser to their IP address... this will give you a clue about partitions (LUNs) and such.

You could help some of your problems by performing a weekly full and daily differentials.   A differential backup doesn't re-set the archive bit, so that it backs up all files changed since the last full... to restore, you need at most the full backup, plus one differential tape.

The problem with sending data to tape if you can only hit 6MB/second, is that you're tearing up your tapes and yout tape drive motor.  Doing a backup to disk *should* let you then copy the backup to disk to physical tape much faster.than from teh fragmented, small-file-filled filesystem.  

Also re: speed of D2D backups: Yes, if the bottleneck is your server disk, then a faster target *won't matter*, even if the target were light-speed SSD... The  bottleneck is the source.   But again, a VTL can help by giving you an intermediate target that can be copied to tape faster, and which can increase concurrency (back up many servers at once to the VTL).

Maybe DiskKeeper isn't the best program of its type; I have used it and thought it did what it did well.... I know that Windows gets fragmented filesystems over time, and those will not only make your disk work harder, but affect performance as well.   (Oh for the days of HPFS!)


You might also find that the AntiVirus program on the server holding the data is checking every file as it moves.

I found this out after I implemented a backup of our e-mail archiver and it created hundreds of thousands of tiny ~30k files. Our Antivirus stopped each file and scanned it as it moved across the network so our backup time increased by around 12 hours. After I excluded the folder from the scanning it performed fine and I saved the 2 hours off our backup window.

Darren
thanks again for your replies and apologies for late reply (really hectic at work these days)

well i have some more information
i had the chance to have a look at the server room today and found out that the data is local, this server has 4 hdd installed on it where all data resides
being the data local to the server i don't understand why the slowness
the antivirus has the same settings in all servers, i'll investigate but don't see why should drag the process here and not on any other servers
apart from the antivirus suggestion, what other factors may slow this process?

apreciate all replies, thanks again
ASKER CERTIFIED SOLUTION
Avatar of Thomas Rush
Thomas Rush
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Quite impressed by your response, I have now lots to do here.
I thought I could just tweak some settings but it seems I'll have to do a lot of data investigation.
I'll follow your steps and hopefully i'll get a faster throughput

thanks again for the time you've all taken replying to this