Solved

Backup hardware and software alternatives for large amount of tif images.

Posted on 2010-08-19
7
628 Views
Last Modified: 2013-12-01
Hi - I am looking looking for suggestions for a backup problem our office is having.  We use Symantec Backup Exec v12 (windows) to backup to tape (HP Storageworks Ultrium 1840 - LTO4) direct connection to server.

Currently, we have a rather large directory of .tif images, approximately:

16 million files, totalling about 780GB
converted = 817,889 280 KB
converted = 798,720 MB

We setup a FULL backup on Friday and Differential Backups Mon-Thurs

The problem is the job takes too long and eventually fails.  The Full Friday job runs approsimately 60 Hours and the Mon-Thurs is up to 11 hours.

Thats about a job rate of about 221MB per minute for the full friday job. LTO4 is rated much higher.

At some point within the next 3 months we are planning to implement a new BDR solution that should aid us with this backup and every other backup in the company, but the ploblem is that we need to get this corrected in the interim by any means.  Even though we replicate this entire server and have a working reduntant available off-site, we need to get some sort of nightly / weekly backup solution in place to comply with our partners regulatory auditors.
0
Comment
Question by:LenCepeda
7 Comments
 
LVL 6

Expert Comment

by:bluepig
ID: 33474727
As a first step, why not create two backup jobs - each job backs up half the data. It means changing the tape more often, but at least you will get the data backed up.
0
 
LVL 6

Accepted Solution

by:
dax_bad earned 250 total points
ID: 33474908
Do you only have a single LTO4 tape streamer?

What's the network bandwidth between the .tif server and the  backup server? 100Mbps / 1Gbps or more?

Is there any SAN attached storage connected to the tif server?

Im guessing you have a 1Gbps LAN

First,  i'd make sure that it's not the anti virus scanning every single file (on access scans) during backup, that could really kill the performance

Second, make sure your NICs and switches are all set to Full Duplex, not auto or half duplx or a mix because the lowest will trumph

Also keep in mind that it always takes more time backup of 780GB of small files rather than 780GB of a big files. And when does the replication occur? That could also slow down the performance of the backup and congest the LAN bandwidth.  

For your future project consider multiple tapestreamers and reserve 1 of them to this backup set and let the remaining streamers handle the normal backup / restores.

You will never get 100% NIC utilization on backup, it's wishfull thinking :-)

Cheers
Daniel
0
 
LVL 5

Expert Comment

by:tygrus2
ID: 33475075
Let me guess, are the files read from a single drive (SATA) or from multiple SATA drives but one at a time ?

It is probably not a problem of the backup software but the server hardware ie. the reading of 16M files. Running so slowly means the tape drive is running out of data, re-winding and writing 3secs of data stop, rewind ..
It is not good for the drive or tape life to be used like that. The system should be able to supply 40MB/s minimum and ideally >200MB/s (eg. 6x 10Krpm SAS in RAID5 array, >600 IO/s random 64KB reads or more if files are badly fragmented). My part tape backups take 1hrs-2hrs, but 4 hrs for a full tape with the newer tape drives that doubled their capacity while only a 20%-50% increase to max tape speed. I would not like a backup to go any longer than that because of the shoe shinning (stop-reverse-start).

Should be using RAID array, and a defrag program that arranges/orders the files by filename.

Please provide more details:
1) Is 20% added each week or 20% changed in the week ? Are files temporary to a job, randomly referred to or can they be archived separate to recent files ?
2) How similar are the files? Are they already compressed ?
3) Are they scanned pages of documents? It would be better if multipage documents were stored as one file eg. multi-page TIFF, fax document, PDF or 100 per ZIP file.
4) Should a document/image management system be used to arrange and save meta data for searching and grouping?
5) Are you accessing files from another system over a network or from a NAS ?
6) What server/hardware config do you have ? CPU, RAM, MB, disks, RAID, network (if applicable), OS ?

Some backup products allow you to interleave several backup sources (parallel) to read from multiple disk volumes independently. Some options allow you to stage a backup (need a spare 1.5TB HD) to reduce the backup window and load on the tape because by then it's sequential.
0
Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

 

Author Comment

by:LenCepeda
ID: 33476156
Thanks very much for your very quick responses.  I guess I should have included more information in my orginal post.

1. This is a single LTO 4 tape streamer
2. The backup software is installed on the server and the tape drive is an external attachment to the server (Ultra 320 SCSI)
3. There is a Windows Storage server on the LAN that stores archived images from the .tif server - 1Gbps
4. Our document (image) management software "Imageright" runs a nightly archive which deletes images from the .tif server (older than two years) and copies them to the Storage Server.  i thought the archive / deletion process copleted prior to the backup job running but i will confirm
5. The .tif server is a DELL poweredge 2850 6 350GB SCSI HD Raid 5
6. Images are stored and grouped by the "Imageright" application, most files are scanned single page .tif images with some pdf / .doc files mixed in.
7. AVG anti virus scan runs every night at 4am, which runs into the backup job.  i have just amended that to run weekly at 8pm on sunday
8. Double-Take replication is real time (i beleive) i will need to double check that!!!

Right now, i intend on disabling the the archive feature on the document management system this weekend and the anti virus scan to see if there is a difference this weekend.

thanks
0
 
LVL 3

Expert Comment

by:tussharawaghan
ID: 33480816
All your queries will be answered by this document.
http://support.veritas.com/docs/231488
0
 
LVL 5

Assisted Solution

by:tygrus2
tygrus2 earned 250 total points
ID: 33480964
So the tape drive is connected to the tif server not the Storage server ?
Are you doing backups over the LAN by reading from the Storage server (archived files)?
Are you using the backup agent on the server without the tape drive and is it properly configured and in use ?

You might have to test your setup after-hours. Can you run a few speed tests
1) 2GB of small files across network.
2) 4GB of >200MB files accross network.
3) Backup the above sets of files (2 jobs) from the local RAID and over the network.  
4) Test raw network speed. There are program that operates as client/server to test in both directions (eg. netspeed).
Check task manager, windows perfmon etc. for IO rate, network, CPU %.
5) Check any suggested system changes using the above.
0
 

Author Closing Comment

by:LenCepeda
ID: 33802618
THANKS
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

By default, Carbonite Server Backup manages your encryption key for you using Advanced Encryption Standard (AES) 128-bit encryption. If you choose to manage your private encryption key, your backups will be encrypted using AES 256-bit encryption.
Workplace bullying has increased with the use of email and social media. Retain evidence of this with email archiving to protect your employees.
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now