We have an XServe G5 running OS X 10.4.11 connected to a very full 2.7TB XRaid via FibreChannel. We also have an ADIC Fastor2 auto-loader LTO2 tape drive connected to the XServe via a SCSI-2 PCI card (Atto). The backup software is Retrospect 6.
The problem is when we do a backup we only get throughputs of around 500mb/minute (8mb/s), or about 3.5+ days to backup all 2.6TB on the XRaid (just the backup - this does not include the verify).
I feel there is a bottleneck somewhere. While I don't expect to achieve max speeds of the various connections, I do expect to get better than 5-10% of the rated throughput. Is this this the best I can expect or am I making a basic setup/configuration error?
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
I'd be very surprised if using software compression (vs. hardware) compression speeds things up any. I understand the argument that if less data is being squirted down the SCSI bus, then it should speed things up, but in my experience the bottlenecks will almost certainly lie elsewhere, and the additional CPU overhead will (often) actually slow things down.
But it would be a good idea to conduct a few tests with (say) 10GB of data - please let us know the results if you do.
Sorry for the long post!
LTO2 can run about 70-72GB/hr with the hardware compression and ~30GB/hr w/o compression (also your speed). There are a huge number of factors that will affect overall throughput, I will outline what I think are some of the biggest ones:
Hope this gets you a start,
Mark
1. Content - This is a marketing server so yes, it's full of many .TIFs, .JPGs, etc. Turning off compression may perk things up a bit, but the base throughput still seems too slow (see bottom).
2. The XRaid is dedicated for data. The 80GB server HD is the boot drive with all applications, temp files, etc.
4. We're looking into upgrading the Retrospect software.
5. The original backup strategy was to store the backup on a LaCie 4Big via FireWire 800. But even that took 2+ days to complete. And it provided no 'depth' for incremental backups as a complete backup almost fills the Lacie. The LTO2 autoloader was connected as a way of providing 'infinite' depth (just keep adding tapes...).
I created the 1GB scratch file and ran a series of backups using Retrospect. Here are the results:
Hardware Compression : 1224.4 MB/min
Software Compression : 923 MB/min
No Compression : 631.5 MB/min
But the specs for the HP Ultrium2 drive at the heart of the autoloader is 32MBps/64MBps (1920MB/min / 3840MB/min). While I don't expect to reach max quoted speeds, I would've expected something better than what I am getting.
Lexmar makes some great points, but I have to disagree with the suggestion that lots of small files slow down the tape drive itself and stop it streaming.
This is because packages like Retrospect glue all the files together sequentially (along with file metadata, of course) and squirt it at the drive in one long stream.
The bottleneck created by backing up many small files actually arises from the hardware, OS and network overheads arising from transferring many small files from disc storage to Retrospect (and of course Retrospect itself has to process the files' metadata and create headers, but this is a relatively trivial overhead). So bottlenecks caused by many small files can be resolved by speeding up hard discs, motherboard/cpu/memory.
It's interesting you're getting 2.5 x throughput on your scratch file than on the backups themselves... was the scratch file also on the RAID, or on the system disc?
I'm not by any means a Retrospect expert, but something else to check is that i) it's using a tape block size > 16k, and ii) is using fixed (as opposed to variable) block mode.
On most *nixes you can check this by doing an
mt -f <drive> status
after the backup software (in this case Retrospect, obviously) runs.
The scratch file was in a TEST folder on the RAID.
I'll admit that the bulk of my PC experience is on Wintel boxes, so permit a few stupid questions - Right now the RAID is mounted as a drive on the Desktop of the server. Is it possible the slow-down is because Retrospect is going through Finder to get to the RAID? Should the drive be mounted another way for more direct throughput?
TapeDude - Fixed block size comment takes me back to SCO Unix with Backup Exec (Before Symantec) when it was from Catus International and SCO Airbag for floppy based bare metal recovery and a DAT 2 drive. That was the only way in SCO to get the tape drives to stream steadily and avoid all the stop/start/rewind that would wear out a drive every 8 mos. I still remember walking out the door every night with 2 DAT tapes one for SCO and one for the Mac network used to fit everything for 2 servers and a dozen workstations on a single 12GB tape...
Ok Kirk,
The figure you have of 1224.4MB/min is pretty typical for what I see on a Tandberg/Exabyte Storageloader with an LTO-2 (IBM in my case) drive on mixed business data. I just compared a 334,000 file 284GB transfer from SATA RAID 5 (SW) on WIndows Server 2003 and it ran 1199MB/min in about 4 hours. Verify on same ran 1770MB/min. In same transfer set a different archive of operating system files was transfered to tape 47,429 files 160.3GB at 2266.9MB/min and 2673.9MB/min on verify.
On my system the performance is varies greatly with how well the hardware compression can do it's thing.
Industry wide on tapes all the published speed specs are based on assumed 2:1 compression. I have NEVER seen a production drive achieve better than 50% of those numbers under sustained operation.
Your numbers indicate to me that you are getting about all you are going to on LTO-2 and you are getting fairly good compression out of the tape drive itself.
Didn't see your last post. Desktop locations in Mac are just links to hard mount points ala linux/BSD so the actual location is not particularly important to speed. the only difference would be if the mount is a network share and not direct attached storage. SInce you already said you are fiberchannel it is a hard mount.
Just a thought and this is a Retrospect thing. How many files are in the 2.6TB full backup when Retrospect does its scan? If you are trying to backup 1 million plus files Retrspect can get downright pokey.
Try setting up a backup with <500,000 files and say 600GB or so of data at a time and see how the speed fares? You may find you have to split your backup into 3-4 smaller chunks. Retrospect spends a lot of time looking to see if a file really needs to be backed up or not. Best rule of thumb - if the scan takes more than an hour, you are trying to backup too much at one time.
Also, if you have been running those big 3.5 day long backups, you are running afoul of the tape cleaning timer/sensing. If you are not running with a cleaning tape in the loader at all times that hasn't expired, performance will go to h**l really fast. Retrospect needs to know the tape is there - usually the barcode on the tape starts with "CL". Watch out for your loader's cleaning settings as most loaders can do auto cleaning. Retrospect is smart enough to see the 'Clean Me' signal from the drive, pause the backup, run a cleaning cycle and resume the backup.
One other note, the speed and performance statistics generated by Retrospect are well known to be questionable - there is much grumbling on the forums about that at EMC. Don't take the numbers as absolute. Your concern is legitimate on my post at 1:05PM above on performance I was refering to the TEST backup not the production one. If the production backup ran at the speed of the test, I assume you be happy with it.
The RAID has 550,000 files on it. The initial scan takes under 30 minutes before Retrospect starts writing to tape. The TEST backup was a single 1GB file so it's not quite a real-world test. I'm running another backup now of 6,000 files totaling 6GB with no HW or SW compression. That should give me a better real-world sampling.
Something else that gives me pause is the amount of time it takes to backup disk-to-disk. Backing up the RAID to the Lacie 4Big connected via FireWire800 takes about 2.5 days. It makes me think there's a bottleneck in the basic backup process.
The tape-drive specs are advertised as 32MBps/64MBps (native/compressed). So a theoretical uncompressed throughput would be 1920MB/min. I'm getting about a third of that figure - I'm hoping that 0.33 is not the standard "correction-factor" to be applied to backup performance claims :).
Just completed the 6GB/6,000 file test backup. The overall throughput was 270MB/min. While it was running I noticed that the progress bar would occasionally pause, the text "Copying..." would change briefly to "Writing Index..." and then "Please wait...". After a few seconds "Copying..." would appear again and the progress bar would start moving. The initial throughput was over 700MB/min, but would drop down with each "Please wait..." before settling at 270MB/min for the final score. If I had to guess, I'd say the tape drive was outrunning the incoming data-stream.
Part of the problem is this is an "inherited" system. The people who set it up left with all their tribal knowledge. The tape drive is my own recent addition to supplement the disk-only backup solution currently in place. Last night was the first time I tried to do a full backup, and I quickly realized the results were too slow.
So it wasn't until now that I thought to check the server specs - this server has only 1GB of RAM. Checking the Activity Monitor during the backup shows the CPU nearly maxed out. I'm thinking that an additional 1-2GB of RAM would probably improve things.
Retrospect uses virtual memory well. If you have 1GB physical RAM then make sure you give it 1.5GB of virtual memory but, 2GB plus of physical RAM would be a very good thing. I think the Xserve G5 was 2GHz and was 400MHz DDR RAM which won't help you for I/O intensive activities. Backup is all about how fast you can move a lot of data from point A to point B.
Watch out, a lot of the tape drives are SCSI LVD but still have the 68pin plug. When things don't match up it drops back to SCSI2 Wide legacy mode which will kill the performance of the LTO-2 drive. If you are running an Adaptec card run the SCSI probe tool to confirm what mode each SCSI address device is actually running?
On the LaCie external drive, the answer is that Firewire is better than USB in actual throughput but neither interface was ever intended to handle big sustained transfers and all the little 1TB portables get really hot and tend to slow down after a few minutes. It is not suitable device for backing up a RAID system. When you need it, it will fail.
D2D is typically of like kind over the network on a gigabit Ethernet or faster. So you need a healthy switch on Gigabit Ethernet at the minimum. Retrospect is very efficient at network backup but is better suited to direct attached storage like a 4TB SATA RAID setup or iSCSI for primary disk based datastorage.
Have you tried the mt status command? Because if Retrospect is using a small block size or is in variable mode it will KILL throughput to the tape drive.
Given the number of files that are being backed up, you want to optimise disc performance, which means ensuring there's enough disc buffering allocated to minimise seeks required to access inodes (directory info) and actual file data. If you do add more RAM, it would be worth experimenting with different disc buffering allocations.
The Lacie 4Big is not a lilttle portable drive - it's four 1TB 3.5" drives mounted in a single enclosure. Maybe not an ideal backup solution but it does provide a large capacity in a robust, fan-cooled housing. Right now it's connected via USB 2.0 because the FireWire connections on the XServe always see three "uknown devices" even when nothing's connected (an apparently common issue I'm dealing with separately). As a USB 2.0 connection I'm getting about 650MB/min throughput during backups to the Lacie. When it was Firewire800 it was faster, but I don't have the exact number.
I put in a request for 8GB of RAM to max out this motherboard. Hopefully that will improve things somewhat.
See one of the things that some backup software can perform multiple backup streams whic drastically improve backup speeds. In my current backup software, Arcserve, I can backup 4TB of data is about 12 hours(sure from a Windows server)
I think you need to consider some other Enterprise level backup software. Over fibre, I don't see why you could not do a full backup in under 12 hours with that much data.
The lacie would be a ton faster if you used its eSATA port rather than USB or Firewire. I am not sure there is an eSATA card for the Xserve. You could get by with a SATA to eSATA adapter cable plugged into one of the motherboard SATA ports - would probably lose hot swap capability. Expect minimum 2-3X the speed you get over USB. There is a low profile adapter plate and cable from several sources (try StarTech or NewEgg) figure $20 and shipping.
I still think the SCSI is throttling itself down.
Is it possible to connect a quad-core desktop directly to the XRaid via the gigabit ethernet to perform backups? The idea is to have a separate desktop running Retrospect connected between the tape drive and the array, bypassing the XServe entirely. Would this work, or is the XRaid ethernet connection strictly for administering the array, not for data access?
The 8GB chips are in, the change-management request has been submitted and approved (thank you Sarbanes-Oxley...), and the upgrade has taken place. The system is peppier with all the RAM, but the backup performance has not changed - still roughly 650MB/min.
We're getting close to our end-of-year "change-freeze" when further modifications will not be possible (except for emergencies). We're going to remove the tape-drive and connect it to a stand-alone PC for more performance tests, but I think this thread has run its course. The points will go to Lexmar since he provided the most complete information (and pessimism :) ) about our current setup. As a result, we'll likely put in a budget request for a bigger array, and LTO4 (or 5 when available) autoloader, and a dedicated backup-server.
Business Accounts
Answer for Membership
by: dlethePosted on 2009-10-07 at 21:38:29ID: 25522358
You have a large number of unknowns, but one of the biggest bottlenecks is typically the software because the tape drive could be starting/stopping continuously, perhaps even every few seconds. This will kill performance.
Try a sanity check by copying a few GB of raw data directly to tape , i.e.
dd if=/dev/zero of=/usr/testfile bs=1024k count=1000 To create a large scratch file
Then
time tar cvf /dev/rmt/0m /usr/testfile.bin {substitute device name as necessary}