Hi,
I am currently having a little trouble with the speed of my rsync command between 2 ready nas DUOs.
The rsync process runs between 2 netgear ready nas duo devices and the files involved are backup files created by Acronis True Image Backup software. These files are very large (a full backup is about 70GB and the differential backups are about 15GB each per day). This means a single directory can have over 100GB made up of just a few files within.
The command I am using currently outside my bash script to see the progress live is:
rsync --verbose --progress --stats --recursive --times --perms --links --delete /backup/* READYNAS01::backup
The command works how I intended but I believe I require more knowledge of what rsync is actually doing behind the scenes so I can understand why it is taking so long and how I can fix this issue.
I am guessing the problem is where rsync compares the directories for changes.
Does it look through the entire file for where data has changed or does it simply compare dates and sizes?
I find that with the above used information output switches (--verbose --progress --stats) there is a long period of time between outputting the directory and before actually transferring the files found different within that directory.
Is it looking inside the 100GB of data for changes instead of just the comparing their sizes and dates to know if to overwrite them with the sources files.
Is there a switch I have missed that can fix this?
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
Also I typically run the command with only the following switches.
--delete --compress --stats --archive
I have it automated on a linux system via cron so I don't need the progress switch and the archive switch contains several other switches, compress will compress the data prior to sending it across the network.
James
Thanks for the feedback so far and sorry for the delayed reply over the weekend.
FIrst of all a little more info on the environment:
Network:
The source and destination are linked on a LAN via a gigabit switch and cat 5 cable (although when the transfer starts I am getting extreamly low speeds of 400kB/s).
File info:
The files range from 10GB to 80GB in size each.
Approximate rsync share's (backup) disk usage is 300GB of a 1TB RAID1 array.
Back to the problem:
Just to get this straight what is rsync doing between verbosing the following 2 lines.
--------------------
s
W:\SERVER_E_DRIVE\Scri
-----
This is one point where the process takes a long time (over 2 hours).
What is it actually doing at this point (I notice that the rsync process on the destination server is very CPU bound (running at 60-100%) but the source rsync process is not using the CPU much at all.
bnoyzy,
Am I correct in assuming if you do an ifconfig from the server running the rsync you don't seen any error's on the network card?
Very high cpu on the destination server is normal. The larger the files the longer you'll see the high cpu as it processes the file.
How much memory is on both of these servers? Rsync is very memory intensive.
What file system are you using on both of them?
Bnoyzy,
Here are a few things to try.
I did find some information online about ext3 and the speed of the file system. You can change the setting to speed it up
http://www.ubuntugeek.com/
I've never done it so make sure your data is backed up prior. I typically run reiserfs.
If you need the rsync command running from the nas devices I would try upgrading the memory.
Ideally you would use another computer to run the rsync. The second computer would need between 2 to 4 gigabytes of memory. Then you would run rsync from the computer, mount both nas devices and rsync between the mount points. That way no processing of the rsync was happening on the nas devices. I would go this route if I were you. Just get a decent cpu with a few gigabytes of ram. Load a version of linux that has rsync many do.
James
Thanks for the feedback guys but I believe the answer is not spending money in this case as this is a job for a client. I have just tried mounting the other NAS with NFS and using the cp -r -u <source> <destination> command. Just off a rough calculation I was transferring 1GB per 3mins so definetly nothing wrong with the network, and so definetly a problem with the CPU and/or ram not being sufficient enough for the rsync process.
I am just going to write a bash script to use the cp command instead. I hope to get to this today so I will let you all know how I go.
Read on how to improve:
http://samba.anu.edu.au/rs
Good link gheist but unfortunatley none of the resolutions there were of help to this case. I have gone to using cp which seems to be working much faster as it does not care about what the internal structure of the files really are, it just looks to if its there or not and copys it if it isn't. Sometimes a dumber process is often much faster as in this case. Rsync is great as I use it between other systems but it just didn't cut it for this odd scenerio.
Business Accounts
Answer for Membership
by: jmcse1Posted on 2009-09-04 at 07:41:28ID: 25260153
Rsync builds a listing of the source directory first, if the link to your destination READYNAS01::backup is slow then this will take along time. Then rsync compares the files at the byte or delta level for changes. Then it transfers just the changed bytes so it wouldn't have to transfer the entire file, this is what makes rsync fast once you have a copy of the data, however if there are large changes and a slow connection it can still fall behind.
James