Solved

Suggestions regarding copying files in UNIX

Posted on 2011-09-08
9
224 Views
Last Modified: 2016-02-10
Hi,

Right now we are copying files from windows to UNIX using R-sync utility . But  sometimes it is getting hung up due to overload of data.

 We are using R-sync to compare checksum and date of the files to copy over.

Is  there any other best utility to replace R-sync.
0
Comment
Question by:new_perl_user
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
9 Comments
 
LVL 9

Expert Comment

by:parparov
ID: 36514133
rsync has an option --bwlimit=XXXX to limit its data transfer rate. The limit is given in kilobytes. Use --bwlimit=10000 to limit to 10 (9.7 to be precise) Mbyte/sec.
0
 
LVL 62

Expert Comment

by:gheist
ID: 36530134
rsync will only access disk and do very little over network connection if file sets (or their big parts) are same, but keeping reading all data at full disk speed at the same time.
If this influences the rest of the system you can "renice" rsync process on UNIX.

Please tell us version of FreeBSD you are using, version of rsync, and version of SSH.
Also would be nice to know command line used to invoke rsync if it needs some parameter tuning.
0
 

Author Comment

by:new_perl_user
ID: 36531521
Hi.

Below is the command we are using and invoking it through a script.

$RSYNC ${DRYRUN} ${OPTIONS} --filter="exclude ${TEMPDIR}" -e "ssh -i ${KEY}" ${STRING}:${DIR}/ ${LOCALDIR} |$AEE ${TMPLOG} >> ${LOGFILE} 2>&1

Is there a way ( like kind of alert messages) to know when the rsync breaks /stops copying files.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 9

Expert Comment

by:parparov
ID: 36531554
we need to see what ${OPTIONS} are during the actual run.
To see if rsync exited abnormally, test $? value after rsync's execution. If everything was alright, then it is 0, otherwise it contains the error code of the problem rsync encountered.
0
 

Author Comment

by:new_perl_user
ID: 36531613
options being used are as follows.

OPTIONS=' -rltzc --chmod=u+rwx,go=rx -u -K -v -i  --exclude=Thumbnail --exclude=Thumbs.db '

And where should we actually use this "test $?".  Will it be able to write to a log file.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36531700
you may add the --bwlimit I posted earlier to OPTIONS sequence.
Right after the rsync execution, add something like that:
if [ ! $? ]
  then
	echo "`date` Rsync exited abnormally with code $?" >> /your/log/file/location
fi

Open in new window

0
 

Author Comment

by:new_perl_user
ID: 36532068
Thank you Will try..
0
 

Author Comment

by:new_perl_user
ID: 36543539
Hi,
One more quick question. Will the R-sync be able to copy files  by reading a file with all the filenames it should copy.
0
 
LVL 9

Accepted Solution

by:
parparov earned 500 total points
ID: 36543679
from rsync's manual:
            --files-from=FILE       read list of source-file names from FILE

Open in new window

0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
unix scripting question 1 115
FTP on FreeBSD server 2 154
Unix/bash: scripted arithmetic 13 102
replace content of certain columns in a *.csv file by a defined string 12 80
This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
I have been running these systems for a few years now and I am just very happy with them.   I just wanted to share the manual that I have created for upgrades and other things.  Oooh yes! FreeBSD makes me happy (as a server), no maintenance and I al…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question