Solved

rsync

Posted on 2013-07-01
4
439 Views
Last Modified: 2014-04-01
HI

I am using rsync shell script to copy files from one server other.  Content of shell script is below

rsync -e ssh -avzp --remove-source-files $REMOTE_DIR $LOCAL_DIR \
                    --exclude-from=$FILTER > $TRANS_TMP 2>>$RSYNC_ERR_LOG

The shell script runs every 10 mints to copy files between server.  We place files in source directory using sftp tool.  Sometimes rsync copy files which is not ready to move  or we still uploading files to source directory.  Becuase of this reason we sometimes get truncated files pushed from source to target directory.  We want to modify rsync script only rysnc only completed files not partial files. what is option which avoids partial files and only move files which are completed
0
Comment
Question by:vadicherla
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 23

Expert Comment

by:nemws1
ID: 39291583
I don't think rsync does that (or would be able to know that something is still being uploaded).  What you should do is upload to a different directory on the same server and then move it once its up - there's still a window for this to happen, but much less so.

For example, what if there's a network issue and your upload stops for 20 seconds.  Should 'rsync' consider that 'uploaded'?  (It shouldn't - the file hasn't completed uploading yet).  The logic of "is it done or not" is something that you *can* do - with MD5 checksums or something similar, but it requires more logic than rsync provides.

What you could also do is to use a simple locking mechanism, but this assumes that you are uploading only one file at a time:

1) Remote site connects to your site and changes to the proper directory
2) Remote site creates an 'upload_in_progress' file (contains nothing)
3) Remote site starts uploading new file(s)
4) rsync script runs - checks to see if an 'uploading' file exists
  4a) If *no* 'upload_in_progress' file, run rsync command
  4b) If 'upload_in_progress' file is there, exit without running rsync command
        Next run in 10 minutes will try again
5) Remote site finishes uploading file(s)
6) Remote site removes 'upload_in_progress' file.
0
 
LVL 27

Expert Comment

by:skullnobrains
ID: 39292182
there's still a window for this to happen, but much less so

if the temporary uploads dir is on the same filesystem, there is no window since mv will not rewrite the inode but rather lik it to another name

in scp, you can for example add .part to your filenames, and rename the files once the upload is complete, and exclude those files in the rsync command

--

this is a bit dirty but it may be acceptable to you to use an option like --min-age so recent files will have time to download before they are copied.

--

since you remove the source files, i'm not sure rsync is the best tool.

you probably also can write a shell script that uses lsof to check wether files are completed before they are transferred. this is only meaningful if you cannot change the sftp command which would be way cleaner
0
 
LVL 6

Accepted Solution

by:
Ryan Smith earned 500 total points
ID: 39316293
make a temp directory and have the files uploaded there.  In your script have it move *.whatever extension it's looking for to move the whole uploaded files to your rsync directory.
0
 
LVL 27

Expert Comment

by:skullnobrains
ID: 39970785
what if the file gets moved while it is still being written to ?
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Find unused columns in a table 12 75
Problem with MySQL query - graph 3 27
BatchFile-Disk Detection 23 31
Need quicker response from an Execption table 11 24
Confronted with some SQL you don't know can be a daunting task. It can be even more daunting if that SQL carries some of the old secret codes used in the Ye Olde query syntax, such as: (+)     as used in Oracle;     *=     =*    as used in Sybase …
In threads here at EE, each comment has a unique Identifier (ID). It is easy to get the full path for an ID via the right-click context menu. However, we often want to post a short link within a thread rather than the full link. This article shows a…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question