Solved

Rsync to list files that did not get copied to destination directory

Posted on 2014-09-15
9
522 Views
Last Modified: 2014-09-18
rsync -avzh path/to/directory/A/ path/to/directory/B

The above commands copies everything under directory A to directory B. Sometimes during the copy the files in destination B are locked by some other process which makes the copy incomplete. Is there a way in Rsync to list all such locked files that did not get copied over to the destination.
0
Comment
Question by:gaurav sharma
9 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40323400
Add "--dry-run" to the options of your command.

rsync will tell you which files are still to be copied without actually transferring anything.
0
 

Author Comment

by:gaurav sharma
ID: 40323519
Dry run gives me the list of all the objects that get copied to the destination.
I only need the list of files that are not transferred from source.
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 300 total points
ID: 40323542
Since rsync (the way you use it) will copy only files which do not yet exist (or are newer than those) on the destination "--dry-run" should do what you desire. In other words, "--dry-run" will list only files or file versions that have not been transferred from source yet.
If you don't want to see files which have been created or modified in the meantime (after the failed run but before "--dry-run") on the source I fear you will have to rely on the error log of the previous (failed) run in question to find out the non-copied files.
0
 
LVL 37

Expert Comment

by:Gerwin Jansen
ID: 40324255
Try adding a log file and check that you've run the rsync command:

--log-file=FILE
0
Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 13

Expert Comment

by:Sandy
ID: 40324578
Gaurav.. but i would suggest doing the rsync on open files won't be done.. i tried it numerous times but always it has a need of downtime to get the migration done.

I would suggest take the downtime with app owner and do the rsync, else open or locked files will always be acting like this only.

Ty/SA
0
 

Author Comment

by:gaurav sharma
ID: 40328223
Thank you for all the comments. I have a folder of size 4 gb that needs to be copied to a location. It should overwrite everything each time on the destination. The copy fails many a times since the files in the destination are locked. I need something that would tell me which files could not be copied due to that encountered lock.
0
 

Author Comment

by:gaurav sharma
ID: 40328225
The copy can be done using cygwin or windows.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 40328232
Consult the logfile, I keep telling you.
0
 
LVL 27

Assisted Solution

by:serialband
serialband earned 200 total points
ID: 40328617
I concur with woolmilkporc.  Consult the log file.  You'll have to generate it.

It sounds like you're running rsync on a Windows system.  In the Linux/Unix world, files do not get locked that way.  They are locked by inodes and file names are overwritten with new files with different inodes.  The old files may still be running, but the file names are replaced.  You only need to restart the process for it to disappear and load the new file.


On Windows, you can't ever overwrite the locked files because those are locked by the file names.  No matter which method you use to copy to a locked file, you can't overwrite it.  You also can't load it to read.  The only workaround for reading a locked file for copying is to enable VSS.  You can then use an open sourced 3rd party tool called hobocopy to make the copy.  http://hobocopy.sourceforge.net/ https://candera.github.io/hobocopy/

Unless you're copying from a linux or unix box to your windows box, it's better to use robocopy or hobocopy if you want to presevere windows ACLs.  rsync is better for unix copies.  Even with the windows tools, you need to consult the log file.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now