Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 713
  • Last Modified:

Need a reliable method to copy large amounts of files

Hello Experts!

I have a problem in that I have to transfer hundreds of gigabytes of files from a linux server to an external usb drive and the fact that the transfer will literally take weeks to accomplish.

I have the fear that once the transfer begins, something will happen that will cause the copying to stop, and I will be stuck with part of the files transferred and the rest not.

Is there a method I can use whereby I can start the transfer, and if anything happens in mid-stream to stop it, I can restart where I left off?

Thanks!
0
OmniUnlimited
Asked:
OmniUnlimited
  • 4
  • 4
1 Solution
 
woolmilkporcCommented:
I think "rsync" is the tool you should use.

"rsync" will transfer only those files which either do not yet exist on the target or those which have changed in size or in last-modified time.

You use it just like "rcp", specifying source and target, one of which may be remote.

There is a lot of options, best see  the "rsync" man page for details:

http://rsync.samba.org/ftp/rsync/rsync.html

A quite common way of invoking rsync is

- for local copy:

rsync -avz /source/data /target/directory

- for remote copy:

rsync -avz /local/data remotehost:/target/directory

-a implies recursion, copy symlinks as links, preserve permissions, time stamps, group and owner

-v means "verbose", and "-z" forces compression while transferring

Please note that rsync will create an additional directory level on the target if invoked the way I posted above - the copied files and directories will go to /target/directory/data.
To change this behaviour so that  creating an additional directory level at the destination is avoided add a trailing slash to the source specification, e.g.

rsync -avz /local/data/ remotehost:/target/directory

The copied files and directories will now go directly to target/directory.
0
 
OmniUnlimitedAuthor Commented:
Thanks woolmilkporc (love the moniker by the way) for your response.  Do I have to run it verbose (I really don't need to see output, I was going to do some checking from time to time by just seeing how many files had been transferred and if the process is still going or not.  I plan on running this job in the background) and do I need compression?  (I'm transferring to a disk that is exclusively for these files.)

So are  you saying that if, for whatever reason, the copying (or in this case resyncing) stops, I can restart using the same commands, and it will restart from the point it left off?  Or does have to go through and check everything again?
0
 
woolmilkporcCommented:
You don't have to run it in verbose mode, nor do you have to use compression (just omit "v" and "z").

You restart the transfer by specifying exactly the same commands. "rsync" will not really resume where it has left off, but the check it performs is really fast, so you won't loose too much time.

If you're going to transfer really big files consider using the "--partial" option. This way partially transferred files wil be kept on the target, and rsync will transfer just the remainder of the partial file at its next invocation.
0
Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

 
OmniUnlimitedAuthor Commented:
Ok, another question: the reason I asked this question is because I already had a failure on one other disk I have.  On that disk I had used the cp command (with obviously no way to start from where it ended.)

Could I use rsync on it and catch up to where it is quickly and finish up its transfer?
0
 
woolmilkporcCommented:
Yes, of course.

"rsync" performs its checks regardless of whether the previous copy has been done by "rsync" itself or by any other tool, that's to say regardless of how the files found their way to their current location.

Oviparous Woolmilkporc

(by full name)
0
 
OmniUnlimitedAuthor Commented:
LOL!  I think we have a new king of the beasts in our midst.

Oviparous Woolmilkporc, you have been very helpful.  One last question:  will the --partial option slow down things a bit?
0
 
woolmilkporcCommented:
Not really.

During the initial transfer this option just makes rsync keep a partiallly transferred file on the target instead of deleting the stub.

On subsequent transfers, when rsync finds a file on the target to be a fragment it adds the missing data to it. A little effort is needed to find the right point to resume, but this should be minute.
0
 
OmniUnlimitedAuthor Commented:
Oviparous Woolmilkporc, you certainly rule in this kingdom.  Thank you so much for your expert help.  It is much appreciated!
0

Featured Post

Receive 1:1 tech help

Solve your biggest tech problems alongside global tech experts with 1:1 help.

  • 4
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now