how to copy large files across net?

Posted on 2009-04-24
Medium Priority
Last Modified: 2013-12-01
I am trying to do a relatively simple task. I want to copy a  15,618,606,818 (15GB) file from one host to another. I've tried ftp, rsync and gftp. ftp and rsync die with "file too large" errors. gftp is probably dying for the same reason, but doesn't give that as a message. The target device has 250GB free.

nfs is not a desired solution. Nor is breaking the file up into smaller files.

How do I do this?
Question by:jmarkfoley
  • 2

Expert Comment

ID: 24230690
scp wil transfer the file for you.

Command to use is:
scp myfiletocopy username@remoteip:remotefile

I've done 100GB files this way and is secure.

Expert Comment

ID: 24231358
rsync -vrPtz -e ssh host:/remote_path/* /local_path/


-e ssh rsync will use ssh client instead of rsh
-z compress file transfer
-t preserve time (other attributes as owner or permissions are also possible)
-P resume incomplete file transfer
-r recursive into subdirectories
-v verbose

if it stops running run it again Cheers!

Accepted Solution

martin_2110 earned 1500 total points
ID: 24231381
Read this wrong the first time, I would try upgrading rsync and you kernel to the latest version.   Any chance your rsyncing to fat, the limit of fat32 is 4gb.  What os and filesystem are you syncing to and from?

Author Comment

ID: 24232119
I used scp. It works and I like it. I like the progress info. as it turned out, the problem was with none of the client programs. They probably all would work. The problem was my target filesystem. I mounted an external usb drive which was preformatted to Windows FAT. I thought that would be nice because I could access the files from my windows laptop but, as everyone *should* know, there is a 2GB file size limit. So, I reformated it to ext2 and no problem.

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Microservice architecture adoption brings many advantages, but can add intricacy. Selecting the right orchestration tool is most important for business specific needs.
Windows Server 2003 introduced persistent Volume Shadow Copies and made 2003 a must-do upgrade.  Since then, it's been a must-implement feature for all servers doing any kind of file sharing.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question