Solved

Transfer multuple files simultaneously by FTP in linux

Posted on 2008-06-25
8
529 Views
Last Modified: 2013-12-16
Hi,
I have a centos 4 box , I need to transfer thousands of files through a wan connection to another box , I am looking for a text based FTP client which can upload several files in parallel to my new box to maximize my throughput. I tried lftp which seems have this capability , but I had no success with it. any help is appreciated.
Regards
0
Comment
Question by:fifthelement80
  • 4
  • 4
8 Comments
 
LVL 1

Expert Comment

by:danstuken
Comment Utility

can you not use the 'mget' / 'mput' commands?  I would think the overall time of sequential transfers done that way should be similar to that achived with parallel transfers.

Alternatively try tarring the files into a compressed archive first : tar cvfj archive.tar.bz2 path/to/files/*

Then uncompress at the other end using tar xvfj archive.tar.bz2

0
 
LVL 1

Expert Comment

by:danstuken
Comment Utility

Oh!

don't forget to turn off the prompt before each file if you opt for 'mget'/'mput' commands.  Issue the 'prompt' command before the mxxx command.

e.g.

ftp> prompt
ftp> mput files.*

0
 
LVL 7

Author Comment

by:fifthelement80
Comment Utility
mput transfers files 1 by 1 so it is slow , I need to transfer them in parallel to maximize my network throughput.
0
 
LVL 1

Expert Comment

by:danstuken
Comment Utility

Really?  I would be surprised if it makes that much difference especially compared to a compressed archive transfer.  If your heart's stuck on parallel transfers, then you could script up something using wget to do fetches from the remote end perhaps.  Or look at rsync.  But I still reckon you'd be finished with a sequential transfer by the time you got those solutions working.  I have incomplete understanding of TCP/IP but is a single connection so inefficient at using bandwith that further connections will be able to transmit data without impeding the bandwidth used by the first connection?  I find that hard to believe.
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 
LVL 7

Author Comment

by:fifthelement80
Comment Utility
Thats the same theory why download managers are faster than windows regular download.
the same theory applies to upload by FTP.
0
 
LVL 1

Expert Comment

by:danstuken
Comment Utility
Hmm, I'm  more inclined to believe the dl managers work because hosting servers throttle rates on a connection so opening more connections gets you more bandwidth - this probably doesn't apply to your local lan.  But that doesn't help you with your problem.

What have you tried in lftp?  Are you trying to script it from bash or using its own -f command line option?  

0
 
LVL 7

Author Comment

by:fifthelement80
Comment Utility
I am not on a Lan , I am on Wan.
neither , I just thought it may work like my cuteftp pro client , which I can set to use 10 cuncurrent sessions to upload files.
0
 
LVL 7

Accepted Solution

by:
fifthelement80 earned 0 total points
Comment Utility
I found a simple solution , open a file and copy the following commands in it :

open ftp.site.com
user USERNAME PASSWORD
mput 0* 1* 2* 3* 4* 5* 6* 7* 8* 9* &
mput a* b* c* d* e* f* &
mput g* h* i* j* k* l* m* &
mput n* o* p* q* r* s* &
mput t* u* v* w* x* y* z* &
mput A* B* C* D* E* F* &
mput G* H* I* J* K* L* M* &
mput N* O* P* Q* R* S* &
mput T* U* V* W* X* Y* Z* &
exit

save the file and run it by command : lftp -f filename
it uploads files by 9 concurrent connections.

0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

This article will explain how to establish a SSH connection to Ubuntu through the firewall and using a different port other then 22. I have set up a Ubuntu virtual machine in Virtualbox and I am running a Windows 7 workstation. From the Ubuntu vi…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now