Solved

Transfer multuple files simultaneously by FTP in linux

Posted on 2008-06-25
8
538 Views
Last Modified: 2013-12-16
Hi,
I have a centos 4 box , I need to transfer thousands of files through a wan connection to another box , I am looking for a text based FTP client which can upload several files in parallel to my new box to maximize my throughput. I tried lftp which seems have this capability , but I had no success with it. any help is appreciated.
Regards
0
Comment
Question by:fifthelement80
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
8 Comments
 
LVL 1

Expert Comment

by:danstuken
ID: 21864989

can you not use the 'mget' / 'mput' commands?  I would think the overall time of sequential transfers done that way should be similar to that achived with parallel transfers.

Alternatively try tarring the files into a compressed archive first : tar cvfj archive.tar.bz2 path/to/files/*

Then uncompress at the other end using tar xvfj archive.tar.bz2

0
 
LVL 1

Expert Comment

by:danstuken
ID: 21865001

Oh!

don't forget to turn off the prompt before each file if you opt for 'mget'/'mput' commands.  Issue the 'prompt' command before the mxxx command.

e.g.

ftp> prompt
ftp> mput files.*

0
 
LVL 7

Author Comment

by:fifthelement80
ID: 21865023
mput transfers files 1 by 1 so it is slow , I need to transfer them in parallel to maximize my network throughput.
0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 
LVL 1

Expert Comment

by:danstuken
ID: 21865072

Really?  I would be surprised if it makes that much difference especially compared to a compressed archive transfer.  If your heart's stuck on parallel transfers, then you could script up something using wget to do fetches from the remote end perhaps.  Or look at rsync.  But I still reckon you'd be finished with a sequential transfer by the time you got those solutions working.  I have incomplete understanding of TCP/IP but is a single connection so inefficient at using bandwith that further connections will be able to transmit data without impeding the bandwidth used by the first connection?  I find that hard to believe.
0
 
LVL 7

Author Comment

by:fifthelement80
ID: 21865120
Thats the same theory why download managers are faster than windows regular download.
the same theory applies to upload by FTP.
0
 
LVL 1

Expert Comment

by:danstuken
ID: 21873745
Hmm, I'm  more inclined to believe the dl managers work because hosting servers throttle rates on a connection so opening more connections gets you more bandwidth - this probably doesn't apply to your local lan.  But that doesn't help you with your problem.

What have you tried in lftp?  Are you trying to script it from bash or using its own -f command line option?  

0
 
LVL 7

Author Comment

by:fifthelement80
ID: 21874192
I am not on a Lan , I am on Wan.
neither , I just thought it may work like my cuteftp pro client , which I can set to use 10 cuncurrent sessions to upload files.
0
 
LVL 7

Accepted Solution

by:
fifthelement80 earned 0 total points
ID: 21967980
I found a simple solution , open a file and copy the following commands in it :

open ftp.site.com
user USERNAME PASSWORD
mput 0* 1* 2* 3* 4* 5* 6* 7* 8* 9* &
mput a* b* c* d* e* f* &
mput g* h* i* j* k* l* m* &
mput n* o* p* q* r* s* &
mput t* u* v* w* x* y* z* &
mput A* B* C* D* E* F* &
mput G* H* I* J* K* L* M* &
mput N* O* P* Q* R* S* &
mput T* U* V* W* X* Y* Z* &
exit

save the file and run it by command : lftp -f filename
it uploads files by 9 concurrent connections.

0

Featured Post

Portable, direct connect server access

The ATEN CV211 connects a laptop directly to any server allowing you instant access to perform data maintenance and local operations, for quick troubleshooting, updating, service and repair.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Apache module 5 87
Migrating a Linux server to VMware 3 141
SMTP log file for IMSVA 5 76
CentOS 7 wireless 2 84
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Fine Tune your automatic Updates for Ubuntu / Debian
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question