Solved

Transfer multuple files simultaneously by FTP in linux

Posted on 2008-06-25
8
534 Views
Last Modified: 2013-12-16
Hi,
I have a centos 4 box , I need to transfer thousands of files through a wan connection to another box , I am looking for a text based FTP client which can upload several files in parallel to my new box to maximize my throughput. I tried lftp which seems have this capability , but I had no success with it. any help is appreciated.
Regards
0
Comment
Question by:fifthelement80
  • 4
  • 4
8 Comments
 
LVL 1

Expert Comment

by:danstuken
ID: 21864989

can you not use the 'mget' / 'mput' commands?  I would think the overall time of sequential transfers done that way should be similar to that achived with parallel transfers.

Alternatively try tarring the files into a compressed archive first : tar cvfj archive.tar.bz2 path/to/files/*

Then uncompress at the other end using tar xvfj archive.tar.bz2

0
 
LVL 1

Expert Comment

by:danstuken
ID: 21865001

Oh!

don't forget to turn off the prompt before each file if you opt for 'mget'/'mput' commands.  Issue the 'prompt' command before the mxxx command.

e.g.

ftp> prompt
ftp> mput files.*

0
 
LVL 7

Author Comment

by:fifthelement80
ID: 21865023
mput transfers files 1 by 1 so it is slow , I need to transfer them in parallel to maximize my network throughput.
0
 
LVL 1

Expert Comment

by:danstuken
ID: 21865072

Really?  I would be surprised if it makes that much difference especially compared to a compressed archive transfer.  If your heart's stuck on parallel transfers, then you could script up something using wget to do fetches from the remote end perhaps.  Or look at rsync.  But I still reckon you'd be finished with a sequential transfer by the time you got those solutions working.  I have incomplete understanding of TCP/IP but is a single connection so inefficient at using bandwith that further connections will be able to transmit data without impeding the bandwidth used by the first connection?  I find that hard to believe.
0
VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

 
LVL 7

Author Comment

by:fifthelement80
ID: 21865120
Thats the same theory why download managers are faster than windows regular download.
the same theory applies to upload by FTP.
0
 
LVL 1

Expert Comment

by:danstuken
ID: 21873745
Hmm, I'm  more inclined to believe the dl managers work because hosting servers throttle rates on a connection so opening more connections gets you more bandwidth - this probably doesn't apply to your local lan.  But that doesn't help you with your problem.

What have you tried in lftp?  Are you trying to script it from bash or using its own -f command line option?  

0
 
LVL 7

Author Comment

by:fifthelement80
ID: 21874192
I am not on a Lan , I am on Wan.
neither , I just thought it may work like my cuteftp pro client , which I can set to use 10 cuncurrent sessions to upload files.
0
 
LVL 7

Accepted Solution

by:
fifthelement80 earned 0 total points
ID: 21967980
I found a simple solution , open a file and copy the following commands in it :

open ftp.site.com
user USERNAME PASSWORD
mput 0* 1* 2* 3* 4* 5* 6* 7* 8* 9* &
mput a* b* c* d* e* f* &
mput g* h* i* j* k* l* m* &
mput n* o* p* q* r* s* &
mput t* u* v* w* x* y* z* &
mput A* B* C* D* E* F* &
mput G* H* I* J* K* L* M* &
mput N* O* P* Q* R* S* &
mput T* U* V* W* X* Y* Z* &
exit

save the file and run it by command : lftp -f filename
it uploads files by 9 concurrent connections.

0

Featured Post

Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
The purpose of this article is to show how we can create Linux Mint virtual machine using Oracle Virtual Box. To install Linux Mint we have to download the ISO file from its website i.e. http://www.linuxmint.com. Once you open the link you will see …
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now