Solved

run multiple scp then wait

Posted on 2015-01-17
15
193 Views
Last Modified: 2015-02-08
we have a massive volume of files to scp,we need to run them in batches
like 10 scp in the background then wait till they finish then start 10 more etc..
what should we do?
0
Comment
Question by:it-rex
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
  • 3
  • +1
15 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 40555466
how do you run the scp commands today?
0
 
LVL 11

Author Comment

by:it-rex
ID: 40555471
u mean it is old..
LOL
I understand.I tried rsync between our redhat 6.5 and exadata yesterday and it failed.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40555516
What I want to know, are you running the scp commands from a script?
0
Get MongoDB database support online, now!

At Percona’s web store you can order your MongoDB database support needs in minutes. No hassles, no fuss, just pick and click. Pay online with a credit card. Handle your MongoDB database support now!

 
LVL 11

Author Comment

by:it-rex
ID: 40555517
No just normal runs not scripted
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40555528
you can do the following in a script:

scp command1
scp command2
scp command3
scp command4
scp command5
scp command6
scp command7
scp command8
scp command9
scp command10
sleep 300
scp command10
scp command11
scp command12
scp command13
scp command14
scp command15
scp command16
scp command17
scp command18
scp command19
scp command20

The sleep 300  command will hold the script 5 min before resuming next 10 scp commands
0
 
LVL 11

Author Comment

by:it-rex
ID: 40555653
Say
10 scp running at once
I wanna wait
Till the first 1 finishes then kick the 11th
Or say 2 finishes after I wait I wanna kick the 11th and the 12th..
How to script that?
0
 
LVL 11

Author Comment

by:it-rex
ID: 40555654
So I have only 10 scp running at a time not more.
0
 
LVL 62

Accepted Solution

by:
gheist earned 200 total points
ID: 40557180
3 ideas, choose one:

1)
xargs -P `nproc` -I abc scp user@host:xxx ./xxx

2)
for x in a b c d ; do
 scp $x host:$x &
done
wait
3) rsync?
0
 
LVL 11

Author Comment

by:it-rex
ID: 40557205
Rsync does 1 file at a time...

The issues with wait is im not sure how should wait till the previous file gets copied..

What does the 1st one do?
0
 
LVL 40

Assisted Solution

by:omarfarid
omarfarid earned 200 total points
ID: 40557233
How do you run the scp commands? You may do similar to below:

for job in scp1 scp2 scp3 scp4 scp5 scp6 scp7
do
    count=`ls /tmp/lock* | wc -l`
    while true
    do
        if [ $count -lt 3 ]
        then
           break
        else
           sleep 10
        fi
     done
     touch /tmp/lock_$job
     $job
done

The jobs scp1 scp2 scp3 scp4 scp5 scp6 scp7 are scrips that do the scp commands
0
 
LVL 62

Expert Comment

by:gheist
ID: 40557235
First starts 1 scp per processor until list of files is exhausted. it is more like $ cat listoffiles | xargs -P 10 ....
Second starts heap of SCPs in background and waits until done (note the & before done and wait command after)
Rsync transfers 1 file at a time, but never transfers same data twice
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40557238
One more point, each scp script should remove the lock file from /tmp dir to let other scripts run.
0
 
LVL 30

Assisted Solution

by:serialband
serialband earned 100 total points
ID: 40565474
The reason you might have speed ups by running separate simultaneous commands is that your ssh window size is set too small.  You should speed up single scp or rsync by adjusting the size.  http://serverfault.com/questions/434724/multi-thread-rsync-transfer

You might switch to hpn-ssh and you'll just have to run a single scp command.  http://www.psc.edu/index.php/hpn-ssh

Why isn't your rsync working?  Do you need to specify ssh?

rsync -av -e 'ssh'  path user@remote:path

If you nfs mount your remote filesystem, you could use cpio, which will be slightly faster for the initial copy than rsync or scp.  rsync can be used after you've copied everything and only need to copy the deltas.

To address your question directly:

Why not just run 10 scp with wildcards?  Why are you copying 1 file at a time with each SCP command?

The same goes for rsync.  You could run 10 separate rsync on different portions of your file system.

You could run something like this.

scp -pr /path1/ user@remote:/path1/; scp -pr /path21/ user@remote:/path21/ ; scp -pr /path31/ user@remote:/path31
scp -pr /path2/ user@remote:/path2/; scp -pr /path22/ user@remote:/path22/ ;  ...
scp -pr /path3/ user@remote:/path3/; scp -pr /path23/ user@remote:/path23/ ;  ...
scp -pr /path4/ user@remote:/path4/; scp -pr /path24/ user@remote:/path24/ ;  ...
scp -pr /path5/ user@remote:/path5/; scp -pr /path25/ user@remote:/path25/ ;  ...
scp -pr /path6/ user@remote:/path6/; scp -pr /path26/ user@remote:/path26/ ;  ...
scp -pr /path7/ user@remote:/path7/; scp -pr /path27/ user@remote:/path27/ ;  ...
scp -pr /path8/ user@remote:/path8/; scp -pr /path28/ user@remote:/path28/ ;  ...
scp -pr /path9/ user@remote:/path9/; scp -pr /path29/ user@remote:/path29/ ;  ...
scp -pr /path10/ user@remote:/path10/; scp -pr /path30/ user@remote:/path30/

Run each of the above lines in a separate terminal window.

If it's files your could change it

scp -pr [A-H]* user@remote:/path/
scp -pr [I-P]* user@remote:/path/
scp -pr [O-U]* user@remote:/path/
.
.
.
scp -pr [o-u]* user@remote:/path/
.
.
.
scp -pr [0-9]* user@remote:/path/;
0
 
LVL 62

Expert Comment

by:gheist
ID: 40565531
You dont have to force rsync to use SSH, it is default for last 7 years or so.
0
 
LVL 11

Author Closing Comment

by:it-rex
ID: 40597018
thank you
0

Featured Post

Optimize your web performance

What's in the eBook?
- Full list of reasons for poor performance
- Ultimate measures to speed things up
- Primary web monitoring types
- KPIs you should be monitoring in order to increase your ROI

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses

630 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question