run multiple scp then wait

we have a massive volume of files to scp,we need to run them in batches
like 10 scp in the background then wait till they finish then start 10 more etc..
what should we do?
LVL 11
it-rexAsked:
Who is Participating?
 
gheistConnect With a Mentor Commented:
3 ideas, choose one:

1)
xargs -P `nproc` -I abc scp user@host:xxx ./xxx

2)
for x in a b c d ; do
 scp $x host:$x &
done
wait
3) rsync?
0
 
omarfaridCommented:
how do you run the scp commands today?
0
 
it-rexAuthor Commented:
u mean it is old..
LOL
I understand.I tried rsync between our redhat 6.5 and exadata yesterday and it failed.
0
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
omarfaridCommented:
What I want to know, are you running the scp commands from a script?
0
 
it-rexAuthor Commented:
No just normal runs not scripted
0
 
omarfaridCommented:
you can do the following in a script:

scp command1
scp command2
scp command3
scp command4
scp command5
scp command6
scp command7
scp command8
scp command9
scp command10
sleep 300
scp command10
scp command11
scp command12
scp command13
scp command14
scp command15
scp command16
scp command17
scp command18
scp command19
scp command20

The sleep 300  command will hold the script 5 min before resuming next 10 scp commands
0
 
it-rexAuthor Commented:
Say
10 scp running at once
I wanna wait
Till the first 1 finishes then kick the 11th
Or say 2 finishes after I wait I wanna kick the 11th and the 12th..
How to script that?
0
 
it-rexAuthor Commented:
So I have only 10 scp running at a time not more.
0
 
it-rexAuthor Commented:
Rsync does 1 file at a time...

The issues with wait is im not sure how should wait till the previous file gets copied..

What does the 1st one do?
0
 
omarfaridConnect With a Mentor Commented:
How do you run the scp commands? You may do similar to below:

for job in scp1 scp2 scp3 scp4 scp5 scp6 scp7
do
    count=`ls /tmp/lock* | wc -l`
    while true
    do
        if [ $count -lt 3 ]
        then
           break
        else
           sleep 10
        fi
     done
     touch /tmp/lock_$job
     $job
done

The jobs scp1 scp2 scp3 scp4 scp5 scp6 scp7 are scrips that do the scp commands
0
 
gheistCommented:
First starts 1 scp per processor until list of files is exhausted. it is more like $ cat listoffiles | xargs -P 10 ....
Second starts heap of SCPs in background and waits until done (note the & before done and wait command after)
Rsync transfers 1 file at a time, but never transfers same data twice
0
 
omarfaridCommented:
One more point, each scp script should remove the lock file from /tmp dir to let other scripts run.
0
 
serialbandConnect With a Mentor Commented:
The reason you might have speed ups by running separate simultaneous commands is that your ssh window size is set too small.  You should speed up single scp or rsync by adjusting the size.  http://serverfault.com/questions/434724/multi-thread-rsync-transfer

You might switch to hpn-ssh and you'll just have to run a single scp command.  http://www.psc.edu/index.php/hpn-ssh

Why isn't your rsync working?  Do you need to specify ssh?

rsync -av -e 'ssh'  path user@remote:path

If you nfs mount your remote filesystem, you could use cpio, which will be slightly faster for the initial copy than rsync or scp.  rsync can be used after you've copied everything and only need to copy the deltas.

To address your question directly:

Why not just run 10 scp with wildcards?  Why are you copying 1 file at a time with each SCP command?

The same goes for rsync.  You could run 10 separate rsync on different portions of your file system.

You could run something like this.

scp -pr /path1/ user@remote:/path1/; scp -pr /path21/ user@remote:/path21/ ; scp -pr /path31/ user@remote:/path31
scp -pr /path2/ user@remote:/path2/; scp -pr /path22/ user@remote:/path22/ ;  ...
scp -pr /path3/ user@remote:/path3/; scp -pr /path23/ user@remote:/path23/ ;  ...
scp -pr /path4/ user@remote:/path4/; scp -pr /path24/ user@remote:/path24/ ;  ...
scp -pr /path5/ user@remote:/path5/; scp -pr /path25/ user@remote:/path25/ ;  ...
scp -pr /path6/ user@remote:/path6/; scp -pr /path26/ user@remote:/path26/ ;  ...
scp -pr /path7/ user@remote:/path7/; scp -pr /path27/ user@remote:/path27/ ;  ...
scp -pr /path8/ user@remote:/path8/; scp -pr /path28/ user@remote:/path28/ ;  ...
scp -pr /path9/ user@remote:/path9/; scp -pr /path29/ user@remote:/path29/ ;  ...
scp -pr /path10/ user@remote:/path10/; scp -pr /path30/ user@remote:/path30/

Run each of the above lines in a separate terminal window.

If it's files your could change it

scp -pr [A-H]* user@remote:/path/
scp -pr [I-P]* user@remote:/path/
scp -pr [O-U]* user@remote:/path/
.
.
.
scp -pr [o-u]* user@remote:/path/
.
.
.
scp -pr [0-9]* user@remote:/path/;
0
 
gheistCommented:
You dont have to force rsync to use SSH, it is default for last 7 years or so.
0
 
it-rexAuthor Commented:
thank you
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.