Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 226
  • Last Modified:

run multiple scp then wait

we have a massive volume of files to scp,we need to run them in batches
like 10 scp in the background then wait till they finish then start 10 more etc..
what should we do?
0
it-rex
Asked:
it-rex
  • 6
  • 5
  • 3
  • +1
3 Solutions
 
omarfaridCommented:
how do you run the scp commands today?
0
 
it-rexAuthor Commented:
u mean it is old..
LOL
I understand.I tried rsync between our redhat 6.5 and exadata yesterday and it failed.
0
 
omarfaridCommented:
What I want to know, are you running the scp commands from a script?
0
Veeam Task Manager for Hyper-V

Task Manager for Hyper-V provides critical information that allows you to monitor Hyper-V performance by displaying real-time views of CPU and memory at the individual VM-level, so you can quickly identify which VMs are using host resources.

 
it-rexAuthor Commented:
No just normal runs not scripted
0
 
omarfaridCommented:
you can do the following in a script:

scp command1
scp command2
scp command3
scp command4
scp command5
scp command6
scp command7
scp command8
scp command9
scp command10
sleep 300
scp command10
scp command11
scp command12
scp command13
scp command14
scp command15
scp command16
scp command17
scp command18
scp command19
scp command20

The sleep 300  command will hold the script 5 min before resuming next 10 scp commands
0
 
it-rexAuthor Commented:
Say
10 scp running at once
I wanna wait
Till the first 1 finishes then kick the 11th
Or say 2 finishes after I wait I wanna kick the 11th and the 12th..
How to script that?
0
 
it-rexAuthor Commented:
So I have only 10 scp running at a time not more.
0
 
gheistCommented:
3 ideas, choose one:

1)
xargs -P `nproc` -I abc scp user@host:xxx ./xxx

2)
for x in a b c d ; do
 scp $x host:$x &
done
wait
3) rsync?
0
 
it-rexAuthor Commented:
Rsync does 1 file at a time...

The issues with wait is im not sure how should wait till the previous file gets copied..

What does the 1st one do?
0
 
omarfaridCommented:
How do you run the scp commands? You may do similar to below:

for job in scp1 scp2 scp3 scp4 scp5 scp6 scp7
do
    count=`ls /tmp/lock* | wc -l`
    while true
    do
        if [ $count -lt 3 ]
        then
           break
        else
           sleep 10
        fi
     done
     touch /tmp/lock_$job
     $job
done

The jobs scp1 scp2 scp3 scp4 scp5 scp6 scp7 are scrips that do the scp commands
0
 
gheistCommented:
First starts 1 scp per processor until list of files is exhausted. it is more like $ cat listoffiles | xargs -P 10 ....
Second starts heap of SCPs in background and waits until done (note the & before done and wait command after)
Rsync transfers 1 file at a time, but never transfers same data twice
0
 
omarfaridCommented:
One more point, each scp script should remove the lock file from /tmp dir to let other scripts run.
0
 
serialbandCommented:
The reason you might have speed ups by running separate simultaneous commands is that your ssh window size is set too small.  You should speed up single scp or rsync by adjusting the size.  http://serverfault.com/questions/434724/multi-thread-rsync-transfer

You might switch to hpn-ssh and you'll just have to run a single scp command.  http://www.psc.edu/index.php/hpn-ssh

Why isn't your rsync working?  Do you need to specify ssh?

rsync -av -e 'ssh'  path user@remote:path

If you nfs mount your remote filesystem, you could use cpio, which will be slightly faster for the initial copy than rsync or scp.  rsync can be used after you've copied everything and only need to copy the deltas.

To address your question directly:

Why not just run 10 scp with wildcards?  Why are you copying 1 file at a time with each SCP command?

The same goes for rsync.  You could run 10 separate rsync on different portions of your file system.

You could run something like this.

scp -pr /path1/ user@remote:/path1/; scp -pr /path21/ user@remote:/path21/ ; scp -pr /path31/ user@remote:/path31
scp -pr /path2/ user@remote:/path2/; scp -pr /path22/ user@remote:/path22/ ;  ...
scp -pr /path3/ user@remote:/path3/; scp -pr /path23/ user@remote:/path23/ ;  ...
scp -pr /path4/ user@remote:/path4/; scp -pr /path24/ user@remote:/path24/ ;  ...
scp -pr /path5/ user@remote:/path5/; scp -pr /path25/ user@remote:/path25/ ;  ...
scp -pr /path6/ user@remote:/path6/; scp -pr /path26/ user@remote:/path26/ ;  ...
scp -pr /path7/ user@remote:/path7/; scp -pr /path27/ user@remote:/path27/ ;  ...
scp -pr /path8/ user@remote:/path8/; scp -pr /path28/ user@remote:/path28/ ;  ...
scp -pr /path9/ user@remote:/path9/; scp -pr /path29/ user@remote:/path29/ ;  ...
scp -pr /path10/ user@remote:/path10/; scp -pr /path30/ user@remote:/path30/

Run each of the above lines in a separate terminal window.

If it's files your could change it

scp -pr [A-H]* user@remote:/path/
scp -pr [I-P]* user@remote:/path/
scp -pr [O-U]* user@remote:/path/
.
.
.
scp -pr [o-u]* user@remote:/path/
.
.
.
scp -pr [0-9]* user@remote:/path/;
0
 
gheistCommented:
You dont have to force rsync to use SSH, it is default for last 7 years or so.
0
 
it-rexAuthor Commented:
thank you
0

Featured Post

NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

  • 6
  • 5
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now