Solved

run multiple scp then wait

Posted on 2015-01-17
15
159 Views
Last Modified: 2015-02-08
we have a massive volume of files to scp,we need to run them in batches
like 10 scp in the background then wait till they finish then start 10 more etc..
what should we do?
0
Comment
Question by:it-rex
  • 6
  • 5
  • 3
  • +1
15 Comments
 
LVL 40

Expert Comment

by:omarfarid
Comment Utility
how do you run the scp commands today?
0
 
LVL 11

Author Comment

by:it-rex
Comment Utility
u mean it is old..
LOL
I understand.I tried rsync between our redhat 6.5 and exadata yesterday and it failed.
0
 
LVL 40

Expert Comment

by:omarfarid
Comment Utility
What I want to know, are you running the scp commands from a script?
0
 
LVL 11

Author Comment

by:it-rex
Comment Utility
No just normal runs not scripted
0
 
LVL 40

Expert Comment

by:omarfarid
Comment Utility
you can do the following in a script:

scp command1
scp command2
scp command3
scp command4
scp command5
scp command6
scp command7
scp command8
scp command9
scp command10
sleep 300
scp command10
scp command11
scp command12
scp command13
scp command14
scp command15
scp command16
scp command17
scp command18
scp command19
scp command20

The sleep 300  command will hold the script 5 min before resuming next 10 scp commands
0
 
LVL 11

Author Comment

by:it-rex
Comment Utility
Say
10 scp running at once
I wanna wait
Till the first 1 finishes then kick the 11th
Or say 2 finishes after I wait I wanna kick the 11th and the 12th..
How to script that?
0
 
LVL 11

Author Comment

by:it-rex
Comment Utility
So I have only 10 scp running at a time not more.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 61

Accepted Solution

by:
gheist earned 200 total points
Comment Utility
3 ideas, choose one:

1)
xargs -P `nproc` -I abc scp user@host:xxx ./xxx

2)
for x in a b c d ; do
 scp $x host:$x &
done
wait
3) rsync?
0
 
LVL 11

Author Comment

by:it-rex
Comment Utility
Rsync does 1 file at a time...

The issues with wait is im not sure how should wait till the previous file gets copied..

What does the 1st one do?
0
 
LVL 40

Assisted Solution

by:omarfarid
omarfarid earned 200 total points
Comment Utility
How do you run the scp commands? You may do similar to below:

for job in scp1 scp2 scp3 scp4 scp5 scp6 scp7
do
    count=`ls /tmp/lock* | wc -l`
    while true
    do
        if [ $count -lt 3 ]
        then
           break
        else
           sleep 10
        fi
     done
     touch /tmp/lock_$job
     $job
done

The jobs scp1 scp2 scp3 scp4 scp5 scp6 scp7 are scrips that do the scp commands
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
First starts 1 scp per processor until list of files is exhausted. it is more like $ cat listoffiles | xargs -P 10 ....
Second starts heap of SCPs in background and waits until done (note the & before done and wait command after)
Rsync transfers 1 file at a time, but never transfers same data twice
0
 
LVL 40

Expert Comment

by:omarfarid
Comment Utility
One more point, each scp script should remove the lock file from /tmp dir to let other scripts run.
0
 
LVL 27

Assisted Solution

by:serialband
serialband earned 100 total points
Comment Utility
The reason you might have speed ups by running separate simultaneous commands is that your ssh window size is set too small.  You should speed up single scp or rsync by adjusting the size.  http://serverfault.com/questions/434724/multi-thread-rsync-transfer

You might switch to hpn-ssh and you'll just have to run a single scp command.  http://www.psc.edu/index.php/hpn-ssh

Why isn't your rsync working?  Do you need to specify ssh?

rsync -av -e 'ssh'  path user@remote:path

If you nfs mount your remote filesystem, you could use cpio, which will be slightly faster for the initial copy than rsync or scp.  rsync can be used after you've copied everything and only need to copy the deltas.

To address your question directly:

Why not just run 10 scp with wildcards?  Why are you copying 1 file at a time with each SCP command?

The same goes for rsync.  You could run 10 separate rsync on different portions of your file system.

You could run something like this.

scp -pr /path1/ user@remote:/path1/; scp -pr /path21/ user@remote:/path21/ ; scp -pr /path31/ user@remote:/path31
scp -pr /path2/ user@remote:/path2/; scp -pr /path22/ user@remote:/path22/ ;  ...
scp -pr /path3/ user@remote:/path3/; scp -pr /path23/ user@remote:/path23/ ;  ...
scp -pr /path4/ user@remote:/path4/; scp -pr /path24/ user@remote:/path24/ ;  ...
scp -pr /path5/ user@remote:/path5/; scp -pr /path25/ user@remote:/path25/ ;  ...
scp -pr /path6/ user@remote:/path6/; scp -pr /path26/ user@remote:/path26/ ;  ...
scp -pr /path7/ user@remote:/path7/; scp -pr /path27/ user@remote:/path27/ ;  ...
scp -pr /path8/ user@remote:/path8/; scp -pr /path28/ user@remote:/path28/ ;  ...
scp -pr /path9/ user@remote:/path9/; scp -pr /path29/ user@remote:/path29/ ;  ...
scp -pr /path10/ user@remote:/path10/; scp -pr /path30/ user@remote:/path30/

Run each of the above lines in a separate terminal window.

If it's files your could change it

scp -pr [A-H]* user@remote:/path/
scp -pr [I-P]* user@remote:/path/
scp -pr [O-U]* user@remote:/path/
.
.
.
scp -pr [o-u]* user@remote:/path/
.
.
.
scp -pr [0-9]* user@remote:/path/;
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
You dont have to force rsync to use SSH, it is default for last 7 years or so.
0
 
LVL 11

Author Closing Comment

by:it-rex
Comment Utility
thank you
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now