Solved

run multiple scp then wait

Posted on 2015-01-17
15
183 Views
Last Modified: 2015-02-08
we have a massive volume of files to scp,we need to run them in batches
like 10 scp in the background then wait till they finish then start 10 more etc..
what should we do?
0
Comment
Question by:it-rex
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
  • 3
  • +1
15 Comments
 
LVL 40

Expert Comment

by:omarfarid
ID: 40555466
how do you run the scp commands today?
0
 
LVL 11

Author Comment

by:it-rex
ID: 40555471
u mean it is old..
LOL
I understand.I tried rsync between our redhat 6.5 and exadata yesterday and it failed.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40555516
What I want to know, are you running the scp commands from a script?
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 11

Author Comment

by:it-rex
ID: 40555517
No just normal runs not scripted
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40555528
you can do the following in a script:

scp command1
scp command2
scp command3
scp command4
scp command5
scp command6
scp command7
scp command8
scp command9
scp command10
sleep 300
scp command10
scp command11
scp command12
scp command13
scp command14
scp command15
scp command16
scp command17
scp command18
scp command19
scp command20

The sleep 300  command will hold the script 5 min before resuming next 10 scp commands
0
 
LVL 11

Author Comment

by:it-rex
ID: 40555653
Say
10 scp running at once
I wanna wait
Till the first 1 finishes then kick the 11th
Or say 2 finishes after I wait I wanna kick the 11th and the 12th..
How to script that?
0
 
LVL 11

Author Comment

by:it-rex
ID: 40555654
So I have only 10 scp running at a time not more.
0
 
LVL 62

Accepted Solution

by:
gheist earned 200 total points
ID: 40557180
3 ideas, choose one:

1)
xargs -P `nproc` -I abc scp user@host:xxx ./xxx

2)
for x in a b c d ; do
 scp $x host:$x &
done
wait
3) rsync?
0
 
LVL 11

Author Comment

by:it-rex
ID: 40557205
Rsync does 1 file at a time...

The issues with wait is im not sure how should wait till the previous file gets copied..

What does the 1st one do?
0
 
LVL 40

Assisted Solution

by:omarfarid
omarfarid earned 200 total points
ID: 40557233
How do you run the scp commands? You may do similar to below:

for job in scp1 scp2 scp3 scp4 scp5 scp6 scp7
do
    count=`ls /tmp/lock* | wc -l`
    while true
    do
        if [ $count -lt 3 ]
        then
           break
        else
           sleep 10
        fi
     done
     touch /tmp/lock_$job
     $job
done

The jobs scp1 scp2 scp3 scp4 scp5 scp6 scp7 are scrips that do the scp commands
0
 
LVL 62

Expert Comment

by:gheist
ID: 40557235
First starts 1 scp per processor until list of files is exhausted. it is more like $ cat listoffiles | xargs -P 10 ....
Second starts heap of SCPs in background and waits until done (note the & before done and wait command after)
Rsync transfers 1 file at a time, but never transfers same data twice
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 40557238
One more point, each scp script should remove the lock file from /tmp dir to let other scripts run.
0
 
LVL 29

Assisted Solution

by:serialband
serialband earned 100 total points
ID: 40565474
The reason you might have speed ups by running separate simultaneous commands is that your ssh window size is set too small.  You should speed up single scp or rsync by adjusting the size.  http://serverfault.com/questions/434724/multi-thread-rsync-transfer

You might switch to hpn-ssh and you'll just have to run a single scp command.  http://www.psc.edu/index.php/hpn-ssh

Why isn't your rsync working?  Do you need to specify ssh?

rsync -av -e 'ssh'  path user@remote:path

If you nfs mount your remote filesystem, you could use cpio, which will be slightly faster for the initial copy than rsync or scp.  rsync can be used after you've copied everything and only need to copy the deltas.

To address your question directly:

Why not just run 10 scp with wildcards?  Why are you copying 1 file at a time with each SCP command?

The same goes for rsync.  You could run 10 separate rsync on different portions of your file system.

You could run something like this.

scp -pr /path1/ user@remote:/path1/; scp -pr /path21/ user@remote:/path21/ ; scp -pr /path31/ user@remote:/path31
scp -pr /path2/ user@remote:/path2/; scp -pr /path22/ user@remote:/path22/ ;  ...
scp -pr /path3/ user@remote:/path3/; scp -pr /path23/ user@remote:/path23/ ;  ...
scp -pr /path4/ user@remote:/path4/; scp -pr /path24/ user@remote:/path24/ ;  ...
scp -pr /path5/ user@remote:/path5/; scp -pr /path25/ user@remote:/path25/ ;  ...
scp -pr /path6/ user@remote:/path6/; scp -pr /path26/ user@remote:/path26/ ;  ...
scp -pr /path7/ user@remote:/path7/; scp -pr /path27/ user@remote:/path27/ ;  ...
scp -pr /path8/ user@remote:/path8/; scp -pr /path28/ user@remote:/path28/ ;  ...
scp -pr /path9/ user@remote:/path9/; scp -pr /path29/ user@remote:/path29/ ;  ...
scp -pr /path10/ user@remote:/path10/; scp -pr /path30/ user@remote:/path30/

Run each of the above lines in a separate terminal window.

If it's files your could change it

scp -pr [A-H]* user@remote:/path/
scp -pr [I-P]* user@remote:/path/
scp -pr [O-U]* user@remote:/path/
.
.
.
scp -pr [o-u]* user@remote:/path/
.
.
.
scp -pr [0-9]* user@remote:/path/;
0
 
LVL 62

Expert Comment

by:gheist
ID: 40565531
You dont have to force rsync to use SSH, it is default for last 7 years or so.
0
 
LVL 11

Author Closing Comment

by:it-rex
ID: 40597018
thank you
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Recover Lacie Edmini data. 11 77
exchange, squid, proxy, linux 6 89
VMware machine is not booting 6 129
How ldap located a Domain controller? 22 79
Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

710 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question