Solved

script to transfer file from one system to multiple system at once

Posted on 2011-03-22
17
664 Views
Last Modified: 2013-12-16
Hi Everyone,

I am trying to transfer some file & could be folder in future..

At the moment i am using scp
Host A to Host1
Host A to Host2
Host A to Host3
Host A to Host4
Host A to Host5
Host A to Host6
Host A to Host7
Host A to Host8
Host A to Host10
Host A to Host11
-->This process is time consuming..

At moment i am doing like this

HostA$scp file1 user@Host1:/userhome/user/
HostA$scp file1 user@Host2:/userhome/user/
HostA$scp file1 user@Host3:/userhome/user/
HostA$scp file1 user@Host4:/userhome/user/
HostA$scp file1 user@Host5:/userhome/user/
HostA$scp file1 user@Host6:/userhome/user/
HostA$scp file1 user@Host7:/userhome/user/
HostA$scp file1 user@Host8:/userhome/user/
HostA$scp file1 user@Host9:/userhome/user/
HostA$scp file1 user@Host10:/userhome/user/
HostA$scp file1 user@Host12:/userhome/user/

I would like to make this automated so in one go I can able to transfer file1 to all  Hosts (1-12)

Note:
I have a one useraccount with same username & password on all the Hosts (1-12)

Pls let me know if you need further information

Thanks
Javeed A

0
Comment
Question by:javeedabdul
  • 6
  • 3
  • 2
  • +4
17 Comments
 
LVL 16

Expert Comment

by:santoshmotwani
ID: 35195663
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 35195666
are they same file ??

why are u not using a script from each server host 1 to host 11 and they will put data from host A rather then A is sending data into all those server

you can use RSYNC script with cron tab..



you will need somethign like this

#!/bin/bash

DESTROOT=”/wheretocopy”
TODAY=`date ‘+%A’`

rsync -e ssh -avz –delete uer@HOSTA:/file/to-copy $DESTROOT/store-fle


if you use key base authentication,then you can do this without even typign username and password

0
 
LVL 6

Author Comment

by:javeedabdul
ID: 35195776
Sontosh,
The link is not complete.

fosiul01

I need a procedure to happen from Host A to Host (1-2) that's the requirement.
The file is not transfer every day it just once in a time.
so when ever a file need to be transfer i would like to run a script on Host A and that should authenticate and transfer file on all 12 host ( one by one ) or all at the same time

let me know if you need more information

thanks
Javeed
0
 
LVL 16

Expert Comment

by:santoshmotwani
ID: 35195909
#/usr/bin/sh
# This is a script to copy files from one host to a group of hosts

# There are three variables accepted via commandline
# $1 = first parameter (/source_path/source_filename)
# $2 = second parameter (/target_directory/)
# $3 = third paramter (file that contains list of hosts)

SOURCEFILE=$1
TARGETDIR=$2
HOSTFILE=$3

if [ -f $SOURCEFILE ]
then
   printf "File found, preparing to transfer\n"
   while read server
   do
      scp -p $SOURCEFILE ${server}:$TARGETDIR
   done < $HOSTFILE
else
   printf "File \"$SOURCEFILE\" not found\n"
   exit 0
fi
exit 0


ref : http://www.unix.com/shell-programming-scripting/99848-script-scp-file-multiple-servers.html
0
 
LVL 4

Expert Comment

by:m_walker
ID: 35195941
If you want to use scp, you will need to use key based authentication in order not to need to type in the password every time.

then have a script that as a line for each remote host
scp -o batchmode=yes $1 <user>@<host>:/<some>/<path>/

where $1 is the file name you pass in when you run the script
eg. save the script file as mycopy.sh
./mycopy.sh CopyThisFile.txt

0
 
LVL 76

Expert Comment

by:arnold
ID: 35196073
If all systems are in one location and you have NFS share exported from HOSTA to all others, you could have a script on all others to monitor the NFS share for file changes and after some validation/sanity checks etc. the script will propagate the changes it sees on the NFS share.
using puppet/SVN might also be a way to push the changes to all hosts.
A plain SCP can under certain unique conditions be unexpected depending on the files you are transferring i.e. the partition to which the file is being SCPed runs out of the space and the transferred file is runcated or zero length.
0
 
LVL 4

Expert Comment

by:m_walker
ID: 35196082
To add to arnold:

If you can nfs, then why not just setup a perment mount point from the other servers back to the main server, this way they all read from the one location over nfs from the master, no copy needed.  
0
 
LVL 6

Author Comment

by:javeedabdul
ID: 35197550
Arnold,
I am not an expert in linux,  plus I don't need a permanent setup.. I need temp setup to my work easier,

Suggest a simple script which does the job for me

thanks

0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 6

Author Comment

by:javeedabdul
ID: 35197559
santosh,

I will give a try to the script,
But it kind of doesn't achieve what i had hoped it would. I am looking for a way to do multiple transfers in parallel to save time. That is, from a source to multiple destinations at the same time. Is this possible ?

thanks
0
 
LVL 76

Accepted Solution

by:
arnold earned 500 total points
ID: 35197696
The simplest of things.

./script filename if you are within the directory or
./script /full/path/to/filename
#!/bin/sh
if [ -z "$1" ]; then
echo "Usage: $0 filename"
exit 1;
fi
if [ ! -e "$1"] ; then 
echo "The filename you provided: $1: does not exist"
exit 2
else 
scp $1 user@Host1:/userhome/user/
scp $1 user@Host2:/userhome/user/
scp $1 user@Host3:/userhome/user/
scp $1 user@Host4:/userhome/user/
scp $1 user@Host5:/userhome/user/
scp $1 user@Host6:/userhome/user/
scp $1 user@Host7:/userhome/user/
scp $1 user@Host8:/userhome/user/
scp $1 user@Host9:/userhome/user/
scp $1 user@Host10:/userhome/user/
scp $1 user@Host12:/userhome/user/
fi

Open in new window

0
 
LVL 8

Expert Comment

by:point_pleasant
ID: 35198820
might also suggest you put each scp command in backround and redirect stderr to an errorfile to see what if  anything fails.

scp $1 user@Host1:/userhome/user/  2>&1 >> /tmp/scp error &
0
 
LVL 76

Expert Comment

by:arnold
ID: 35198886
note that with this setup, you do not have any checks for errors or sanity checks nor dealing with issues such connection drop during transfer, or power failure.
IMHO if this is a configuration file, a script on each system should check whether the data is valid i.e. use cksum/md5sum to make sure the data sent matches the data received. Also have a check to make sure important parameters/data within the configuration do not suddenly go absent because of a typo. etc. Once validation is completed, the file can be "deployed" on the individual system.
0
 
LVL 8

Expert Comment

by:LunarNRG
ID: 35201215
parallel ssh, perhaps?

  http://code.google.com/p/parallel-ssh/
  http://www.linux.com/archive/feature/151340

On Ubuntu ...

$ sudo aptitude install pssh
$ cat hosts.txt
Host1 user
Host2 user
Host3 user
Host4 user
Host5 user
Host6 user
Host7 user
Host8 user
Host9 user
Host10 user
Host11 user
Host12 user
$ parallel-scp --hosts hosts.txt file1 /userhome/user/

Open in new window

0
 
LVL 6

Author Comment

by:javeedabdul
ID: 35213258
Thanks very much

I really need this procedure implemented.I will get back on this asap..

Have few other critical question which i am working on this weekend and will follow on this one. once i sought those out.

Thanks
Javeed A
0
 
LVL 6

Author Comment

by:javeedabdul
ID: 36017236
error downlaoding the pakage..


#sudo aptitude install pssh
The following NEW packages will be installed:
  pssh
The following partially installed packages will be configured:
  gdk-pixbuf-psd
0 packages upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 28.3 kB of archives. After unpacking 201 kB will be used.
Get:1 http://au.archive.ubuntu.com/ubuntu/ natty/universe pssh all 2.1.1-1 [28.3 kB]
Fetched 28.3 kB in 0s (77.2 kB/s)
Selecting previously deselected package pssh.
(Reading database ... 236459 files and directories currently installed.)
Unpacking pssh (from .../archives/pssh_2.1.1-1_all.deb) ...
Processing triggers for man-db ...
Setting up gdk-pixbuf-psd (20081217) ...
/var/lib/dpkg/info/gdk-pixbuf-psd.postinst: 33: cannot create /usr/lib/gtk-2.0/2.10.0/loader-files.d/gdk-pixbuf-psd.loaders: Directory nonexistent
/var/lib/dpkg/info/gdk-pixbuf-psd.postinst: 33: gdk-pixbuf-query-loaders: not found
dpkg: error processing gdk-pixbuf-psd (--configure):
 subprocess installed post-installation script returned error exit status 127
Setting up pssh (2.1.1-1) ...
Processing triggers for python-support ...
Errors were encountered while processing:
 gdk-pixbuf-psd
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install.  Trying to recover:
Setting up gdk-pixbuf-psd (20081217) ...
/var/lib/dpkg/info/gdk-pixbuf-psd.postinst: 33: cannot create /usr/lib/gtk-2.0/2.10.0/loader-files.d/gdk-pixbuf-psd.loaders: Directory nonexistent
/var/lib/dpkg/info/gdk-pixbuf-psd.postinst: 33: gdk-pixbuf-query-loaders: not found
dpkg: error processing gdk-pixbuf-psd (--configure):
 subprocess installed post-installation script returned error exit status 127
Errors were encountered while processing:
 gdk-pixbuf-psd
                                       
0
 
LVL 8

Expert Comment

by:LunarNRG
ID: 36018952
This is unrelated to pssh, you have a broken system. Try running ...
sudo aptitude -f install 

Open in new window

0
 
LVL 6

Author Closing Comment

by:javeedabdul
ID: 36164743
Good script but not what i like to acheive. will ask this question once i go through some scripting tutorials
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

The purpose of this article is to demonstrate how we can use conditional statements using Python.
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now