script to transfer file from one system to multiple system at once
Hi Everyone,
I am trying to transfer some file & could be folder in future..
At the moment i am using scp
Host A to Host1
Host A to Host2
Host A to Host3
Host A to Host4
Host A to Host5
Host A to Host6
Host A to Host7
Host A to Host8
Host A to Host10
Host A to Host11
-->This process is time consuming..
At moment i am doing like this
HostA$scp file1 user@Host1:/userhome/user/
HostA$scp file1 user@Host2:/userhome/user/
HostA$scp file1 user@Host3:/userhome/user/
HostA$scp file1 user@Host4:/userhome/user/
HostA$scp file1 user@Host5:/userhome/user/
HostA$scp file1 user@Host6:/userhome/user/
HostA$scp file1 user@Host7:/userhome/user/
HostA$scp file1 user@Host8:/userhome/user/
HostA$scp file1 user@Host9:/userhome/user/
HostA$scp file1 user@Host10:/userhome/user
HostA$scp file1 user@Host12:/userhome/user
I would like to make this automated so in one go I can able to transfer file1 to all Hosts (1-12)
Note:
I have a one useraccount with same username & password on all the Hosts (1-12)
Pls let me know if you need further information
Thanks
Javeed A
I am trying to transfer some file & could be folder in future..
At the moment i am using scp
Host A to Host1
Host A to Host2
Host A to Host3
Host A to Host4
Host A to Host5
Host A to Host6
Host A to Host7
Host A to Host8
Host A to Host10
Host A to Host11
-->This process is time consuming..
At moment i am doing like this
HostA$scp file1 user@Host1:/userhome/user/
HostA$scp file1 user@Host2:/userhome/user/
HostA$scp file1 user@Host3:/userhome/user/
HostA$scp file1 user@Host4:/userhome/user/
HostA$scp file1 user@Host5:/userhome/user/
HostA$scp file1 user@Host6:/userhome/user/
HostA$scp file1 user@Host7:/userhome/user/
HostA$scp file1 user@Host8:/userhome/user/
HostA$scp file1 user@Host9:/userhome/user/
HostA$scp file1 user@Host10:/userhome/user
HostA$scp file1 user@Host12:/userhome/user
I would like to make this automated so in one go I can able to transfer file1 to all Hosts (1-12)
Note:
I have a one useraccount with same username & password on all the Hosts (1-12)
Pls let me know if you need further information
Thanks
Javeed A
are they same file ??
why are u not using a script from each server host 1 to host 11 and they will put data from host A rather then A is sending data into all those server
you can use RSYNC script with cron tab..
you will need somethign like this
#!/bin/bash
DESTROOT=”/wheretocopy”
TODAY=`date ‘+%A’`
rsync -e ssh -avz –delete uer@HOSTA:/file/to-copy $DESTROOT/store-fle
if you use key base authentication,then you can do this without even typign username and password
why are u not using a script from each server host 1 to host 11 and they will put data from host A rather then A is sending data into all those server
you can use RSYNC script with cron tab..
you will need somethign like this
#!/bin/bash
DESTROOT=”/wheretocopy”
TODAY=`date ‘+%A’`
rsync -e ssh -avz –delete uer@HOSTA:/file/to-copy $DESTROOT/store-fle
if you use key base authentication,then you can do this without even typign username and password
ASKER
Sontosh,
The link is not complete.
fosiul01
I need a procedure to happen from Host A to Host (1-2) that's the requirement.
The file is not transfer every day it just once in a time.
so when ever a file need to be transfer i would like to run a script on Host A and that should authenticate and transfer file on all 12 host ( one by one ) or all at the same time
let me know if you need more information
thanks
Javeed
The link is not complete.
fosiul01
I need a procedure to happen from Host A to Host (1-2) that's the requirement.
The file is not transfer every day it just once in a time.
so when ever a file need to be transfer i would like to run a script on Host A and that should authenticate and transfer file on all 12 host ( one by one ) or all at the same time
let me know if you need more information
thanks
Javeed
#/usr/bin/sh
# This is a script to copy files from one host to a group of hosts
# There are three variables accepted via commandline
# $1 = first parameter (/source_path/source_filen
# $2 = second parameter (/target_directory/)
# $3 = third paramter (file that contains list of hosts)
SOURCEFILE=$1
TARGETDIR=$2
HOSTFILE=$3
if [ -f $SOURCEFILE ]
then
printf "File found, preparing to transfer\n"
while read server
do
scp -p $SOURCEFILE ${server}:$TARGETDIR
done < $HOSTFILE
else
printf "File \"$SOURCEFILE\" not found\n"
exit 0
fi
exit 0
ref : http://www.unix.com/shell-programming-scripting/99848-script-scp-file-multiple-servers.html
# This is a script to copy files from one host to a group of hosts
# There are three variables accepted via commandline
# $1 = first parameter (/source_path/source_filen
# $2 = second parameter (/target_directory/)
# $3 = third paramter (file that contains list of hosts)
SOURCEFILE=$1
TARGETDIR=$2
HOSTFILE=$3
if [ -f $SOURCEFILE ]
then
printf "File found, preparing to transfer\n"
while read server
do
scp -p $SOURCEFILE ${server}:$TARGETDIR
done < $HOSTFILE
else
printf "File \"$SOURCEFILE\" not found\n"
exit 0
fi
exit 0
ref : http://www.unix.com/shell-programming-scripting/99848-script-scp-file-multiple-servers.html
If you want to use scp, you will need to use key based authentication in order not to need to type in the password every time.
then have a script that as a line for each remote host
scp -o batchmode=yes $1 <user>@<host>:/<some>/<pat
where $1 is the file name you pass in when you run the script
eg. save the script file as mycopy.sh
./mycopy.sh CopyThisFile.txt
then have a script that as a line for each remote host
scp -o batchmode=yes $1 <user>@<host>:/<some>/<pat
where $1 is the file name you pass in when you run the script
eg. save the script file as mycopy.sh
./mycopy.sh CopyThisFile.txt
If all systems are in one location and you have NFS share exported from HOSTA to all others, you could have a script on all others to monitor the NFS share for file changes and after some validation/sanity checks etc. the script will propagate the changes it sees on the NFS share.
using puppet/SVN might also be a way to push the changes to all hosts.
A plain SCP can under certain unique conditions be unexpected depending on the files you are transferring i.e. the partition to which the file is being SCPed runs out of the space and the transferred file is runcated or zero length.
using puppet/SVN might also be a way to push the changes to all hosts.
A plain SCP can under certain unique conditions be unexpected depending on the files you are transferring i.e. the partition to which the file is being SCPed runs out of the space and the transferred file is runcated or zero length.
To add to arnold:
If you can nfs, then why not just setup a perment mount point from the other servers back to the main server, this way they all read from the one location over nfs from the master, no copy needed.
If you can nfs, then why not just setup a perment mount point from the other servers back to the main server, this way they all read from the one location over nfs from the master, no copy needed.
ASKER
Arnold,
I am not an expert in linux, plus I don't need a permanent setup.. I need temp setup to my work easier,
Suggest a simple script which does the job for me
thanks
I am not an expert in linux, plus I don't need a permanent setup.. I need temp setup to my work easier,
Suggest a simple script which does the job for me
thanks
ASKER
santosh,
I will give a try to the script,
But it kind of doesn't achieve what i had hoped it would. I am looking for a way to do multiple transfers in parallel to save time. That is, from a source to multiple destinations at the same time. Is this possible ?
thanks
I will give a try to the script,
But it kind of doesn't achieve what i had hoped it would. I am looking for a way to do multiple transfers in parallel to save time. That is, from a source to multiple destinations at the same time. Is this possible ?
thanks
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
might also suggest you put each scp command in backround and redirect stderr to an errorfile to see what if anything fails.
scp $1 user@Host1:/userhome/user/
scp $1 user@Host1:/userhome/user/
note that with this setup, you do not have any checks for errors or sanity checks nor dealing with issues such connection drop during transfer, or power failure.
IMHO if this is a configuration file, a script on each system should check whether the data is valid i.e. use cksum/md5sum to make sure the data sent matches the data received. Also have a check to make sure important parameters/data within the configuration do not suddenly go absent because of a typo. etc. Once validation is completed, the file can be "deployed" on the individual system.
IMHO if this is a configuration file, a script on each system should check whether the data is valid i.e. use cksum/md5sum to make sure the data sent matches the data received. Also have a check to make sure important parameters/data within the configuration do not suddenly go absent because of a typo. etc. Once validation is completed, the file can be "deployed" on the individual system.
parallel ssh, perhaps?
http://code.google.com/p/parallel-ssh/
http://www.linux.com/archive/feature/151340
On Ubuntu ...
http://code.google.com/p/parallel-ssh/
http://www.linux.com/archive/feature/151340
On Ubuntu ...
$ sudo aptitude install pssh
$ cat hosts.txt
Host1 user
Host2 user
Host3 user
Host4 user
Host5 user
Host6 user
Host7 user
Host8 user
Host9 user
Host10 user
Host11 user
Host12 user
$ parallel-scp --hosts hosts.txt file1 /userhome/user/ASKER
Thanks very much
I really need this procedure implemented.I will get back on this asap..
Have few other critical question which i am working on this weekend and will follow on this one. once i sought those out.
Thanks
Javeed A
I really need this procedure implemented.I will get back on this asap..
Have few other critical question which i am working on this weekend and will follow on this one. once i sought those out.
Thanks
Javeed A
ASKER
error downlaoding the pakage..
#sudo aptitude install pssh
The following NEW packages will be installed:
pssh
The following partially installed packages will be configured:
gdk-pixbuf-psd
0 packages upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 28.3 kB of archives. After unpacking 201 kB will be used.
Get:1 http://au.archive.ubuntu.com/ubuntu/ natty/universe pssh all 2.1.1-1 [28.3 kB]
Fetched 28.3 kB in 0s (77.2 kB/s)
Selecting previously deselected package pssh.
(Reading database ... 236459 files and directories currently installed.)
Unpacking pssh (from .../archives/pssh_2.1.1-1_
Processing triggers for man-db ...
Setting up gdk-pixbuf-psd (20081217) ...
/var/lib/dpkg/info/gdk-pix
/var/lib/dpkg/info/gdk-pix
dpkg: error processing gdk-pixbuf-psd (--configure):
subprocess installed post-installation script returned error exit status 127
Setting up pssh (2.1.1-1) ...
Processing triggers for python-support ...
Errors were encountered while processing:
gdk-pixbuf-psd
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up gdk-pixbuf-psd (20081217) ...
/var/lib/dpkg/info/gdk-pix
/var/lib/dpkg/info/gdk-pix
dpkg: error processing gdk-pixbuf-psd (--configure):
subprocess installed post-installation script returned error exit status 127
Errors were encountered while processing:
gdk-pixbuf-psd
#sudo aptitude install pssh
The following NEW packages will be installed:
pssh
The following partially installed packages will be configured:
gdk-pixbuf-psd
0 packages upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 28.3 kB of archives. After unpacking 201 kB will be used.
Get:1 http://au.archive.ubuntu.com/ubuntu/ natty/universe pssh all 2.1.1-1 [28.3 kB]
Fetched 28.3 kB in 0s (77.2 kB/s)
Selecting previously deselected package pssh.
(Reading database ... 236459 files and directories currently installed.)
Unpacking pssh (from .../archives/pssh_2.1.1-1_
Processing triggers for man-db ...
Setting up gdk-pixbuf-psd (20081217) ...
/var/lib/dpkg/info/gdk-pix
/var/lib/dpkg/info/gdk-pix
dpkg: error processing gdk-pixbuf-psd (--configure):
subprocess installed post-installation script returned error exit status 127
Setting up pssh (2.1.1-1) ...
Processing triggers for python-support ...
Errors were encountered while processing:
gdk-pixbuf-psd
E: Sub-process /usr/bin/dpkg returned an error code (1)
A package failed to install. Trying to recover:
Setting up gdk-pixbuf-psd (20081217) ...
/var/lib/dpkg/info/gdk-pix
/var/lib/dpkg/info/gdk-pix
dpkg: error processing gdk-pixbuf-psd (--configure):
subprocess installed post-installation script returned error exit status 127
Errors were encountered while processing:
gdk-pixbuf-psd
This is unrelated to pssh, you have a broken system. Try running ...
sudo aptitude -f install ASKER
Good script but not what i like to acheive. will ask this question once i go through some scripting tutorials
http://www.unix.com/shell-programming-scripting/99848-script-scp-file-multiple-servers.html