GAUTAM
asked on
Help needed with bash shell script
Can anyone explain the bash script line by line.
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:ready/$2/*.zip ./$2 > slate.out 2>&1
ls -1 $2/* > curr.lst 2>/dev/null
while
read i
do
if
test -e ../done/"$i"
then
diff "$i" ../done/"$i" >/dev/null 2>&1
if
test $? -eq 0
then
rm "$i"
fi
fi
done < curr.lst
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
The slate.out is like a log file. It takes all output of the scp, redirects error and standard out into that file:
To further clarify: You would need a file called hello.zip in your local directory to get a file named hello.zip pulled from remote; the local file will then be overwritten.
I assume you rather want to copy all files with the extension *.zip on remote, then you would need to escape *:
# redirects output into file - but not only standard out but standard error out as well
> slate.out 2>&1
Please keep my comment about the wildcard evail in mind. This script would only copy files which are in your current working dir on localhost and have a .zip extension and have the same name on remote, and would overwrite these. It is only seldomly wanted.To further clarify: You would need a file called hello.zip in your local directory to get a file named hello.zip pulled from remote; the local file will then be overwritten.
I assume you rather want to copy all files with the extension *.zip on remote, then you would need to escape *:
scp -i ~/.ssh/id_rsa.sample 'gaara@stuid.student.com:ready/$2/\*.zip' ./$2 > slate.out 2>&1
If the difference is not obvious: 'user@server:/\*.zip'
ASKER
@helge000 : Thanks for replying.
So do you mean to say only if I have a file named hello.zip, only then scp would pull the file from the remote server to the local server or else discard the file.
So if I don't have a file named sample.zip in my local then the server would not pull the file named sample.zip even if it is present in the remote server.
Can you elaborate why is this happening.
So do you mean to say only if I have a file named hello.zip, only then scp would pull the file from the remote server to the local server or else discard the file.
So if I don't have a file named sample.zip in my local then the server would not pull the file named sample.zip even if it is present in the remote server.
Can you elaborate why is this happening.
Correct.
You use a wildcard, in your case *. But that wildcard needs to be evaluated. In unix, this eval will happen at the first possible moment - eg. after you press enter. In your case, this would happen in the current working dir, because it matches, and it will match every file with a zip extension. Image it like this:
But you can see, all this is far from transparent. Do yourself a favor and use rsync for that, try:
You use a wildcard, in your case *. But that wildcard needs to be evaluated. In unix, this eval will happen at the first possible moment - eg. after you press enter. In your case, this would happen in the current working dir, because it matches, and it will match every file with a zip extension. Image it like this:
scp user@server:./(ls *.zip) .
which will result in the local matches:$ ls .
. .. hello.zip sample.zip
$ ssh user@server ls .
. .. hello.zip xyz.zip
$ scp user@server:./hello.zip .
$ scp user@server:./sample.zip .
scp: sample.zip: No such file or directory
With the escape char, you effectively delay the expansion/evaluation one step, since the escape char is 'removed' after the command was locally and passing it on to the character, not the wildcard, to the remote host. The remote host in turn can expand/evaluate the selection, since it 'sees' the wildcard special character.But you can see, all this is far from transparent. Do yourself a favor and use rsync for that, try:
rsync -a --include '*.zip' --exclude '*' -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:ready/$2/ ./$2
(Note: You need to explicitly exclude all stuff and then include your zips again)
The * in the scp line is fine - it will be evaluated on the remote host.
This is because the expansion is of the entire token - in this case:
gaara@stuid.student.com:re ady/$2/*.z ip
So unless you have a directory called gaara@stuid.student.com:re ady in your current directory (with a subdirectory called the value of $2, and .zip files in that subdirectory), then no files on the loacl machine will match that pattern, so the * will be left unchanged.
You can test this yourself:
If you run
echo *.zip
in a directory with ZIP files, it will echo the names of those ZIP files on standard out.
if you run
echo not_there_at_all*.zip
(assuming you have no zip files starting "not_there_at_all), it will echo the string "not_there_at_all*.zip", including the asterisk.
So, the shell only expands * and ? if it can find a matching file.
As @helge000 suggests, though, if the bare file name is passed as a parameter (without the gaara@stuid.student.com prefix), you *do* need to escape asterisks and question marks.
This is because the expansion is of the entire token - in this case:
gaara@stuid.student.com:re
So unless you have a directory called gaara@stuid.student.com:re
You can test this yourself:
If you run
echo *.zip
in a directory with ZIP files, it will echo the names of those ZIP files on standard out.
if you run
echo not_there_at_all*.zip
(assuming you have no zip files starting "not_there_at_all), it will echo the string "not_there_at_all*.zip", including the asterisk.
So, the shell only expands * and ? if it can find a matching file.
As @helge000 suggests, though, if the bare file name is passed as a parameter (without the gaara@stuid.student.com prefix), you *do* need to escape asterisks and question marks.
The * in the scp line is fine - it will be evaluated on the remote host.But - this only the case if you have no such directory locally. In my experience handling of wildcards in scp/ssh only works as expected (and therefore always) when escaped:
http://docstore.mik.ua/orelly/networking_2ndEd/ssh/ch07_05.htm
IMHO the whole ssh/wildcard thing is prone to errors and should be avoided, esp. in scripts; since we still do not know if this script shall run on BSD or Linux (and therefore technically even do not know if SSH1 is used) - you should use rsync,
The chances of having a directory called "gaara@stuid.student.com:r eady" are pretty slim, but as you say, escaping is always a good idea.
ASKER
@helge000 & @simon3270 : Thanks for the info.
I have one doubt.
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:re ady/$2/*.z ip ./$2 > slate.out 2>&1
In the above line we are pulling from an assumed directory "ready" shouldn't the command be:
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:/r eady/$2/*. zip ./$2 > slate.out 2>&1
Here shouldn't the ready be preceded by "/" how can it pull without having the fill path from root directory.
BTW this command is running on Linux.
I have one doubt.
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:re
In the above line we are pulling from an assumed directory "ready" shouldn't the command be:
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:/r
Here shouldn't the ready be preceded by "/" how can it pull without having the fill path from root directory.
BTW this command is running on Linux.
If you don't specify an absolute path (I.e. starting with a /) for a remote file, scp starts from the user's home directory (in this case, gaara's home directory on stuid.student.com).
ASKER
I got almost all of what you explained.
I have one query in the scp command :
# this line opens scp with a fixed key. It takes the second argument/parameter ($2) of the script to copy the Zip files so slate.out
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:re
Here is the zip file pulled from the server and placed in $2 directory right. And what is the purpose of slate.out here.