Help needed with bash shell script

Can anyone explain the bash script line by line.
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:ready/$2/*.zip ./$2 > slate.out 2>&1

ls -1 $2/* > curr.lst 2>/dev/null

while
read i
do
  if
    test -e ../done/"$i"
  then
      diff "$i" ../done/"$i" >/dev/null 2>&1
    if
      test $? -eq 0
    then
      rm "$i"
    fi
  fi
done < curr.lst

Open in new window

gaugetaAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Daniel HelgenbergerCommented:
First off, I do not think this works. But anyway,  I commented each relevant line with #. Imported note: The script needs to be called with parameters, but only uses the second parameter. What the first does is beyond me:

# this line opens scp with a fixed key. It takes the second argument/parameter ($2) of the script to copy (assumed zip) files to local ./$2 dir. The output is written to slate.out
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:ready/$2/*.zip ./$2 > slate.out 2>&1

# this line creates a text file named curr.lst, in there the contents of the dir $2 (argument), one entry per line 
ls -1 $2/* > curr.lst 2>/dev/null

while
read i
do
# this goes over every line of the previously created listing, putting the value of the line in $i. The file is acually fed into the loop by the very last line with a redirection/pipe
  if
# if the queried file is exists in ../done ..
    test -e ../done/"$i"
  then
# then compere them...
      diff "$i" ../done/"$i" >/dev/null 2>&1
    if
# if they are equal, (eg. the exit code of the previous command zero, queried by $? == 0),
      test $? -eq 0
    then
# then we remove the file in $2
      rm "$i"
    fi
  fi
done < curr.lst

Open in new window


As I said, without testing I have my doubts about the fist line, since you cannot use wildcards via ssh. The evaluation of the wildcards is done on localhost. Better use rsync for that.
Also, use md5sum or sth. to compare files, this is much too complicated and prone to errors (with diff).
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
gaugetaAuthor Commented:
@helge000 : Thanks for the reply.

I got almost all of what you explained.

I have one query in the scp command :

# this line opens scp with a fixed key. It takes the second argument/parameter ($2) of the script to copy the Zip files so slate.out
scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:ready/$2/*.zip ./$2 > slate.out 2>&1

Here is the zip file pulled from the server and placed in $2 directory right. And what is the purpose of slate.out here.
0
Daniel HelgenbergerCommented:
The slate.out is like a log file. It takes all output of the scp, redirects error and standard out into that file:
# redirects output into file  - but not only standard out but standard error out as well
> slate.out                     2>&1

Open in new window

Please keep my comment about the wildcard evail in mind. This script would only copy files which are in your current working dir on localhost and have a .zip extension and have the same name on remote, and would overwrite these. It is only seldomly wanted.
To further clarify: You would need a file called hello.zip in your local directory to get a file named hello.zip pulled from remote; the local file will then be overwritten.
I assume you rather want to copy all files with the extension *.zip on remote, then you would need to escape *:
scp -i ~/.ssh/id_rsa.sample 'gaara@stuid.student.com:ready/$2/\*.zip' ./$2 > slate.out 2>&1

Open in new window

If the difference is not obvious: 'user@server:/\*.zip'
0
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

gaugetaAuthor Commented:
@helge000 : Thanks for replying.

So do you mean to say only if I have a file named hello.zip, only then scp would pull the file from the remote server to the local server or else discard the file.

So if I don't  have a file named sample.zip in my local then the server would not pull the file named sample.zip even if it is present in the remote server.

Can you elaborate why is this happening.
0
Daniel HelgenbergerCommented:
Correct.

You use a wildcard, in your case *. But that wildcard needs to be evaluated. In unix, this eval will happen at the first possible moment - eg. after you press enter. In your case, this would happen in the current working dir, because it matches, and it will match every file with a zip extension. Image it like this:
scp user@server:./(ls *.zip) .

Open in new window

which will result in the local matches:
$ ls .
. .. hello.zip sample.zip
$ ssh user@server ls . 
. .. hello.zip xyz.zip
$ scp user@server:./hello.zip .
$ scp user@server:./sample.zip .
scp: sample.zip: No such file or directory

Open in new window

With the escape char, you effectively delay the expansion/evaluation one step, since the escape char is 'removed' after the command was locally and passing it on to the character, not the wildcard, to the remote host. The remote host in turn can expand/evaluate the selection, since it 'sees' the wildcard special character.

But you can see, all this is far from transparent. Do yourself a favor and use rsync for that, try:
rsync -a --include '*.zip' --exclude '*' -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:ready/$2/ ./$2

Open in new window

(Note: You need to explicitly exclude all stuff and then include your zips again)
0
simon3270Commented:
The * in the scp line is fine - it will be evaluated on the remote host.

This is because the expansion is of the entire token - in this case:
    gaara@stuid.student.com:ready/$2/*.zip

So unless you have a directory called gaara@stuid.student.com:ready in your current directory (with a subdirectory called the value of $2, and .zip files in that subdirectory), then no files on the loacl machine will match that pattern, so the * will be left unchanged.

You can test this yourself:

If you run
    echo *.zip
in a directory with ZIP files, it will echo the names of those ZIP files on standard out.

if you run
    echo not_there_at_all*.zip
(assuming you have no zip files starting "not_there_at_all), it will echo the string "not_there_at_all*.zip", including the asterisk.

So, the shell only expands * and ? if it can find a matching file.

As  @helge000 suggests, though, if the bare file name is passed as a parameter (without the gaara@stuid.student.com prefix), you *do* need to escape asterisks and question marks.
0
Daniel HelgenbergerCommented:
The * in the scp line is fine - it will be evaluated on the remote host.
But - this only the case if you have no such directory locally. In my experience handling of wildcards in scp/ssh only works as expected (and therefore always) when escaped:
http://docstore.mik.ua/orelly/networking_2ndEd/ssh/ch07_05.htm

IMHO the whole ssh/wildcard thing is prone to errors and should be avoided, esp. in scripts; since we still do not know if this script shall run on BSD or Linux (and therefore technically even do not know if SSH1 is used) - you should use rsync,
0
simon3270Commented:
The chances of having a directory called "gaara@stuid.student.com:ready" are pretty slim, but as you say, escaping is always a good idea.
0
gaugetaAuthor Commented:
@helge000 & @simon3270 : Thanks for the info.

I have one doubt.

scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:ready/$2/*.zip ./$2 > slate.out 2>&1

In the above line we are pulling from an assumed directory "ready"  shouldn't  the command be:

scp -i ~/.ssh/id_rsa.sample gaara@stuid.student.com:/ready/$2/*.zip ./$2 > slate.out 2>&1

Here shouldn't the ready be preceded by "/" how can it pull without having the fill path from root directory.

BTW this command is running on Linux.
0
simon3270Commented:
If you don't specify an absolute path (I.e. starting with a /) for a remote file, scp starts from the user's home directory (in this case, gaara's home directory on stuid.student.com).
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.