We help IT Professionals succeed at work.

Shell script for comparing and moving files from remote server

Medium Priority
868 Views
Last Modified: 2013-12-27
Dear Experts,

I need a script, which can do the following.

1. it will copy a file named A (file A contains filenames like 1.txt.gz, 2.txt.gz and so on in separate lines) to a directory /test

2. It will search for a file named B in /test directory

3. If no file B exists, it will read all the lines of file A without .gz (for example if file names are 1.txt.gz then the input will be like 1.txt) (as I previously mentioned file A contains 1 filename in each lines) and transfer those files from remote server by ssh or ftp (scp is preferable).

4. After transferring it will rename the file A to file B in /test directory.

5. upon next execution if file B exists in the /test directory, then also it will copy file A but now we need to compare it with file B. If some uniq entries are found between file A and file B, it will copy those files from remote server(solaris 10) and rename file A to file B. if file A is same as file B the script simply exit.

6. All activity needs to be logged.

Please help me.

Thanks and regards,
Mohona.
Comment
Watch Question

That is possible but will take quite a while to write.
Have you looked at rsync ?   It does most of that if you are looking at backing up changed files.

If you really do want this written as script then explain one thing....
if it finds
1.txt.gz
in file A,

WHich one of these should it do :
1. ignore that line
2. assume the file exists called 1.txt and copy that instead
3. unzip the gz file to get 1.txt and copy that
4. find the 1.txt file, zip it and copy that
5. Something else ?

Author

Commented:
file A contains only some file names which are not successfully got transferred as a result of execution of another script (Which does gzip and transfer those files to remote server). For example file A looks like following:-

1.txt.gz
2.txt.gz
3.txt.gz

So I need to transfer the files from remote server.  Actual files in the remote servers are without the extension .gz.  

We need to assume the file exists called 1.txt and copy that instead from remote server.

Before copying through scp or ftp, it will be really helpful if you can zip it into the remote server in some tmp directory and transfer the same in local server.

Please tell me if more clarifications needed.

CERTIFIED EXPERT
Top Expert 2007

Commented:

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | while read file
      do
          scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' | while read file
      do
         scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi
CERTIFIED EXPERT
Top Expert 2007

Commented:
Corrections:

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }' | while read file
      do
          scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' /test/A | while read file
      do
         scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi
fridomCEO/Programmer
CERTIFIED EXPERT

Commented:
scp is a good bet, but also curl, wget or unison.

Regards
Friedrich

Author

Commented:
diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }'

Above command yields one blank line, please tell me how to avoid, output as follows

#Blank line goes here
1
2
3
........

Please help.

Thanks and Regards,
Mohona.
CERTIFIED EXPERT
Top Expert 2007
Commented:
try

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }' |  sed '/^$/ d' | while read file
      do
          scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' /test/A |  sed '/^$/ d' | while read file
      do
         scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts

Author

Commented:
Ok I got the answer, please ignore the last question

Author

Commented:
Thank you Expert, Last question, please solve this also. When I am executing some remote ssh commands from the same script, the script is not looping ( I mean to say it is only working for the first file rest fo the files are getting ignored. Please check the follwing:-

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }' |  sed '/^$/ d' | while read file
      do
           ssh username@remotehost cp -p /path/to//$file /tempdir of remote server/stage
           ssh username@remotehost  gzip /tempdir of remote server/$file
           scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' /test/A |  sed '/^$/ d' | while read file
      do
           ssh username@remotehost cp -p /path/to//$file /tempdir of remote server/stage
           ssh username@remotehost  gzip /tempdir of remote server/$file
           scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi

Author

Commented:
I got the answer, -n with ssh solves the same.

Thanks and Regards,
Mohona.
CERTIFIED EXPERT
Top Expert 2007

Commented:
I think you need to replace

ssh username@remotehost cp -p /path/to//$file /tempdir of remote server/stage

with

ssh username@remotehost "cp -p /path/to/$file /tempdir_of_remote_server/stage" # spaces are going to cause problem

Same with the next line (gzip line)

Author

Commented:
Excellent, Thanks
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.