?
Solved

Shell script for comparing and moving files from remote server

Posted on 2009-02-17
12
Medium Priority
?
849 Views
Last Modified: 2013-12-27
Dear Experts,

I need a script, which can do the following.

1. it will copy a file named A (file A contains filenames like 1.txt.gz, 2.txt.gz and so on in separate lines) to a directory /test

2. It will search for a file named B in /test directory

3. If no file B exists, it will read all the lines of file A without .gz (for example if file names are 1.txt.gz then the input will be like 1.txt) (as I previously mentioned file A contains 1 filename in each lines) and transfer those files from remote server by ssh or ftp (scp is preferable).

4. After transferring it will rename the file A to file B in /test directory.

5. upon next execution if file B exists in the /test directory, then also it will copy file A but now we need to compare it with file B. If some uniq entries are found between file A and file B, it will copy those files from remote server(solaris 10) and rename file A to file B. if file A is same as file B the script simply exit.

6. All activity needs to be logged.

Please help me.

Thanks and regards,
Mohona.
0
Comment
Question by:mohona
12 Comments
 
LVL 20

Expert Comment

by:edster9999
ID: 23660854
That is possible but will take quite a while to write.
Have you looked at rsync ?   It does most of that if you are looking at backing up changed files.

If you really do want this written as script then explain one thing....
if it finds
1.txt.gz
in file A,

WHich one of these should it do :
1. ignore that line
2. assume the file exists called 1.txt and copy that instead
3. unzip the gz file to get 1.txt and copy that
4. find the 1.txt file, zip it and copy that
5. Something else ?
0
 

Author Comment

by:mohona
ID: 23661309
file A contains only some file names which are not successfully got transferred as a result of execution of another script (Which does gzip and transfer those files to remote server). For example file A looks like following:-

1.txt.gz
2.txt.gz
3.txt.gz

So I need to transfer the files from remote server.  Actual files in the remote servers are without the extension .gz.  

We need to assume the file exists called 1.txt and copy that instead from remote server.

Before copying through scp or ftp, it will be really helpful if you can zip it into the remote server in some tmp directory and transfer the same in local server.

Please tell me if more clarifications needed.

0
 
LVL 40

Expert Comment

by:omarfarid
ID: 23661586

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | while read file
      do
          scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' | while read file
      do
         scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
LVL 40

Expert Comment

by:omarfarid
ID: 23661637
Corrections:

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }' | while read file
      do
          scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' /test/A | while read file
      do
         scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi
0
 
LVL 24

Expert Comment

by:fridom
ID: 23667471
scp is a good bet, but also curl, wget or unison.

Regards
Friedrich
0
 

Author Comment

by:mohona
ID: 23668380
diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }'

Above command yields one blank line, please tell me how to avoid, output as follows

#Blank line goes here
1
2
3
........

Please help.

Thanks and Regards,
Mohona.
0
 
LVL 40

Accepted Solution

by:
omarfarid earned 2000 total points
ID: 23668447
try

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }' |  sed '/^$/ d' | while read file
      do
          scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' /test/A |  sed '/^$/ d' | while read file
      do
         scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi
0
 

Author Comment

by:mohona
ID: 23668476
Ok I got the answer, please ignore the last question
0
 

Author Comment

by:mohona
ID: 23668541
Thank you Expert, Last question, please solve this also. When I am executing some remote ssh commands from the same script, the script is not looping ( I mean to say it is only working for the first file rest fo the files are getting ignored. Please check the follwing:-

cd /path/to/mydir
cp /path/to/A /test
if [ -f /test/B ]
then
      diff /test/A /test/B | awk '{ print $2 }' | awk -F'.' '{ print $1 }' |  sed '/^$/ d' | while read file
      do
           ssh username@remotehost cp -p /path/to//$file /tempdir of remote server/stage
           ssh username@remotehost  gzip /tempdir of remote server/$file
           scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
else
      awk -F'.' '{ print $1 }' /test/A |  sed '/^$/ d' | while read file
      do
           ssh username@remotehost cp -p /path/to//$file /tempdir of remote server/stage
           ssh username@remotehost  gzip /tempdir of remote server/$file
           scp username@remotehost:/path/to/$file $file
      done
      mv /test/A /test/B
fi
0
 

Author Comment

by:mohona
ID: 23668737
I got the answer, -n with ssh solves the same.

Thanks and Regards,
Mohona.
0
 
LVL 40

Expert Comment

by:omarfarid
ID: 23668750
I think you need to replace

ssh username@remotehost cp -p /path/to//$file /tempdir of remote server/stage

with

ssh username@remotehost "cp -p /path/to/$file /tempdir_of_remote_server/stage" # spaces are going to cause problem

Same with the next line (gzip line)
0
 

Author Closing Comment

by:mohona
ID: 31547831
Excellent, Thanks
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
Suggested Courses

829 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question