[Webinar] Streamline your web hosting managementRegister Today


script to copy, compress and transfer archive logfiles to another server in Linux

Posted on 2011-05-12
Medium Priority
Last Modified: 2012-05-11
I have a requirement to compress the Oracle database archive log files having the format 'arch_%t_%s.arc'  and transfer it to another server, both running on Linux. I am planning to do it like this -

1. copy the archive logfiles to another location in the same server using rsync.
2. compress the logfiles from that location. A script will run every 1 hour and compress all the uncompressed files using 'tar'.
3. transfer the compressed logfiles using SCP to the another server. (for every 1 hour).

But the big question at the 2nd & 3rd step how it'll identify the files which have already been compressed(and copied) and which are not. Please let me know if there is any other way too to achieve this.
Question by:makk2010
LVL 19

Expert Comment

ID: 35745395
I would suggest rsync to a temporary location - a location in which all the work is performed & only once a file is copied, is it moved to your intended destination.

Picking up the different arc files is simple enough with a 'for x in *.arc ; do tar .... "$x" ; done' style for loop.

Another for loop can then iterate through the tar files and scp them across.  The bash $? special variable will inform you of the exit code from the previous command.  You can therefore test whether the scp was successful [ $? -eq 0 ] and, if successful, move the tar file to your end folder on the Oracle machine.
LVL 31

Accepted Solution

farzanj earned 2000 total points
ID: 35745713
Why don't you simply rsync it from one server to another, with compression?

rsync -avz /path/source/ user@remoteServer:/target/path

z- compresses the network delivery
a- keeps all permissions and ownership.
LVL 12

Expert Comment

ID: 35747280
farzanj beat me to what I would suggest.

Use rsync for the transfer rather than scp (rsync can use the ssh protocol) and you will know what is already transferred.  

If you need the files compressed, you can also use something like logrotate to compress the older logs and keep the newer one uncompressed.

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have written articles previously comparing SARDU and YUMI.  I also included a couple of lines about Easy2boot (easy2boot.com).  I have now been using, and enjoying easy2boot as my sole multiboot utility for some years and realize that it deserves …
This article will show you step-by-step instructions to build your own NTP CentOS server.  The network diagram shows the best practice to setup the NTP server farm for redundancy.  This article also serves as your NTP server documentation.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

607 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question