script to copy, compress and transfer archive logfiles to another server in Linux

Posted on 2011-05-12
Last Modified: 2012-05-11
I have a requirement to compress the Oracle database archive log files having the format 'arch_%t_%s.arc'  and transfer it to another server, both running on Linux. I am planning to do it like this -

1. copy the archive logfiles to another location in the same server using rsync.
2. compress the logfiles from that location. A script will run every 1 hour and compress all the uncompressed files using 'tar'.
3. transfer the compressed logfiles using SCP to the another server. (for every 1 hour).

But the big question at the 2nd & 3rd step how it'll identify the files which have already been compressed(and copied) and which are not. Please let me know if there is any other way too to achieve this.
Question by:makk2010
    LVL 19

    Expert Comment

    I would suggest rsync to a temporary location - a location in which all the work is performed & only once a file is copied, is it moved to your intended destination.

    Picking up the different arc files is simple enough with a 'for x in *.arc ; do tar .... "$x" ; done' style for loop.

    Another for loop can then iterate through the tar files and scp them across.  The bash $? special variable will inform you of the exit code from the previous command.  You can therefore test whether the scp was successful [ $? -eq 0 ] and, if successful, move the tar file to your end folder on the Oracle machine.
    LVL 31

    Accepted Solution

    Why don't you simply rsync it from one server to another, with compression?

    rsync -avz /path/source/ user@remoteServer:/target/path

    z- compresses the network delivery
    a- keeps all permissions and ownership.
    LVL 12

    Expert Comment

    farzanj beat me to what I would suggest.

    Use rsync for the transfer rather than scp (rsync can use the ssh protocol) and you will know what is already transferred.  

    If you need the files compressed, you can also use something like logrotate to compress the older logs and keep the newer one uncompressed.

    Featured Post

    How to run any project with ease

    Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
    - Combine task lists, docs, spreadsheets, and chat in one
    - View and edit from mobile/offline
    - Cut down on emails

    Join & Write a Comment

    I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension ( This reminded me of questions tha…
    Join Greg Farro and Ethan Banks from Packet Pushers ( and Greg Ross from Paessler ( for a discussion about smart network …
    Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
    Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

    745 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now