Copying files from the Production server to a target database server

As part of the new ECO Migration process, we will migrate the Production database Agile ECOs to a target test database.  Part of this new process will involve copying files from the Production server (mesa) to a target database server - in this case UAT2 (extapuata).  We are hoping you can write a Unix script to take care of the file copying.  This Unix script should exist on the Production Server and will be scheduled to run once a day.  The specifics of the script are below:
The source directory for the files will be on mesa in /n01/applprod/PROD11i/appl/ilink/11.5.0/processed.
The target directory for the files will be on extapuata in /n01/uat2/prod11iappl/ilink/11.5.0/incoming.
The files you would need to copy will all begin with a 'TO' and have a extension of 'AXML'.  For example TO00001.AXML.
The number of files in the source directory will keep growing.  You don't need to copy them all.  You will have to use the following logic to determine if a file needs to be copied or not:
If the specific file in the source directory, say TO00001.AXML, exists on the target server in the  /n01/uat2/prod11iappl/ilink/11.5.0/processed directory, then you don't need to copy that file - it has already been processed.  
If it doesn't exist in that directory, then you need to check if the file exists on the target server in the  /n01/uat2/prod11iappl/ilink/11.5.0/incoming directory.  If it does exist there, then you don't want to copy that file - it is already there waiting to be processed.  
If it doesn't exist  in /n01/uat2/prod11iappl/ilink/11.5.0/incoming , then you need to copy that file to that directory.
You will need to make sure the files which are copied have the correct permissions and ownership/grouping so that Agile can process them.  Currently on mesa the files are put in the /n01/applprod/PROD11i/appl/ilink/11.5.0/incoming directory as agile:oaa.  When Agile processes the files, it moves them to the /n01/applprod/PROD11i/appl/ilink/11.5.0/processed directory as applprod:oaa.  When you are copying the files from the mesa  /n01/applprod/PROD11i/appl/ilink/11.5.0/processed directory and putting them in extapuata /n01/uat2/prod11iappl/ilink/11.5.0/incoming, you will have to change their ownership/grouping from applprod:oaa to agile:oaa.
When you are writing this script, keep in mind that these files may need to be moved to more than one target database server.  Initially we are only migrating the ECO files from Production to UAT2.  In the near future we may be migrating them from Production to SIT2 or TST or some other database also.  So make your script generic enough that it can be used for multiple target databases or easy enough to modify if a different version of the script is needed for each target database.  In your script, the source database will ALWAYS be Production (mesa).  Just the target database may change.
Who is Participating?
Check out the rsync utility, it does exactly the type of thing you're looking to do without the need for writing a complex script.
Is this a task you were assigned and you need assistance?
Please post what you have come up with so far.
I'm unfamiliar with the database you mentioned could you provide additional detail?
Depending on the capability of the database, do you have an option to replicate the database? Mirror the database, backup the database?
Gerwin Jansen, EE MVETopic Advisor Commented:
I'd go for rsync as well. You could start with this command:
rsync /n01/applprod/PROD11i/appl/ilink/11.5.0/processed/TO*.AXML extapuata:/n01/uat2/prod11iappl/ilink/11.5.0/incoming/

Open in new window

Add other targets in a later stage by just copying the line and changing the destination (extapuata:/ part).

Man page rsync is here with some samples.
Cloud Class® Course: CompTIA Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

pride-itAuthor Commented:
Will this work if this is not a SAMBA share?
Gerwin Jansen, EE MVETopic Advisor Commented:
Hello pride-it, did you try the suggestions we gave you and did you get any errors?

Rsync in essence is just a smart 'remote copy' utility from 1 system to another (remote) system.
rsync does not require SAMBA or any other remote mounting.  It uses its own protocol.  You do need to install the rsync daemon on the server side though, so this may depend on whether you have root access.
pride-itAuthor Commented:
Rsync works great, but I still need to check two locations on the destination server to see if the files exist in these locations and then only run the rsync if the files do not exist is these location on destination server.
pride-itAuthor Commented:
The two dirctories I have to check for the files are:
n01/uat2/prod11iappl/ilink/11.5.0/processed, /n01/uat2/prod11iappl/ilink/11.5.0/incoming
pride-itAuthor Commented:
So far I have this as my script:

rsync -avvvvz -e "ssh -i /home/rsync/.ssh/rsync-key" /n01/applprod/PROD11i/appl/ilink/11.5.0/processed/TO*.AXML rsync@extapuata:/n01/uat2/prod11iappl/ilink/11.5.0/incoming/
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.