automate a process in Linux Server

Posted on 2011-10-26
Medium Priority
Last Modified: 2012-06-27
I am on a shared hosted server. I have access to a cpanel and to ftp services. I have an app which creates about 6 files per day each of about 25 mb. The files stay on the server but I would like some way to automate a process of retrieving the latest files as they come in or at specified times every day. Can anyone suggest how I might go about this? The cpanel looks something like the below

THANK YOU> control panel
Question by:onyourmark
  • 2
  • 2
  • 2
  • +1
LVL 17

Accepted Solution

bigeven2002 earned 2000 total points
ID: 37036005

Normally this is done with CRON jobs.  They are just simple batch scripts that you create that run a series of commands including in this case, copying files at specified intervals.  However, since your server is shared, these jobs may be disabled.  Looking at your screenshot, my only guess is they may be under Script Handlers in the center.  If not, ask your provider about enabling them if they are not.

The link below is to a tutorial on using cron jobs:

LVL 19

Expert Comment

ID: 37036216
What operating system (and version/distribution if appropriate) is the system that you will use to retrieve these files running?

Are you currently at this time able to use FTP to connect to your shared hosted server, and be able to manually retrieve these files?  If not, how exactly do you view these files, and do you know where (as in /what/file/path) they are on the server?

These answers should help me craft a solution to your problem.

Author Comment

ID: 37037281
Hi. Thank you. The current hosting platform of the hosting product: Linux.
I can FTP to manually get the files and I do know the full path to the files. There are about 2000 files there but I only want to get the ones generated in the previous 24 hours (there would be about 6 generated in that time frame).
Thanks again.
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

LVL 19

Expert Comment

ID: 37038003
I was asking what the operating system that you will be using to RETRIEVE them, ie. the client not the server.  That is where the magic will most likely have to happen.
LVL 16

Expert Comment

ID: 37038985
If Windows is on the machine used to retrieve the files you could use the following Robo-FTP script to download any files with a timestamp less than 24 hours.  This logic also checks to see if you already downloaded a file with the exact same name, date and size so it won't waste time downloading the same files over and over again.  The logic will retry a failed download 3 times before giving up.
WORKINGDIR "c:\local\download\destination\folder"
set check_date = %datetime
DATETIMESUB check_date 1 /day
FTPLOGON "ftp.mydomain.com" /user="UserID" /pw="Secret"
IFDATETIME< %sitefiledatetime check_date GOTO get_file_stats
;; did we already download this one?
IFNUM!= %sitefilesize %nextfilesize GOTO download
IFDATETIME!= %sitefiledatetime %nextfiledatetime GOTO download
;; if we reached this point we'll skip the download because we 
;; already have a local file with the same name, size, and date
GOTO get_file_stats
RCVFILE %sitefile /serverdate
PAUSE /for=5  
LOOPTO download_attempt
;; if we reached this point the download failed 3 times
GOTO get_file_stats
CRON "@hourly"
GOTO top

Open in new window

This script is written to do its download and then sleep until the next hour at which point it wakes up and does another run.  It keeps doing this indefinately.  The idea is that you would use the Service Installer program that comes with Robo-FTP to install this script as a Windows Server so the OS will automatically restart this whenever the computer is rebooted.  To use this script you'd need to modify the local destination folder on line 2 and the FTP server address and credentials on line 5.

Author Comment

ID: 37042403
Hello and thanks very much. I guess this means I need to purchase Robo-ftp.
Also, you said "to install this script as a Windows Server". Does this happen automatically when I install Robo-ftp on Windows 7, does it become a server?

What is the difference between
DATETIMESUB check_date 1 /day


CRON "@hourly" ?

If I want to check at specific times like 1:45 am and every 4 hours after that what would I do?
And is there a way to specify that I only want the most recent file (or even the 2nd to most recent file)?

Thanks again.
LVL 16

Expert Comment

ID: 37042513
I already have Robo-FTP and I use it to do a lot of stuff... on the other hand they say "if your only tool is a hammer then every problem looks like a nail" so take that for what its worth!

I don't know if Robo-FTP would be worth it to you but the download has a 30-day trial so you can see if it works for you and if it is worth it.

The program is all about automating file processes so the interface is geared to that.  I mostly use the Script Editor to make scripts because it has keyword highlighting.  Then, once I get my script written, I run it in the main Robo-FTP console window to see if it works.  Edit to fix bugs.  Save changes.  Run again.  Wash, risne repeat until it works.  Then use the Service Installer program in the Robo-FTP start menu folder to create a Windows Service that runs your script.

So I guess I would say Robo-FTP is more of a development tool to create automation scripts that you can use as Scheduled Tasks or Windows Services.

With regard to your specific script questions:

DATETIMESUB check_date 1 /day   --> This subtracts 1 day from check_date variable

CRON "@hourly"   --> This suspends script execution until the beginning of the next hour

If you want to start at 1:45am and then every four hours you have a few options.  The easiest would be to insert PAUSE /until=01:45 as the very top row and then replace the CRON "@hourly" on the next to last row with PAUSE /for=14400 but the problem with this is it would tend to drift by the amount of execution time.  If execution time was fairly predictable you could mitigate by using a number slightly smaller than 14400 (# of seconds in 4 hours) but that is kinda hacky.  Another option would be to use the IFTIME conditional branch at the bottom like this:
IFTIME> "21:00" GOTO wait_until_145
PAUSE /for=14400
GOTO top
PAUSE /until=1:45
GOTO top

The third option, and perhaps the best, would be to use an external CRON schedule file.  The default name for such a file is crontab.txt but you can pass an alternate name as a input argument to CRON if you want.  I would have mentioned this first but the crontab file format can be hard to grok.  To meet your proposed schedule the crontab file would containt this one row:
45      1-21/4      *      *      *
Basically this means 45 minutes after the hour and between the hours of 1 and 21 every 4th hour on every day of every month and every day of the week... but all you need to know is save that line in a file named crontab.txt in the Robo-FTP install folder and then change the CRON "@hourly" line near the bottom to just plain CRON and it will work.

If you want to only download the most recent file you can remove all that stuff about comparing the remote file and the local file and just do this:
GETSITEFILE "*" /newest
RCVFILE %sitefile

I don't know an easy way to get the second to newest.  I think I would start with GETSITEFILE "*" /oldest and then copy the value of %sitefiledatetime into a variable, lets call it T1 and the value of %sitefile into a variable named N1.  Then I would do a FTPGETREWIND to reset the file point and use FTPGETFILE "*" in a loop where each time through the loop I use IFDATETIME to compare the new value of %sitefiledatetime to V1 and, when it was newer, i would copy the value of T1 into a variable lets say T2 and copy N1 into N2 and then copy %sitefiledatetime into T1 and %sitefile into N1.  Then I would have it loop back and process the next file.  If the file isn't newer I just skip the variable value shuffle and go back for another loop iteration.  I keep doing this one at a time until FTPGETFILE "*" returns an error indicating there are no more files.  At this point N1 and T1 should contain the name  and timestamp of the newest file and N2 and T2 should contain the name and timestamp of the second newest file.

If you managed to read all this and understand 20% of it then you are probably the type of person who would have no problem learning to write automation scripts in Robo-FTP

Featured Post

Restore individual SQL databases with ease

Veeam Explorer for Microsoft SQL Server delivers an easy-to-use, wizard-driven interface for restoring your databases from a backup. No expert SQL background required. Web interface provides a complete view of all available SQL databases to simplify the recovery of lost database

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses
Course of the Month15 days, 18 hours left to enroll

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question