automate a process in Linux Server

Posted on 2011-10-26
Last Modified: 2012-06-27
I am on a shared hosted server. I have access to a cpanel and to ftp services. I have an app which creates about 6 files per day each of about 25 mb. The files stay on the server but I would like some way to automate a process of retrieving the latest files as they come in or at specified times every day. Can anyone suggest how I might go about this? The cpanel looks something like the below

THANK YOU> control panel
Question by:onyourmark
    LVL 17

    Accepted Solution


    Normally this is done with CRON jobs.  They are just simple batch scripts that you create that run a series of commands including in this case, copying files at specified intervals.  However, since your server is shared, these jobs may be disabled.  Looking at your screenshot, my only guess is they may be under Script Handlers in the center.  If not, ask your provider about enabling them if they are not.

    The link below is to a tutorial on using cron jobs:
    LVL 19

    Expert Comment

    What operating system (and version/distribution if appropriate) is the system that you will use to retrieve these files running?

    Are you currently at this time able to use FTP to connect to your shared hosted server, and be able to manually retrieve these files?  If not, how exactly do you view these files, and do you know where (as in /what/file/path) they are on the server?

    These answers should help me craft a solution to your problem.

    Author Comment

    Hi. Thank you. The current hosting platform of the hosting product: Linux.
    I can FTP to manually get the files and I do know the full path to the files. There are about 2000 files there but I only want to get the ones generated in the previous 24 hours (there would be about 6 generated in that time frame).
    Thanks again.
    LVL 19

    Expert Comment

    I was asking what the operating system that you will be using to RETRIEVE them, ie. the client not the server.  That is where the magic will most likely have to happen.
    LVL 16

    Expert Comment

    If Windows is on the machine used to retrieve the files you could use the following Robo-FTP script to download any files with a timestamp less than 24 hours.  This logic also checks to see if you already downloaded a file with the exact same name, date and size so it won't waste time downloading the same files over and over again.  The logic will retry a failed download 3 times before giving up.
    WORKINGDIR "c:\local\download\destination\folder"
    set check_date = %datetime
    DATETIMESUB check_date 1 /day
    FTPLOGON "" /user="UserID" /pw="Secret"
    IFDATETIME< %sitefiledatetime check_date GOTO get_file_stats
    ;; did we already download this one?
    GETNEXTFILE %sitefile
    IFERROR GOTO download
    IFNUM!= %sitefilesize %nextfilesize GOTO download
    IFDATETIME!= %sitefiledatetime %nextfiledatetime GOTO download
    ;; if we reached this point we'll skip the download because we 
    ;; already have a local file with the same name, size, and date
    GOTO get_file_stats
    RCVFILE %sitefile /serverdate
    IFERROR= $ERROR_SUCCESS GOTO get_file_stats
    PAUSE /for=5  
    LOOPTO download_attempt
    ;; if we reached this point the download failed 3 times
    GOTO get_file_stats
    CRON "@hourly"
    GOTO top

    Open in new window

    This script is written to do its download and then sleep until the next hour at which point it wakes up and does another run.  It keeps doing this indefinately.  The idea is that you would use the Service Installer program that comes with Robo-FTP to install this script as a Windows Server so the OS will automatically restart this whenever the computer is rebooted.  To use this script you'd need to modify the local destination folder on line 2 and the FTP server address and credentials on line 5.

    Author Comment

    Hello and thanks very much. I guess this means I need to purchase Robo-ftp.
    Also, you said "to install this script as a Windows Server". Does this happen automatically when I install Robo-ftp on Windows 7, does it become a server?

    What is the difference between
    DATETIMESUB check_date 1 /day


    CRON "@hourly" ?

    If I want to check at specific times like 1:45 am and every 4 hours after that what would I do?
    And is there a way to specify that I only want the most recent file (or even the 2nd to most recent file)?

    Thanks again.
    LVL 16

    Expert Comment

    I already have Robo-FTP and I use it to do a lot of stuff... on the other hand they say "if your only tool is a hammer then every problem looks like a nail" so take that for what its worth!

    I don't know if Robo-FTP would be worth it to you but the download has a 30-day trial so you can see if it works for you and if it is worth it.

    The program is all about automating file processes so the interface is geared to that.  I mostly use the Script Editor to make scripts because it has keyword highlighting.  Then, once I get my script written, I run it in the main Robo-FTP console window to see if it works.  Edit to fix bugs.  Save changes.  Run again.  Wash, risne repeat until it works.  Then use the Service Installer program in the Robo-FTP start menu folder to create a Windows Service that runs your script.

    So I guess I would say Robo-FTP is more of a development tool to create automation scripts that you can use as Scheduled Tasks or Windows Services.

    With regard to your specific script questions:

    DATETIMESUB check_date 1 /day   --> This subtracts 1 day from check_date variable

    CRON "@hourly"   --> This suspends script execution until the beginning of the next hour

    If you want to start at 1:45am and then every four hours you have a few options.  The easiest would be to insert PAUSE /until=01:45 as the very top row and then replace the CRON "@hourly" on the next to last row with PAUSE /for=14400 but the problem with this is it would tend to drift by the amount of execution time.  If execution time was fairly predictable you could mitigate by using a number slightly smaller than 14400 (# of seconds in 4 hours) but that is kinda hacky.  Another option would be to use the IFTIME conditional branch at the bottom like this:
    IFTIME> "21:00" GOTO wait_until_145
    PAUSE /for=14400
    GOTO top
    PAUSE /until=1:45
    GOTO top

    The third option, and perhaps the best, would be to use an external CRON schedule file.  The default name for such a file is crontab.txt but you can pass an alternate name as a input argument to CRON if you want.  I would have mentioned this first but the crontab file format can be hard to grok.  To meet your proposed schedule the crontab file would containt this one row:
    45      1-21/4      *      *      *
    Basically this means 45 minutes after the hour and between the hours of 1 and 21 every 4th hour on every day of every month and every day of the week... but all you need to know is save that line in a file named crontab.txt in the Robo-FTP install folder and then change the CRON "@hourly" line near the bottom to just plain CRON and it will work.

    If you want to only download the most recent file you can remove all that stuff about comparing the remote file and the local file and just do this:
    GETSITEFILE "*" /newest
    RCVFILE %sitefile

    I don't know an easy way to get the second to newest.  I think I would start with GETSITEFILE "*" /oldest and then copy the value of %sitefiledatetime into a variable, lets call it T1 and the value of %sitefile into a variable named N1.  Then I would do a FTPGETREWIND to reset the file point and use FTPGETFILE "*" in a loop where each time through the loop I use IFDATETIME to compare the new value of %sitefiledatetime to V1 and, when it was newer, i would copy the value of T1 into a variable lets say T2 and copy N1 into N2 and then copy %sitefiledatetime into T1 and %sitefile into N1.  Then I would have it loop back and process the next file.  If the file isn't newer I just skip the variable value shuffle and go back for another loop iteration.  I keep doing this one at a time until FTPGETFILE "*" returns an error indicating there are no more files.  At this point N1 and T1 should contain the name  and timestamp of the newest file and N2 and T2 should contain the name and timestamp of the second newest file.

    If you managed to read all this and understand 20% of it then you are probably the type of person who would have no problem learning to write automation scripts in Robo-FTP

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Better Security Awareness With Threat Intelligence

    See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

    If you are running a LAMP infrastructure, this little code snippet is very helpful if you are serving lots of HTML, JavaScript and CSS-related information. The mod_deflate module, which is part of the Apache 2.2 application, provides the DEFLATE…
    Over the last ten+ years I have seen Linux configuration tools come and go. In the early days there was the tried-and-true, all-powerful linuxconf that many thought would remain the one and only Linux configuration tool until the end of times. Well,…
    Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
    Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

    760 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    11 Experts available now in Live!

    Get 1:1 Help Now