Basic service to process CSV files and run bash scripts

Posted on 2012-08-22
Last Modified: 2012-08-23
So I have a series of bash scripts that take various parameters.

An external system drops parametrised CSV to a pick up location.

I need a basic service that I will pick up these simple CSV files when then arrive and pass them over to the scripts and execute them.

What's a light weight way to execute this? I'd rather not have a ultra frequently cron job, rather something reactive to the arrival of the file.
Question by:Borgs8472
    LVL 31

    Expert Comment

    May be counter intuitive to you but "ultra frequently cron job" would do it as a daemon process.  That system process is working anyway.  If you want a script "interactively", it would be an infinite loop to constantly check the presence of a newer file.  Cron, IMHO, would be the way to go.

    If you want to make your own service, that would be working in addition to cron, to basically do that same thing.  The service already running in the system is cron, that does this kind of job.  Any allergy to cron?
    LVL 4

    Author Comment

    I'm new to Linux (windows background) but I always found services were more robust than scheduled tasks.

    Also, I don't want more than one instance of the job running at any one time (jobs should be processed sequentially, not concurrently) and I worry running at a cron could allow jobs to run out of control which mustn't happen.
    LVL 31

    Expert Comment

    Cron is a service (crond).  Depending upon what distribution you are using, you can start or stop it.  In Red hat family of OSes, you can do

    service crond start
    service crond stop
    service crond restart

    Running multiple services at a time is not a problem but if you must not run them concurrently, you can do so by either checking if a particular process is running or keeping a lock file.
    LVL 4

    Author Comment

    Right... so I think I'm almost there, I can write a loop to search for the files, check lock conditions and spawn the scripts - so far so good.

    But this script itself, I just need to ensure it's always running. Should I be terminating and re initiating it with cron? Isn't there a more graceful way to ensure my loop script is always running than that?
    LVL 31

    Accepted Solution

    So you mean checking every minute would not be enough?

    So your script goes something like

    #Lock file location
    #Source location -- directory of files
    #Target Dir
    #Check existence of lock file
    [[ -f $FILE ]]  && exit 1
    #create lock file
    touch $FILE
    #Move files created within one minute
    find $SOURCE -type f -mmin -1 -exec mv "{}" $TARGET
    #Remove the lock file
    rm -f $FILE

    Open in new window

    This script is not tested so you will have to test it.  IMHO running once a minute alone is enough but if you want it to run every two seconds, you can put a look around the find command that checks every it about 26 times or so and then it is called again my cron.  Running loop alone is not reliable as it would die and then the whole process will end.

    for i in {1..26}
        find  ...
        sleep 2s

    Open in new window

    LVL 4

    Author Closing Comment

    Thanks, looks like I've got a framework for my task queuing system now!

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Highfive + Dolby Voice = No More Audio Complaints!

    Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

    The purpose of this article is to demonstrate how we can use conditional statements using Python.
    Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
    Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
    Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

    779 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now