[Webinar] Learn how to a build a cloud-first strategyRegister Now


Basic service to process CSV files and run bash scripts

Posted on 2012-08-22
Medium Priority
Last Modified: 2012-08-23
So I have a series of bash scripts that take various parameters.

An external system drops parametrised CSV to a pick up location.

I need a basic service that I will pick up these simple CSV files when then arrive and pass them over to the scripts and execute them.

What's a light weight way to execute this? I'd rather not have a ultra frequently cron job, rather something reactive to the arrival of the file.
Question by:Borgs8472
  • 3
  • 3
LVL 31

Expert Comment

ID: 38321556
May be counter intuitive to you but "ultra frequently cron job" would do it as a daemon process.  That system process is working anyway.  If you want a script "interactively", it would be an infinite loop to constantly check the presence of a newer file.  Cron, IMHO, would be the way to go.

If you want to make your own service, that would be working in addition to cron, to basically do that same thing.  The service already running in the system is cron, that does this kind of job.  Any allergy to cron?

Author Comment

ID: 38322969
I'm new to Linux (windows background) but I always found services were more robust than scheduled tasks.

Also, I don't want more than one instance of the job running at any one time (jobs should be processed sequentially, not concurrently) and I worry running at a cron could allow jobs to run out of control which mustn't happen.
LVL 31

Expert Comment

ID: 38323158
Cron is a service (crond).  Depending upon what distribution you are using, you can start or stop it.  In Red hat family of OSes, you can do

service crond start
service crond stop
service crond restart

Running multiple services at a time is not a problem but if you must not run them concurrently, you can do so by either checking if a particular process is running or keeping a lock file.
Get your Disaster Recovery as a Service basics

Disaster Recovery as a Service is one go-to solution that revolutionizes DR planning. Implementing DRaaS could be an efficient process, easily accessible to non-DR experts. Learn about monitoring, testing, executing failovers and failbacks to ensure a "healthy" DR environment.


Author Comment

ID: 38324316
Right... so I think I'm almost there, I can write a loop to search for the files, check lock conditions and spawn the scripts - so far so good.

But this script itself, I just need to ensure it's always running. Should I be terminating and re initiating it with cron? Isn't there a more graceful way to ensure my loop script is always running than that?
LVL 31

Accepted Solution

farzanj earned 2000 total points
ID: 38324533
So you mean checking every minute would not be enough?

So your script goes something like


#Lock file location

#Source location -- directory of files

#Target Dir

#Check existence of lock file
[[ -f $FILE ]]  && exit 1

#create lock file
touch $FILE

#Move files created within one minute
find $SOURCE -type f -mmin -1 -exec mv "{}" $TARGET

#Remove the lock file
rm -f $FILE

Open in new window

This script is not tested so you will have to test it.  IMHO running once a minute alone is enough but if you want it to run every two seconds, you can put a look around the find command that checks every it about 26 times or so and then it is called again my cron.  Running loop alone is not reliable as it would die and then the whole process will end.

for i in {1..26}
    find  ...
    sleep 2s

Open in new window


Author Closing Comment

ID: 38324668
Thanks, looks like I've got a framework for my task queuing system now!

Featured Post

Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
Suggested Courses
Course of the Month20 days, 10 hours left to enroll

868 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question