shell script for a task management workflow

Hi Friends,

Would like to come-up with a shell script to achieve below workflow with your help
Any suggestions/advice with sample scripts would really help :)

Have attached the input property file. Which I assume…I will have.

1.      Would like to read this file and construct and HTTP request based on the property values.
2.      As a first execution, would like to loop through each job (job1, job2 etc) to check below properties and trigger it in background.
job_n_dependent_on=“none”
job_n_status="not started"
3.      When job triggered, it should change the job_1_status=”started”
4.      When a job finishes it should update the job_1_status=”success” or “failure”
5.      If a job has a dependency. Then it should wait for them to finish.
E.g
job_2_dependent_on=”job1”
job_3_dependent_on=”job1,job2”
job_6_dependent_on=”job3”
6.      Based on this property “max_on_running_task=2” no more than 2 jobs should be running at a given time.
7.      It should constantly monitor running jobs and trigger next job if dependency met or none

I have decent experience in shell script, So if you guide with the workflow and logic. I think I can manage :)

I thinking of having two sh files.
Master script will have few functions
      checkDependency
      checkStatus
      triggerJob
      updateStatus
checkRunningJob
Monitor script to check the running jobs and call master script (master script will invoke this monitor script at the beginning)
task.txt
enthuguyAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

arnoldCommented:
Where are you planing on maintaining the state of scripts?
Maintaining state in a DB simplifies status updates.

Depending on what the tasks are, which shell scripting languages (programming/interpretted) are you considering?
sh, bash, ksh, zsh, perl, python, tcl, php, curl, awk ?
enthuguyAuthor Commented:
hi arnold,

Thought of maintaining the state in the file, but I also thought of open source mysql but I have to really push for this to get this installed. Or any other suggestion on the db?

I was thinking bash with curl (for http call), If python make this bit easier then...we can.

On the response/outcome of http call....I think, I can manage to that...so we can update the job status on response.
arnoldCommented:
depending on the programming, some have built in DBM, berkerley DB, etc.
are you running one task at a time?
Who or what is going to look at the status of the job.
It is one thing to keep the state when the job runs for hours. if it only runs for a few seconds, minutes, what is the point? or you have schedule that triggers the run of this process every minute. and you need the state to make sure you do not have two processes working the same job?
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

enthuguyAuthor Commented:
depending on the programming, some have built in DBM, berkerley DB, etc.
OK


are you running one task at a time?
No, I will be triggering min of 15 tasks at a time. (If i increase the system resource later, we could even increase the running task to 20 or more.


Who or what is going to look at the status of the job.
the web app which we trigger has a post event (wrapped with shell script), which i'm thinking of using it to update the status (SUCCESS OR FAILURE). So the monitor script will identify the new status when its scanning the file in next interval (e.g monitor script will run for every 20 sec looking for possible next task to trigger)


It is one thing to keep the state when the job runs for hours. if it only runs for a few seconds, minutes, what is the point?
Each task will run for atleast 5 minutes and the long running task will run for 5 hours. In total I have 100 Tasks to manage. for simiplicty I had  mentioned only 6 in the sample input file.


or you have schedule that triggers the run of this process every minute.
yes, we can schedule or trigger the master script manually when required. which in turn trigger the monitor script (runs every 20 secs)


and you need the state to make sure you do not have two processes working the same job?
Thanks, Very valid point. May be we can manage by introducing a lock system. (as soon as I trigger a job...I create a lock file. once finished...remove the lock)
arnoldCommented:
There are many ways to do that, with locking system, you would need a secondary check to make sure the processor did not die half way through leaving the lock.
using a DB or a notification event event.

Do you have a process diagram.
start job, read in data, update status or add lock, ...

note that the change to the status in the file will only occur when the file is written out.
if you open a file read/write and update , not sure the monitoring setup will see the change in the status........ when it reads the file.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
enthuguyAuthor Commented:
thx arnold for your prompt help.

yes, thats a valid point too. we have to handle the termination scenario.

No, I dont have but I thought of preparing one. Will upload it.

Ok, not sure about the read file would reflect recent status change. I will perform a quick check :)
enthuguyAuthor Commented:
have attached the high level flow :)

pls correct me if I'm wrong
JobController.pdf
arnoldCommented:
It should work for the flow.
enthuguyAuthor Commented:
thx arnold.
enthuguyAuthor Commented:
Thanks
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.