Unix ksh: scripting

Dear Expert,

Upon arrival of a file a script is kicked off.

1) first thing it does is a check on available resources.
That check creates an outputfile.
Output file example is like this:

CPU#Resource       Available Qty  UsedBy
WKS1#SMALL               1       1     No holders of this resource       
WKS1#LARGE               1       1     No holders of this resource
WKS1#SMALL_LARGE    2       2     No holders of this resource

2) the script should read this output file and check which resource has a qty > 0.
This should be set off to a variable holding number of records ($REC).

If resource >0 = WKS1#SMALL       and $REC << 2500       => action1
If resource >0 = WKS1#LARGE       and $REC  > 2500    => action2
If resource >0 = WKS1#SMALL_LARGE        => action3 (size doesn't matter)

3) If the outputfile shows that no resources are free like in below example it should redo the check each 2 minutes until an appropriate resource is found.
      
CPU#Resource            Available Qty         UsedBy
WKS1#SMALL              1             0             JOB1
WKS1#LARGE              1             0             JOB2
WKS1#SMALL_LARGE   2             0             JOB3
                                                                  JOB4
                                                        
    Possible scenario's here:
            
If resource >0 = WKS1#SMALL       and $REC  > 2500   => redo check
If resource >0 = WKS1#LARGE       and $REC << 2500   => redo check

~~~~~~~~~~~~~~~~~~~
I all comes down to this in fact:

Queue 1: Dedicated to Small requests
Queue 2: Small and Large
Queue 3: Small and Large
Queue 4: Dedicated to Large requests

I understand this may not be too obvious, and not so easy either.
Let me know if you need more explanation, and also if you would think this is worth more than 500.
Thank you very much.
WatnogAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

ozoCommented:
perl -an - <<'EOF' outputfile                                                              
  next unless $F[2];                                                                        
  if( $F[2] < 2500 && $F[0] eq 'WKS1#SMALL' ){                                              
    system $a="action1";                                                                    
  }elsif( $F[2] > 2500 && $F[0] eq 'WKS1#LARGE' ){                                          
    system $a="action2";                                                                    
  }                                                                                        
  if( $F[0] eq 'WKS1#SMALL_LARGE' ){                                                        
    system $a="action3";                                                                    
  }                                                                                        
}continue{ sleep 120 and seek ARGV,0,0 if !$a && eof                                        
EOF        
0
WatnogAuthor Commented:
Thank you Ozo but I couldn't make it work.

This is what I get:
+ perl -an - outputfile
+ 0< /tmp/sh26304.1

There is no reference to 'number of records' [$REC] but when I change 2500 to $REC that does not make any difference.

I don't mean to chase you on this though.
Cheers.
0
ozoCommented:
I was confused about where you were getting $REC
From your examples, it looked like was from the Qty column.
Could you clarify what $REC refers to?
Also, where does "resource" come from?
And what should be done if more than one of the 5 conditions you list are satisfied?
0
Microsoft Azure 2017

Azure has a changed a lot since it was originally introduce by adding new services and features. Do you know everything you need to about Azure? This course will teach you about the Azure App Service, monitoring and application insights, DevOps, and Team Services.

WatnogAuthor Commented:
Ok, sorry Ozo.

The context is a 'job scheduling' software.
A limited number of jobs (4) are allowed to run in parallel.
Those jobs process files.
The number of jobs is limited to 4 by assigning to each job a 'need' for a (virtual) resource (this long time ago used to be physical, a tape drive or so).
The number of resources is limited to 4.
Some files are small others are large.
The are transferred together (mget), so upto 10 files can land at a time.

§REC is generated elsewhere in the script (a [wc] reads number of records in the file).

I'm searching for the most efficient way of processing, in fact mimic some kind of load balancing, so that the max potential of 4 jobs is used while ensuring that at all times at least 1 resource is reserved for small and 1 for large files.  The two others can be either small or large, or both large or both small.


The check on availble resources  is a commmand that outputs a snapshot like this:
(I made the names a bit shorter ...)

CPU#Resource       Available Qty  UsedBy
SMALL                      1              1     No holders of this resource      
LARGE                      1              1     No holders of this resource
SMALL_LARGE          2              2     No holders of this resource

Available= total
Qty=remaining, so that is the column to look at.

Your question on 5 conditions: I hope these are the ones...
If there would be a choice (small file and both SMALL and SMALL_LARGE is available, SMALL would be first choice).

file is either large or small (smaller or larger than 2500 records)
1) if < 2500 and recource small is available: small
2) if > 2500 and resource large is available: large
3) if  < 2500 and resource small_large is availble: small_large
4) if  > 2500 and resource small_large is availble: small_large
5) if no resource is available: wait and redo check (re-produce available resources snapshot)


So the script begins like this:

#!/bin/ksh
#functions per chosen queue/resource
action1 ()
{
kick of job using SMALL resource
}
action2 ()
{
kick of job using LARGE resource
}
action3 ()
{
kick of job using SMALL_LARGE resource
}
#
`check_available_resources_cmd` > snapshot
`REC=[cat inputfile | wc -l `                                 #
`your lines`                                                         # hopefully

A whole lot of explaining, hope you got the picture.
This check would be done for each file (one by one), the process for that is in place.


If you think this is all too much asked, well maybe it is, you can just say it...
Thanks!
Pascal





0
WatnogAuthor Commented:
I will close this question.
Possibly cut into pieces and rephrase it.
Thanks anyway.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
WatnogAuthor Commented:
Ok for me.
0
WatnogAuthor Commented:
Too complex a question I suppose.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.