We help IT Professionals succeed at work.

Unix ksh: read/count values in column

Watnog
Watnog asked
on
910 Views
Last Modified: 2012-05-11
Dear Expert,

A file is produced at every job failure (ABEND).
Each time the job fails 5 times in a row, we need an action.
So at the 5th time, the 10th time, the 15th time etc.
Like in this case:  

>>every run                                 JOB_X                    SUCC  75 15:19  00:01           0 #J891
 >>every run                                 JOB_X                    SUCC  75 15:21  00:01           0 #J4400
 >>every run                                 JOB_X                    SUCC  75 15:23  00:01           0 #J16712
 >>every run                                 JOB_X                    SUCC  75 15:25  00:01           1 #J27860
 >>every run                                 JOB_X                    SUCC  75 15:27  00:01           1 #J4936
 >>every run                                 JOB_X                    SUCC  75 15:29  00:01           1 #J15944
 >>every run                                 JOB_X                    SUCC  75 15:31  00:01           1 #J27567
 >>every run                                 JOB_X                    SUCC  75 15:33  00:01           1 #J6735
 >>every run                                 JOB_X                    SUCC  75 15:35  00:02           1 #J13349
 >>every run                                 JOB_X                    ABEND 75 15:37  00:01           1 #J20682
 >>every run                                 JOB_X                    ABEND 10 15:39  00:01           1 #J25705
 >>every run                                 JOB_X                    ABEND 10 16:01  00:02           1 #J24643
 >>every run                                 JOB_X                    ABEND 10 16:03  00:02           1 #J12880
 >>every run                                 JOB_X                    ABEND 10 16:05  00:02           1 #J24973; &0:02

Open in new window


So in fact what would need to happen is that the file is read and that a variable is set if the number of times ABEND is reported after the last SUCC is a mutiple of 5.

The very last characters in the snippet are [&0:02] meaning that the job runs every 2 minutes. It would be great if that would be taken into account.
If 'repeat' = 0:02 and 'number_of_ABEND_multiple_of_5' = yes then...

With other jobs the repeat is set differently so we may need to be alerted on 3rd or 2nd ABEND in a row.

Feasable?
Thanks and respect.
Comment
Watch Question

Author

Commented:
In other words:

tail -1 | grep &0:02 inputfile
if [ "$?" -eq "0" ]; then
go back to last SUCC and count if last ABEND is a multiple of 10
fi
tail -1 | grep &0:05 inputfile
if [ "$?" -eq "0" ]; then
go back to last SUCC and count if last ABEND is a multiple of 5
fi


Something along those lines...
this should do it for you

success=0;
num_abend=0

while read line
do
        is_succes=`echo $line | grep SUCC`
        is_success=$?
        if [ $is_success != 0 ]; then
        {
                num_abend=`expr $num_abend + 1`
                if [ $num_abend == 5 ]; then
                        snippet=`echo $line | awk '{ print $10 }'`
                        if [ "$snippet" !=  "" ];then
                        {
                                echo "ALERT GOES HERE"
                                num_abend=0
                        }
                        fi
                fi
        }
        else
                eum_abend=0
        fi
done < ./abend.txt
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
An awk solution below.

Please note that I calculated the number of consecutive ABENDs to look for by dividing 20 by the interval. Adjust as needed.
Replace "echo TAKE ACTION!" in the system() call with whatever action you'd like to start.

wmp
#!/bin/sh
IN=/path/to/inputfile
INTERVAL=$(tail -1 $IN | awk -F: '{print $NF}')
ACHUNK=$((20/$INTERVAL)) # or whatever
echo "Interval is $INTERVAL, looking for $ACHUNK consecutive ABENDs in $IN"; echo
awk -v AC=$ACHUNK '{
                    if($0!~"ABEND") {C=0} 
                       else { C++ 
                              if (C%AC == 0) {print "Consecutive ABEND number", C, "found"; system("echo TAKE ACTION!")}
                            }
                   }' $IN

Open in new window

ooops typo

this should do it for you

success=0;
num_abend=0

while read line
do
        is_succes=`echo $line | grep SUCC`
        is_success=$?
        if [ $is_success != 0 ]; then
        {
                num_abend=`expr $num_abend + 1`
                if [ $num_abend == 5 ]; then
                        snippet=`echo $line | awk '{ print $10 }'`
                        if [ "$snippet" !=  "" ];then
                        {
                                echo "ALERT GOES HERE"
                                num_abend=0
                        }
                        fi
                fi
        }
        else
                num_abend=0
        fi
done < ./abend.txt
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
@Watnog - please note that my script will not give any output when run against the snippet you posted.
Because the interval there is 2 we would need at least 10 consecutive ABENDs to get an alert, and there are just five!

Author

Commented:
Thanks both.
I'll get back to you asap, let's say Monday...
:-D
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
TomuniqueSr. AIX Admin

Commented:
Watnog,
    What OS are you looking to run this on?  

Author

Commented:
@Tomunique: it's hpux.
@point_pleasant: I have come a long way only I don't need a loop (and I can't get rid of it)
@wmp: I put some conditionals in your code so it suits different scenario's, see snippet below, and that works fine!
#!/bin/ksh
IN=/path/to/inputfile
INTERVAL=$(tail -1 $IN | awk -F: '{print $NF}')
if [ $INTERVAL -eq 02 ]; then
ACHUNK=$((20/$INTERVAL)) 
echo "Interval is $INTERVAL, looking for $ACHUNK consecutive ABENDs in $IN"; echo
awk -v AC=$ACHUNK '{
                    if($0!~"ABEND") {C=0} 
                       else { C++ 
                              if (C%AC == 0) {print "Consecutive ABEND number", C, "found"; system("echo TAKE ACTION 02!")}
                            }
                   }' $IN 
elif [ $INTERVAL -eq 05 ]; then
ACHUNK=$((25/$INTERVAL)) 
echo "Interval is $INTERVAL, looking for $ACHUNK consecutive ABENDs in $IN"; echo
awk -v AC=$ACHUNK '{
                    if($0!~"ABEND") {C=0} 
                       else { C++ 
                              if (C%AC == 0) {print "Consecutive ABEND number", C, "found"; system("echo TAKE ACTION 05!")}
                            }
                   }' $IN 
elif [ $INTERVAL -eq 15 ]; then
ACHUNK=$((30/$INTERVAL)) 
echo "Interval is $INTERVAL, looking for $ACHUNK consecutive ABENDs in $IN"; echo
awk -v AC=$ACHUNK '{
                    if($0!~"ABEND") {C=0} 
                       else { C++ 
                              if (C%AC == 0) {print "Consecutive ABEND number", C, "found"; system("echo TAKE ACTION 15!")}
                            }
                   }' $IN 
fi

Open in new window



So as long as the repeat rate is < 60' (like 0:02, 0:05, 0:15, ...) no problem.
We do have repeats like 1:00 or even 6:00 hours however .

So I replaced  
INTERVAL=$(tail -1 $IN | awk -F:  '{print $NF}')    
with
INTERVAL=$(tail -1 $IN | awk -F\& '{print $NF}')       # the value is &0:02

So that 0:02 is returned instead of 02.
I do run into problems with the ACHUNK calculation now (of course).
How can I make the arithmetic work:

ACHUNK=$((30/$INTERVAL))             #  10/0:02: syntax error

Thanks!
TomuniqueSr. AIX Admin

Commented:
INTERVAL=$(tail -1 $IN | awk -F\& '{print $NF}')
if [ ! -z "$INTERVAL" ]
then
     HR=$(echo $INTERVAL | cut -f1 -d: )
    MIN=$(echo $INTERVAL | cut -f2 -d: )
   Tmp=$(expr $INTERVAL \* 60 )
   INTERVAL=$(expr $Tmp + $MIN )
fi

# Now use Interval as #Min  (2.5 Hrs = 150)

Author

Commented:
Something is not quite right it seems

INTERVAL=0:15
+ [ ! -z 0:15 ]
+ + echo 0:15
+ cut -f1 -d:
HR=0
+ + echo 0:15
+ cut -f2 -d:
MIN=15
+ + expr 0:15 * 60
expr: An integer value was expected.

Tmp=
+ + expr + 15
expr: Syntax error
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
OK,

let's make the calculation more straightforward:

INTERVAL=$(tail -1 $IN | awk -F\& '{print $NF}')
IFS=:; echo "$INTERVAL" | read hr min; INTERVAL=$(($hr*60+$min))


Then you could so:

case $INTERVAL in
        "2") ACHUNK=10;;
        "5") ACHUNK=5;;
       "15") ACHUNK=2;;
        *) echo "Undefined Interval!"; break;;
esac
echo "Interval is $INTERVAL, looking for $ACHUNK consecutive ABENDs in $IN"; echo

Put this at the beginning of the script. No need to repeat anything. Add more case criteria as needed.

CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013
Commented:
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION

Author

Commented:
Beautiful.
I have a timestamp in my filename like [filename.2011-05-23T02:00:00-tmp.txt]
Possibly awk stumbles over the [:] ?

 
awk: Cannot find or open file /path/to/filename.2011-05-23T02.
 The source line number is 6


The INTERVAL is correctly read and calculated.
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
My ksh and bash work correctly with such filenames. But since it's HPUX ...

Try this:

IN="/path/to/filename.2011-05-23T02:00:00-tmp.txt"
(with quotes) or even:
IN='/path/to/filename.2011-05-23T02:00:00-tmp.txt'
(with apostrophes).

Of course "/path/to/..." must be a real path available on your system.

Author

Commented:
Neither of them does it.
So I'm trying to replace the [:]  to [_]

I can use mv like this
mv $IN `echo $IN | sed 's/:/_/g'`

But I would like to cp instead, and I'm stuck there...
CERTIFIED EXPERT
Most Valuable Expert 2013
Top Expert 2013

Commented:
You could try to escape the colons:

IN=/path/to/filename.2011-05-23T02\:00\:00-tmp.txt
(no quotes or apostrophes).

What's the problem with cp?

NEWIN="$(echo $IN |sed 's/:/_/g')"
cp -p $IN $NEWIN
IN=$NEWIN



Author

Commented:
Thanks, got it.
You are great help and even better than that.
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.