Search for selected keywords and its values in multiple log files

Hi,

I must admit that I have extreme difficulties understanding your script.

So I'll take the few parts that make sense to me to create another version.
Please note: In the "split" function you're specifying "SASFoundation" to later
search "$1" for that string. Unfortunately, file2 has "x= /SASFoundation" so that the
string to match is in "$2" (where it actually belongs for later processing) instead of "$1".
I'll thus change the "split" arguments to contain "x" instead of "SASFoundation".
I assume this will have to be changed later, but since I don't know the original logfile layout that's all I can do now.
The complete "if" construct in the END section is incomprehensible to me, so I'll simplify it to
"if(K[1] || K[2]>1.0 || K[3])". I don't know whether this will meet all your needs, but it seems to work.
I'll have to add it in front of both "pr()" calls. By the way, the "NR>1" condition in that function is useless.
Finally, I'll create two FORMAT strings for "printf", to make the statements better readable.

awk -F '[=:;.]' '
  function pr() {printf FORMAT, K[1],K[2],K[3],K[0]}
  BEGIN {FORMAT="%s\t%s\t%16s\t%s\n"
      printf FORMAT, "MEMSIZE","SECOND","SASEXE","Filename"
      for(i=split("memsize ,Real Time ,x",A,",");i;i--) L[A[i]]=i
      FORMAT="%s\t%.1f\t%16s\t%s\n"
  }
  FNR==1 {
      if(K[1] || K[2]>1.0 || K[3]) pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{if(K[1] || K[2]>1.0 || K[3]) pr()}' *.log > progflag.csv

Select allOpen in new window

The "mailx" procedure (apart from the typos) will send the output file as the mail body, not as an attachment.
Most "mailx" implementations cannot work with attachments, thus do not understand "-a".

But if your mailx understands "-a" (it might be linked to "nail") try this:

[ -s progflag.csv ] && mailx -s "subject text" -a  progflag.csv receiver@domain.com < "Programs flagged"

Select allOpen in new window

The text "Programs flagged" will make up the mail body.

wmp

Several error messages when  I ran the code which is as follows:

1) this is the error  message:

      1:  not found
      NOTE::  not found.
      syntax error at line 5: '(' not expected.
      0402-016 Cannot find or open the files.

2) mailx  error  message: syntax error at line 5: '(' not expected.    0402-016 Cannot find or open the files.
[ -s progflag.csv ] && mailx -s "subject text" -a  progflag.csv receiver@domain.com < "Programs flagged"

3) I removed  *.log |   and [ -s progflag.csv ] && mailx -s "subject text" -a  progflag.csv
    receiver@domain.com < "Programs flagged"

   I rerun the code.  The output in progflag.csv consisted of column headers and the column rows      
    were  blank:

    MEMSIZE SECOND   SASEXE   FILENAME

Select allOpen in new window

 3)  The below is the output  result that needs to show in  progflag (there is no output in  progflag.csv for filen.log because in the file there is no memsize,  no sasfoundation and real time numeric value is less than 1.0  ):

MEMSIZE          SECOND                  SASEXE                           Filename
  200                                                                                             file1x.log
  100                                                SASFOUNDATION              file2x.log
  600                     6.0                      SASFOUNDATION              filet.log
  400                     8.07                    SASFOUNDATION              filew.log
  400                     5.1                                                                    filez.log

Here are some important information about the log files:

1)   In the log files,  there are multiple Real Time variables and numeric
      values. Real Time and its value that are read in and out of the
      program come after  the statement NOTE: DATA statement used
      (Total process time);
   
      NOTE: DATA statement used (Total process time);
      real time     0.06  seconds

2) In some of the log files,  real time variable  has ':'  and in some log files,  real time doesn't have the colon  symbol.

3) In some of the log files,  real time numeric value is formatted with two digits
real time     0.6  seconds

In some of the log files,  real  time value is formatted with three digits,
real time     0.07  seconds

#!/bin/bash
cd  /s/log

*.log |awk -F '[=:'';.]' '
  function pr() {printf FORMAT, K[1],K[2],K[3],K[0]}
  BEGIN {FORMAT="%s\t%s\t%16s\t%s\n"
      printf FORMAT, "MEMSIZE","SECOND","SASEXE","Filename"
      for(i=split("memsize ,Real Time ,x",A,",");i;i--) L[A[i]]=i
      FORMAT="%s\t%.1f\t%16s\t%s\n"
  }
  FNR==1 {
      if(K[1] || K[2]>1.0 || K[3]) pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{if(K[1] || K[2]>1.0 || K[2]>1.00||K[3]) pr()}' *.log > progflag.csv
[ -s progflag.csv ] && mailx -s "subject text" -a  progflag.csv receiver@domain.com < "Programs flagged"

Select allOpen in new window

file1x.log
file2x.log
filen.log
filet.log
filew.log
filez.log
progflag.csv.txt

Looking at the newly posted logfiles and the new requirements I must say, unfortunately,
that your script is by design not capable of handling this.

-- The files sometimes have "MEMSIZE=", sometimes "MEMSIZE =",
as you already mentioned sometimes there is "real time", but sometimes it's "real time:"
So in the case of MEMSIZE $1 is sometimes "MEMSIZE" and sometimes "MEMSIZE ",
in the case of "real time" $1 is sometimes "real time" but sometimes e.g. "real time   1"  
This makes a huge difference for the script, It is way too unflexible because it relies on a
unique keyword being present in $1 which is (see above!) not the case.

-- Because the script is just inspecting $1 for a keyword which also governs the output
a contextual processing is impossible, again "by design".  Looking first for "NOTE: DATA statement used"
to then take the following "real time" value into account cannot be implemented.

-- Please note: I don't think it would just require some hard work to handle these circumstances,
I think it's impossible by design!

I'd suggest developing a new script design which is more flexible in finding keywords and which can handle contextual matching.
Once you have such a concept and run into problems with the first code pieces you created please come back here and ask for assistance. I'll be ready for battle!
Should I find some time I'll experiment a bit for myself.

Sorry, no better news!

By the way, I see that you're on AIX. The "mailx" implementation of AIX cannot handle attachments, so you should either
- send the file as the message body, or
- use the "uuencode" method (doesn't work for all types of client software at the recipient side), or
- install the "mutt" RPM from http://www.perzl.org which has a lot of features including attachment sending.

wmp

I've requested that this question be closed as follows:

Accepted answer: 0 points for dfn48's comment #a41632382

for the following reason:

Thanks

cd /log/tmp/*.log | awk -F '[=:]' '
  function pr() {printf FORMAT, K[1],K[2],K[3],K[0]}
  BEGIN {FORMAT="%s\t%s\t%16s\t%s\n"
      printf FORMAT, "MEMSIZE","SECOND","SASEXE","Filename\n"
        for(i=split("/Memsize/ $2, ,/Real Time/ $2 ,/SASFoundation/ $3",A,",");i;i--) L[A[i]]=i
      FORMAT="%s\t%.1f\t%16s\t%s\n"
  }
  FNR==1 {
      if(K[1] || K[2]>'5:0:00' || K[3]) pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{if(K[1] || K[2]>'5:0:00' || K[3]) pr()}' *.log > progflag.txt

[ -s pflag.txt ] && mailx -s "subject text" -a  progflag.csv receiver@domain.com < "Code Need to be Evaluated"

Select allOpen in new window

no_SASFoundation_no_MEMSIZE.log
more_than_5_hr.log
SASFoundation_MEMSIZE.log
progflag.csv

What do you want to tell me?

I don't know why I'm not getting any data in the  progflag,csv.

The headers print but there isn't any column values

Thanks