Link to home
Create AccountLog in
Avatar of ralph_rea
ralph_rea

asked on

HELP: shell script

My system administrator has given me this shell script (in attach: count_auth.sh) to make the access count to the various servers, but before run it I want to understand what it is.
On the path he showed me there are two types of files:
oblog.log.YYYYMMDD.xxxxxx.zzzzzz.gz
audit.log.YYYYMMDD.xxxxxx.zzzzzz.gz

Can anyone explain how this script work?
count-auth.sh
Avatar of simon3270
simon3270
Flag of United Kingdom of Great Britain and Northern Ireland image

The script looks in a single log file at a time, and requires 3 or 5 parameters.

It first sets up a search string - this is the YYYYMMDD date that you put on the command line, optionally within an inclusive range of hours in that day, e.g. if the 4th and 5th parameters on the line are 09 and 17, it will find all records between 09:00 and 17:59:59 on the specified day.  If you only have 3 parameters on the line, it will match any time on that day.

It then works out what sort of compression is used (gzip, bzip2 or none).  The correct decompression program is selected and stored in the $CAT variable.

The script then uses that decompression program to get the actual text in the file, removes any lines with OblixAnonymous or tstsSonda in them, looks for any line with AUTHN_ in the resulting lines, then looks in those AUTHN_ lines for a line matching the date and time you have selected, followed by whitespace, then any text, more whitespace, then the portal_name (the second parameter of the script.  It then counts those lines and sends the count to standard out.

So if you call the script as:
    auth.sh oblog.log.20121029.123456.abcdef.gz 20122029 PORTAL1 09 17
and the output might be
    7
Avatar of ralph_rea
ralph_rea

ASKER

Ok,
Thanks for the excellent explanation but you said that this script looks in a single log file at a time. Now I have to run this shell for entire month of October and I have over 30 files like these:

first:   -rw-r--r-- 1 xxx users   3410853 Oct  1 10:07 audit.log.20121001.080741.aaaaaaa.gz

-rw-r--r-- 1 xxx users   2741393 Oct  2 10:07 audit.log.20121002.080741.bbbbbbb.gz
....................................................................................
....................................................................................
....................................................................................
....................................................................................
....................................................................................

last:   -rw-r--r-- 1 xxx users   5831307 Oct 31 16:40 audit.log.20121031.154016.ccccccc.gz


how can I edit the file to enable the output of all the files at once?
for file in audit.log.201210*
do count_auth "$file" PORTAL_YOU_KNOW_WHAT_SHOULD_GO_THERE
done

assuming the script outputs a just the number, you can build the sum using

{
for file in audit.log.201210*
do count_auth "$file" PORTAL_YOU_KNOW_WHAT_SHOULD_GO_THERE
done | tr "\n" +
echo 0
} | bc

note that it could be nicer to modify the script so you can directly call it on multiple files and also that this does not handle errors
It's not trivial to change - because the parameters are defined by their position rather than by options (e.g. "portal" is the second parameter - an option-based one could have "-p PORTAL1"), it's not easy to have a variable-length list of files.

You*can* do it if you use a pattern, enclosed in double quotes,  to specify the files (e.g. "audit.log.*.gz"

Then change the file existence check to this (which will only check whether the first file exists, but that's enough if you are using wildcards!):

    if [ ! -f $(ls ${FILENAME} 2>/dev/null | awk '{print $1;exit}') ] ; then

The $CAT line can be left alone - it will search all of the files in one go.

With the above change, the you can still call it with a single file and it will work the same way.

One nice thing is that it works with multiple patterns, so you can have:
    auth.sh "audit*.gz oblog*.gz" PORTAL1 20121105
to search audit and oblog files in one go.  The drawback with this is that it won't check that the oblog*.gz files exist (it just checks that at least on audit*.gz file exists).
your mostly correct, except for this last part


The $CAT line can be left alone - it will search all of the files in one go.

the files may be of different formats. namely the last one is usually not compressed

you can replace the detection and cat command with this one

smartcat(){
  for file in "$@"
  do
    case `file -b $file | cut -d " " -f 1` in

      gzip) zcat "$file" ;;
      bzip2) bzcat "$file" ;;

      *) cat "$file" ;;

   esac || return 1
done
}
my O.S. Linux

for skullnobrains:

Can I create a new script? (e.g. call_count_auth.sh):

{
for file in audit.log.201210*
do count_auth "$file" MY_PORTAL1
done
}

./call_count_auth.sh >> count_access.log

correct?


for simon3270:

Which part of the file count_auth.sh I have to change?
Could you post the entire piece of code?
if you want an external script to build the sum, you need this

{
for file in audit.log.201210*
do count_auth "$file" MY_PORTAL1 | tail -n 1
done | tr "\n" +
echo 0
} | bc

Open in new window


if you want a script to display the count for each file, this should do

for file in audit.log.201210*
do echo -n "$file: "
count_auth "$file" MY_PORTAL1 |tail -n 1
done

Open in new window


if you want a modified script, this should do

#!/bin/bash



ME=${0##*/}



usage() {

        echo -ne "\nUSAGE: ${ME} <file_name> <portal_name> <YYYYMMDD> [ <from_hour> <to_hour> ]\n\n"

        echo -ne "\tfrom_hour : starting hour (in HH format)\n"

        echo -ne "\tto_hour   : ending hour   (in HH format)\n"

        exit 1

}



if [ $# -ne 3 ] && [ $# -ne 5 ] ; then

        usage

fi



FILENAME="$1"

PORTAL="$2"

DATE="$3"



[[ ${DATE} == [0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] ]] || usage



YEAR=${DATE:0:4}

MONTH=${DATE:4:2}

DAY=${DATE:6:2}



if [ $# -eq 5 ] ; then

        FROM=$4

        TO=$5

        [[ ${FROM} == [0-9][0-9] ]] || usage

        [[ ${TO} == [0-9][0-9] ]] || usage

        HOURS="(`seq -s '|' ${FROM} ${TO}`)"

else

        HOURS="[0-9][0-9]"

fi



smartcat(){
  for file in "$@"
  do
    case `file -b $file | cut -d " " -f 1` in

      gzip) zcat "$file" ;;
      bzip2) bzcat "$file" ;;

      *) cat "$file" ;;

   esac || return 1
done
} 

eval smartcat ${FILENAME} | grep -vE "(OblixAnonymous|tstSonda)" | grep " AUTHN_" | grep -E "${MONTH}\\\/${DAY}\\\/${YEAR}\s${HOURS}:[0-9][0-9]:[0-9][0-9]\s.*\s${PORTAL}" | wc -l

Open in new window


beware that error handling is not really good in this script, but you'll at least get a warning when a file does not exist. using bash as the interpreter, you'll also get a non-zero return code

note that @simon's proposal is totally workable as well (as long as all the files are compressed the same way)
the script:

for file in audit.log.201210*
do echo -n "$file: "
count_auth "$file" MY_PORTAL1 |tail -n 1
done

seems interesting, I need to write output to a excel file with:

file_name  count_access

If I create this script into new shell (e.g. call_count_auth.sh) How must I run it for to have output into log (or txt) file and then into excel (or csv) file?
this should do

either make the script executable or run it using "sh script_name"

#!/bin/sh

for file in audit.log.201210*
do echo -n "\"$file\";"
count_auth "$file" MY_PORTAL1 | tr -d '[[:space:]]' | tail -n 1
done 

Open in new window


this should produce an excel-compatible simili-csv file
Sorry, been away

Yes, @skullnobrain, you were right about the compression having to be consistent in my version (I noted that, but had forgotten by the time I wrote the caveats!).

One thing - what do the required lines in the file actually look like?  I think the backslashes are confusing the system - if the dates in the log files are simply like "11/05/2012", then remove the three backslahes before both of the forward slashes in the "smartcat" line, so it ends up as:

eval smartcat ${FILENAME} | grep -vE "(OblixAnonymous|tstSonda)" | grep " AUTHN_" | grep -E "${MONTH}/${DAY}/${YEAR}\s${HOURS}:[0-9][0-9]:[0-9][0-9]\s.*\s${PORTAL}" | wc -l

(I think they may have been there as part of a "sed" line, but left in by mistake).

The auth.sh script will just print out a single number, so a simpler usage for the excel version of the script would be
for file in audit.log.201210*
do echo "\"$file\";$(count_auth "$file" MY_PORTAL1 20121105)"
done

Open in new window

Below the steps that I performed

I connect with my personal user (user1):
cd
vi call_count_auth.sh

for file in /path....../audit.log.201210*.gz
do echo -n "\"$file\";"
count_auth "$file" MYPORTAL1 | tr -d '[[:space:]]' | tail -n 1
done

:wq

the files /path....../audit.log.201210*.gz are under another user  (user2 with password)

ls -ltr /path....../audit.log.201210*.gz

ls: /path....../audit.log.201210*.gz: Permission denied

Can I switch user into shell script call_count_auth.sh?

and finally How Can I run call_count_auth.sh to redirect on log (csv) file?
I solved the problem of changing user and I did this test on a single file:

for file in /path..../audit.log.20121031.xxxxxx.bbbbbb.gz
do echo -n "\"$file\";"
/path..../grepcount.sh "$file" "MYPORTAL1" 20121031 | tr -d '[[:space:]]' | tail -n 1
echo " "
done

It run correctly, the output is:
"/path..../audit.log.20121031.xxxxxx.bbbbbb.gz"; 900

if I don't pass date 20121031 in input I get error.
Now I need to run this script passing as date the month of October (201210*)

How can I modify this script?

Should I use "cut -d" ??

Can I switch user into shell script call_count_auth.sh?

yes, you can add sudo

easy way is to add this add the beginning of the file

! test `whoami` = root \
&& sudo - root sh $0 $* \
&& exit $?

Open in new window


which reads as
if i am not already root, run myself as root, and exit with the error of the previous command


and finally How Can I run call_count_auth.sh to redirect on log (csv) file?

sh call_count_auth.sh >> myfile.excel-csv


if I don't pass date 20121031 in input I get error.
Now I need to run this script passing as date the month of October (201210*)

not sure i understand what you want there but if you want to give the year and month as an argument, try this

! expr "$1" : '^[0-9]\{6\}$' \
&& echo "USAGE sh $0 YYYYmm" \
&& exit 1

for file in /path....../audit.log.$1*.gz
do echo -n "\"$file\";"
count_auth "$file" MYPORTAL1 | tr -d '[[:space:]]' | tail -n 1
done

Open in new window

SOLUTION
Avatar of simon3270
simon3270
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
Yes,
count_auth itself needs the following parameters:

file_name      MYPORTAL  YYYYMMDD

this script:

for file in /path..../audit.log.20121031.xxxxxx.bbbbbb.gz
do echo -n "\"$file\";"
/path..../grepcount.sh "$file" "MYPORTAL1" 20121031 | tr -d '[[:space:]]' | tail -n 1
echo " "
done

run correctly because this string:
/path..../grepcount.sh "$file" "MYPORTAL1" 20121031 | tr -d '[[:space:]]' | tail -n 1
has 3 parameter: "$file"     "MYPORTAL1"       20121031

I'd like to run the similar script but with date variable like this:

for file in /path..../audit.log.20121031*.gz
do echo -n "\"$file\";"
/path..../grepcount.sh "$file" "MYPORTAL1" 201210* | tr -d '[[:space:]]' | tail -n 1
echo " "
done

Any idea?
is correct this script?

! expr "$1" : '^[0-9]\{6\}$' \
&& echo "USAGE sh $0 201210" \
&& exit 1

for file in /path....../audit.log.$1*.gz
do echo -n "\"$file\";"
count_auth "$file" MYPORTAL1 $1* | tr -d '[[:space:]]' | tail -n 1
done
e.g.

echo /path..../audit.log.20121031.aaaaaa.bbbbbb.gz | cut -d'.' -f3

20121031
ASKER CERTIFIED SOLUTION
Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
I tried this script and run correctly:


for file in /path..../audit.log.201210*
do echo -n "$file;"
/path..../count_auth.sh "$file" "MYPORTAL1" $(echo $file | cut -d'.' -f3) | tr -d '[[:space:]]' | tail -n 1
echo ";MYPORTAL1"
echo " "
done

Now, it would be interesting to also handle errors.
we're far from the original question and i have no interest in doing 100% of your work, so i'll keep it simple and assume that count_auth handles it's errors properly

err=0
for file in /path..../audit.log.201210*
do echo -n "$file;"
/path..../count_auth.sh "$file" "MYPORTAL1" $(echo $file | cut -d'.' -f3) | tr -d '[[:space:]]' | tail -n 1 || err=$($err + 1)
echo ";MYPORTAL1"
echo " "
done
return $err

this is an example with basic error handling.
you may deal with other possible errors such as missing log files in a similar way
you may add error handling to the other script as well
you may want to abort the script on error : just stick an exit or continue in the error handling code

if you add error handling to your code and have trouble doing so, please post the code and a clear question ; i'll be glad to answer, but i believe that you are the one who should code, now
Sorry,
I don't want that you do 100% of my work, but I said "it would be interesting to also handle errors" not "you must error handling".

However, Thanks for your answer!
no problem, that was a bit harsh on my side
i probably was annoyed by a piece of posted code where the author complained i had forgotten a ';' or something similar

halt_on_errors=yes
err=0
for file in /path..../audit.log.201210*
do echo -n "$file;"
  ! this_count=$(/path..../count_auth.sh "$file" "MYPORTAL1" $(echo $file | cut -d'.' -f3))\
  && err=$($err + 1)\
  && test -n "$halt_on_errors" \
  && break
  echo "$this_count" | tr -d '[[:space:]]' | tail -n 1 
  echo ";MYPORTAL1"
  echo " "
done
return $err

Open in new window


i left the echo MYPORTAL as i assumed you added them purpoisely

note that we are still assuming that count_auth will handle its own errors, and still assuming that checking for non existent files is not an issue

also note that the modification i made to the error handling code is mandatory fo a posix-complient shell, but useless if you are using bash