Unix log monitoring

hi,

Is there any simple shell script/ awk script which can monitor a log file (maybe read new lines as they are logged) for a set of predefined errors  and lance an alert ?

I found a few tools but they appeared to require quite a bit of configuration.. I dont want something too sofisticated but something more efficient than a :

grep "memory exception"  server.log

thnx,

sg

sgauchoAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

adacunhaCommented:
something like following should help you (ksh is only used):

#!/bin/ksh

LogFile=<log file>
ErrorPattern=<error pattern, ex: @(string1|string2|foo*bar)
JobCmd=<cmd to run, ex: print |Mail -s alert root>


tail -f <log file> | while read;do
  eval [[ $REPLY = *$ErrorPattern* ]] && eval $JobCmd  #2nd 'eval' allows use of '|' and more into $JobCmd...
done

Dean.
0
ahoffmannCommented:
how about using perl with File::Tail
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
ppfoongCommented:

Well, if you don't find SWatch as complicated, you might want to take a look into it.

http://swatch.sourceforge.net/

0
Cloud Class® Course: SQL Server Core 2016

This course will introduce you to SQL Server Core 2016, as well as teach you about SSMS, data tools, installation, server configuration, using Management Studio, and writing and executing queries.

sgauchoAuthor Commented:
Hi,

Swatch requires perl and when executing it gave me some errors like "Warning: prerequisite Date::Calc failed to load:" which requires additional packages to be installed..

I tried using a counterpart of Swatch which is LogSurfer which also has other complications..  I have a team thats developing a customizes log analyzer but I wanted up and running a simple system instead for waiting another few weeks..
0
ppfoongCommented:

You need the following 5 perl modules for Swatch to run:

Time::HiRes
Date::Calc
Date::Format
File::Tail
Date::Parse


And you just need to run the following command to get them installed:

perl -MCPAN -e "shell"
install Time::HiRes
install Date::Calc
install Date::Format
install File::Tail
install Date::Parse
quit

0
sgauchoAuthor Commented:
Hi,

I had already tried that. The system is behind a firewall and cannot access the net to download the packages. Further, am not sure whether I need ROOT access (which I dont have) to the machine to install these packages..

thats the reason, I am out looking for something simple and efficient..

tnx
0
ppfoongCommented:

Then perhaps:

tail -f server.log | grep "memory exception"  > output.log &


0
tel2Commented:
adacunha's solution looks intriguing, but I can't get it to work.  Can anyone explain:
1. ErrorPattern=@(string1|string2|foo*bar)
Some kind of array?
2. *$ErrorPattern*
What's with the "*"s?
0
ahoffmannCommented:
> ErrorPattern=<error pattern, ex: @(string1|string2|foo*bar)
# use something like:

ErrorPattern='(string1|string2|foo*bar)'
tail -f <log file> | while read line;do
   egrep "$ErrorPattern" $line && mail -s"pattern match" you@some.where
done
0
sgauchoAuthor Commented:
hi,

I tried both adacunha's and ahoffmann´s solutions but still getting nowwhere.. I have the script montoring the logs but its not generating any kind of usefull info...

Basically, when I execute the script, I see a lot of output such as:

egrep: can't open at
egrep: can't open weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:971)
egrep: can't open at
egrep: can't open weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:402)
egrep: can't open at

and so on...

any suggestions ? thnx
0
adacunhaCommented:
Hi,

I may miss some "
Try this, it works:

#!/bin/ksh

LogFile=/tmp/foolog
ErrorPattern=@(failed|error)
JobCmd='print "$REPLY"|Mail -s alert root'


tail -f $LogFile | while read;do
  eval [[ "$REPLY" = *$ErrorPattern* ]] && eval "$JobCmd"
done
0
ahoffmannCommented:
> egrep: can't open at

opps, typed suggestion to fast, try:

   echo "$line" | egrep "$ErrorPattern" && mail -s"pattern match" you@some.where

BTW, for some egrep you probably want to add -q or -s option for this kind of test
0
adacunhaCommented:
one correction more:

#!/bin/ksh

LogFile=/tmp/foolog
ErrorPattern=@(failed|error)
JobCmd='print "$REPLY"|Mail -s alert root'


tail -f $LogFile | while read;do
  eval [[ "\$REPLY" = *$ErrorPattern* ]] && eval "$JobCmd"
done
0
sgauchoAuthor Commented:
hi,

adacunha´s soln seems to be working but  I am not receiving the alerts with the Subject.. why is that ?

My $JobCmd='print "$REPLY" | /bin/mail -s "PRD: SERVER LOG ALERT"  abc@test.com'

thnx,
0
adacunhaCommented:
I prefer use:
Mail -s "my subject" recipient@mydomain
0
ahoffmannCommented:
does your /bin/mail work at all to send you mails?
0
sgauchoAuthor Commented:
I  have to user /bin/mail and yes it works fine.. I am able to receive mails sent from the script except without the subject.

thnx
0
ahoffmannCommented:
depends on your version of /bin/mail, some won't allow a space betwee -s and the text for the subject, use:
  mail -s"your subject" ....
0
sgauchoAuthor Commented:
I got it.. Gotta use MAILX and it works..

Just one last question before I close this topic..

Can my ErrorPattern read like this:

ErrorPattern='(Out of memory Exception|LDAP threads|locked|Warning)'  ??

tnx
0
adacunhaCommented:
The korn shell pattern is:
ErrorPattern='@(Out of memory Exception|LDAP threads|locked|Warning)'

The equivalent extented regular expression is:
ErrorPattern='(Out of memory Exception|LDAP threads|locked|Warning)'

0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Unix OS

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.