Solved

script for e-mail notification of a matching syslog message

Posted on 2011-03-04
21
700 Views
Last Modified: 2012-05-11
Hi Team,

   I am looking for a simple bash script that can send me e-mail notification every time a matching text pattern in the syslog is found.

basically from a continuously running :  tail -f Syslog > some.program

then somewhere in some.program, there should be a text match string like:

 ¨Interface down¨

and a command like:
./mailx -s "Alert: Interface down" me@myemail.com </dev/null.

The message need not have any body, just a subject will be fine.

Finally, I want the script to exit gracefully after the FIRST grep match of the match string so that it doesn´t generate hundreds of e-mails everytime it seems the same message in the syslog.

Could someone assist me setting up such a script?

Thanks very much.



0
Comment
Question by:rleyba828
  • 8
  • 6
  • 4
  • +1
21 Comments
 

Author Comment

by:rleyba828
ID: 35036035
Incidentally....for the tail -f Syslog.... portion above, what happens if the log rotates?   Would the tail command fail because the file no longer exists and has become Syslog.1.gz or will it detect that a new Syslog file has been created?  Can the script I want to request also take into account this log rotate scenario, so that it will keep on tailing the syslog even though it has rotated?

thanks again
0
 
LVL 23

Expert Comment

by:savone
ID: 35036245
Here is your script:

#!/bin/bash
tac /var/log/messages | grep -m 1 "interface down" > /tmp/alert
mail -s "ALERT!" your.email@domain.com < /tmp/alert

Then set a cron job to run every five minutes or so like this

*/5 * * * * /path/to/script




0
 
LVL 23

Expert Comment

by:savone
ID: 35036252
You can set this to run every minute if you want, its not going to take up a lot of system resources.

0
 
LVL 8

Expert Comment

by:point_pleasant
ID: 35036533
i would also add a test to check that /tmp/alert is not zero size other wise you will get an email every five minutes wether the interface is down of not

some thing like

if [ -s /tmp/alert ]
then
     mail -s "ALERT!" your.email@domain.com < /tmp/alert
fi

0
 
LVL 23

Expert Comment

by:savone
ID: 35037087
Good idea!

here is the complete script:

#!/bin/bash
tac /var/log/messages | grep -m 1 "interface down" > /tmp/alert
if [ -s /tmp/alert ]
then
mail -s "ALERT!" your.email@domain.com < /tmp/alert
fi


0
 
LVL 8

Expert Comment

by:point_pleasant
ID: 35038615
Another thought, Need to avoid getting alerted for the same message ever 5 minutes.  This script saves the last message sent and compares it to the new one if they are identical no mail is sent.




#!/bin/bash
tac /var/log/messages | grep -m 1 "interface down" > /tmp/alert
if [ -f /tmp/sent_alert ]
then
        diff /tmp/alert /tmp/sent_alert > /dev/null 2>&1
        if [ $? -ne 0 ]
        then
                if [ -f /tmp/alert ]
                then
                        #New Mail to send
                        mail -s "ALERT!" your.email@domain.com < /tmp/alert
                fi
        fi
else  #First Time Being Run so No /tmp/sent_alert file yet
        if [ -f /tmp/alert ]
        then
                mail -s "ALERT!" your.email@domain.com < /tmp/alert
        fi

fi
cp /tmp/alert /tmp/sent_alert
0
 
LVL 11

Expert Comment

by:tel2
ID: 35041709
Looking at the script above, it looks to me as if this could happen:
  - The interface goes down.
  - An alert gets sent.
  - Someone then brings the interface up.
  - Then later it goes down again.
  - But this time, no message is sent, because /tmp/alert and /tmp/sent_alert are identical.
Unless the process of bringing the interface up removes /tmp/send_alert, of course.

Or have I missed something?
0
 
LVL 23

Expert Comment

by:savone
ID: 35041737
sometimes simpler is better....  I am not great at scripting, but how about this?

#!/bin/bash
# The next line puts the date and time into variable NEWEST
$NEWEST=`tac /var/log/messages | grep -m 1 "interface down" | awk {'print $1 $2 $3'}`
# The next line checks the date and time of the last alert
$CONTROL=`awk {'print $1 $2 $3'} /tmp/alert`
#If the times do not match then it means it is a new alert, lets send the email...  if not do nothing...
if [[ "$NEWEST` != "$CONTROL" ]]
then
tac /var/log/messages | grep -m 1 "interface down" > /tmp/alert
mail -s "ALERT!" your.email@domain.com < /tmp/alert
fi
0
 
LVL 23

Assisted Solution

by:savone
savone earned 100 total points
ID: 35041738
Fixing a syntax error:

#!/bin/bash
# The next line puts the date and time into variable NEWEST
$NEWEST=`tac /var/log/messages | grep -m 1 "interface down" | awk {'print $1 $2 $3'}`
# The next line checks the date and time of the last alert
$CONTROL=`awk {'print $1 $2 $3'} /tmp/alert`
#If the times do not match then it means it is a new alert, lets send the email...  if not do nothing...
if [[ "$NEWEST" != "$CONTROL" ]]
then
tac /var/log/messages | grep -m 1 "interface down" > /tmp/alert
mail -s "ALERT!" your.email@domain.com < /tmp/alert
fi
0
 
LVL 11

Expert Comment

by:tel2
ID: 35041745
Does the messages log contain a timestamp?  If so, you can ignore my previous post, cos /tmp/alert and /tmp/sent_alert won't match the 2nd time the interface goes down.
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 8

Expert Comment

by:point_pleasant
ID: 35043701
the syslog entries should have different time stamps so the diff would be different and email sent
0
 
LVL 23

Expert Comment

by:savone
ID: 35043731
Wouldn't this accomplish the same thing but with a lot less complication?



#!/bin/bash
# The next line puts the date and time into variable NEWEST
$NEWEST=`tac /var/log/messages | grep -m 1 "interface down" | awk {'print $1 $2 $3'}`
# The next line checks the date and time of the last alert
$CONTROL=`awk {'print $1 $2 $3'} /tmp/alert`
#If the times do not match then it means it is a new alert, lets send the email...  if not do nothing...
if [[ "$NEWEST" != "$CONTROL" ]]
then
tac /var/log/messages | grep -m 1 "interface down" > /tmp/alert
mail -s "ALERT!" your.email@domain.com < /tmp/alert
fi

Open in new window

0
 
LVL 8

Expert Comment

by:point_pleasant
ID: 35043826
this would fail because /tmp/alert would not get created,   the awk stmt in line five would fail because /tmp/alert does not exist awk would fail and the script would exit.  
0
 

Author Comment

by:rleyba828
ID: 35045850
Hi Team,

   Thanks a lot for all the advice...some important details though.

The machine I was referring to is our syslog server which aggregates all the syslogs of our various network devices.  Unfortunately,   I am not the sysadmin of the box so i cannot login as root   and I cannot use cron.  the tail -f Syslog works though.

I suppose I can to the same as tac /path/to/Syslog, however I don't know how to schedule the job (e.g. every 5 mins) as I can't access cron.

The script above of savone looks quite promising, if only I can trigger the running of this script automatically every 5 mins....like put it in a loop or something and then check system date/time.


Is there a way to modify the script above, such that it will run every 5 mins and parse the Syslog file only then?

Thanks again.
0
 
LVL 23

Expert Comment

by:savone
ID: 35046056
Short answer NO.  There is no way to run a scheduled job if you do not have access to cron.

I suggest you speak to your sysadmin and have his set this job for you.

0
 
LVL 11

Assisted Solution

by:tel2
tel2 earned 100 total points
ID: 35049354
Hi rleyba828,

My long answer is: maybe, I think.

You could either put it inside a loop, something like this:
    #!/bin/bash
    while true   # This test always succeeds
    do
      # <= Insert the main processing here
      sleep 300   # 300 secs = 5 mins
    done
Then kick that off with something like this:
    nohup ./the-script-name >the-script-name.out 2>&1&
The trailing "&" will make it run in the background, and the "nohup" will allow it to run even after you logoff.
The problem is, this will not automatically continue after a reboot (one advantage with cron).
Hopefully you have access to the "nohup" command.

Another option is the "at" command.  Do you have access to use it?  I've rarely used it, but you could probably do something like this:
- Don't have a loop in your code.
- Put an "at" command at the end of the script, to kick that script off again in 5 mins.  It might be something like:
    at -f ./the-script-name now + 5 minutes   # Not sure of the syntax.
Then run the script yourself, to get it started.
I'm not sure if this will survive a reboot.
Have a look at "man at" for more details.
But you might not have access to "at" either (I don't, so I can't easily test it).  To find out if you have access, just type "at" at the command line, and see if you get a permission error.

Are you absolutely sure you don't have access to cron?  You could type:
    crontab -e
to check.  Even non-root users can have their own crontab, if they've been setup.
0
 
LVL 8

Accepted Solution

by:
point_pleasant earned 300 total points
ID: 35056262
individual users should be able to set up their ow crontab entries.
also the script by savone will not work as explained in ID:35043826Author
this script does work and can be put in a user cron job or a 5 min sleep loop as suggested above.

#!/bin/bash
tac /var/log/messages | grep -m 1 "interface down" > /tmp/alert
if [ -f /tmp/sent_alert ]
then
        diff /tmp/alert /tmp/sent_alert > /dev/null 2>&1
        if [ $? -ne 0 ]
        then
                if [ -f /tmp/alert ]
                then
                        #New Mail to send
                        mail -s "ALERT!" your.email@domain.com < /tmp/alert
                fi
        fi
else  #First Time Being Run so No /tmp/sent_alert file yet
        if [ -f /tmp/alert ]
        then
                mail -s "ALERT!" your.email@domain.com < /tmp/alert
        fi

fi
cp /tmp/alert /tmp/sent_alert
0
 

Author Comment

by:rleyba828
ID: 35068473
Hi Team....

For user tel1,  I have tried the crontab -e already and it just says that "....you are not allowed to use this program...".   And the "at" command just says "killed".... But the "sleep" works though....so I can use this instead.

To everyone else,  thanks for all the help.

I have tried the script on my test machine and it is working as I need it to....I'll just do a few adjustments before running in on our production box.
0
 
LVL 8

Expert Comment

by:point_pleasant
ID: 35069126
please read the cron instructions below, you may need your administrator to adjust the cron.allow or cron.deny files.  Also make sure you are notified when your machine is rebooted because your script will not restart automatically as it would with cron.

Controlling Access to cron

Cron has a built in feature of allowing you to specify who may, and who
may not use it. It does this by the use of /etc/cron.allow and /etc/cron.deny
files. These files work the same way as the allow/deny files for other
daemons do. To stop a user using cron, just put their name in cron.deny, to
allow a user put their name in the cron.allow. If you wanted to prevent all
users from using cron, you could add the line ALL to the cron.deny file:

root@pingu # echo ALL >>/etc/cron.deny

If you want user cog to be able to use cron, you would add the line cog
to the cron.allow file:

root@pingu # echo cog >>/etc/cron.allow

If there is neither a cron.allow nor a cron.deny file, then the use of cron
is unrestricted (i.e. every user can use it).  If you were to put the name of
some users into the cron.allow file, without creating a cron.deny file, it
would have the same effect as creating a cron.deny file with ALL in it.
This means that any subsequent users that require cron access should be
put in to the cron.allow file.  
0
 

Author Comment

by:rleyba828
ID: 35082034
Thanks for the info above...Considering the strict security rules regarding granting privileges to non sysadmins, I doubt though it they would let me.  But I may broach the topic to them someday.   Thanks again.
0
 
LVL 23

Expert Comment

by:savone
ID: 35082670
There isnt much risk to allowing users access to cron jobs that I can think of.  Basically you are still restricted to what a user can do, only you would be allow to schedule things to happen in the background.

What script did you finally settle on?
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Over the years I've spent many an hour playing on hardened, DMZ'd servers, with only a sub-set of the usual GNU toy's to keep me company; frequently I've needed to save and send log or data extracts from these server back to my PC, or to others, and…
Background Still having to process all these year-end "csv" files received from all these sources (including Government entities), sometimes we have the need to examine the contents due to data error, etc... As a "Unix" shop, our only readily …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now