Forwarding syslog messages to user mailbox

best thing is to read about syslogd and what it can do. Here some links from google:

http://www.linux-tutorial.info/modules.php?name=MContent&pageid=57
http://content.hccfl.edu/pollock/aunix2/logging.htm

The syslog daemon can write data into a file, send the information to another syslog server (central syslog repository) or as the example in your link shows when an event of a specific nature occurs pass the information to a script (bash in this case) which then generates an email.  IMHO, it would be better not to use email client mailx and actually generate the email message to the local MTA.

You might want to switch from syslog to using rsyslog which could be configured to check the events and generate email notifications.

rsyslog is an inplace replacement with additional features.
http://www.rsyslog.com/

If I were going to do this, I'd use a perl script to extract the messages I was interested in from the log file and mail that. Normal rotation is weekly, but you can change that to daily and invoke the script from /etc/logrotate.d/syslog.

You might be able to configure logging of critical messages to a named pipe and have some process picking up contents from there and mailing them. This will give far more immediacy than pretty much anything else. The process would have to re-open the pipe after every message (a shell script that repeatedly invokes cat perhaps) because it will see end-of-file every time syslog closes the pipe.
Never tried this myself...

I found this link which may solve the problem, we'll see: https://help.ubuntu.com/community/Logwatch
I've followed the instructions on the page and so far there's no evidence that it's working. I'll study the customization how to at the bottom of the page next...

It's only a daily digest update at best. Not OK if you want real-time emails about serious conditions. Is that what you wanted?

I ran logwatch manually and it works, thankfully. Also tested in the hourly cron so the daily cron (the default) should work.

I do get Cron Daemon emails periodically with this error:
/etc/cron.daily/logrotate:
[timestamp] [warn] NameVirtualHost internal-ip:80 has no VirtualHosts
due to a misconfiguration of apache which I haven't figured out how to fix yet.

Would like to see that for critical errors, you're right about the need for that. Let me review the above and ponder this, thanks.

Ubuntu uses rsyslog, as I have learned. For some reason I had the rsyslog.d configuration files but not the package, so I installed rsyslog and it's now running.

The default rsyslog config file contains this line for sending critical messages to mailbox:
local0.crit    |/etc/syslog.pipes/criticalMessages
which I enabled, but nothing happens except this:

root@orion:/etc/syslog.pipes# ls -l
total 0
prw------- 1 root root 0 2011-10-22 06:21 criticalMessages

criticalMessages is empty, either because nothing critical has happened since rsyslog was installed, or because it points to my mailbox (I haven't worked with fifo file types so I can't be sure).

In order to troubleshoot this, I added these lines to the config file:
local0.warn     |/etc/syslog.pipes/warningMessages
local0.info     |/etc/syslog.pipes/infoMessages

and restarted syslog. Unfortunately, there's no change to /etc/syslog.pipes, so nothing happened.

I looked for documentation under rsyslog.com - http://www.rsyslog.com/doc/rsyslog_conf_actions.html looked promising, but I'm still stuck.

I created the missing files under /etc/syslog.pipes. The default permissions are 644, but the permission that was set for criticalMessages was 600, so they're now all set to 600. Not sure if that will help...

The permissions thing is irrelevant. I prefer 644, because that way I don't need to become root in order to real log files.
Since criticalMessages  is a pipe, it will never contain anything. You must have a process reading from it at all times, otherwise any attempt to open it for writing will block, except a non-blocking open will return an error. To make sure that is not happening right now, try cat /etc/syslog.pipes/criticalMessages. If that hangs (because no process is waiting to write), interrupt it (control-C).
I would check first that logging is working as you expect before using pipes. Log to regular files for a bit and make sure messages are appearing.
You must have a separate process reading from each pipe - have you done that yet?

Thanks :) I ran cat /etc/syslog.pipes/criticalMessages criticalMessages and got all the entries I'm used to seeing in /var/log/syslog
I then disabled the warningMessages and infoMessages lines in the rsyslog config file, restarted the daemon and ran cat /etc/syslog.pipes/criticalMessages again: this time it hangs, and when I ran ls -l I saw that criticalMessages was empty.

I get the gist of how logging writes to a file. I just don't understand pipes, or piping to email.

Thanks very much!! The way that pipes work is much clearer to me now.

I added a bash script to /etc/rsyslog.d (with a '.conf' extension so that rsyslog would read it) like so:
#!/bin/bash
subject="Log Error Message"
while true; do cat /etc/syslog.pipes/criticalMessages | mail -s "$subject" me@myserver.com; done

and then ran
echo xxx >criticalMessages
to put something in the queue for rsyslog to process. Then
cat /var/log/syslog |grep criticalMessages
Oct 23 17:18:56 myserver rsyslogd-3000: unknown priority name "pipes/criticalMessages" [try http://www.rsyslog.com/e/3000 ]

The url was not helpful but there's a lot more documentation that comes with the rsyslog-doc package which I installed. I found this:
"Rsyslog provides a template-based system to specify the output formats. A template with the traditional syslog format looks like this:
$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
Rsyslog is looking for a priority name, so I looked up recording the priority: I'm told to add a PRI field to the template, which would then look like this:
$template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"

To write the new template to a file, I would need to change the syslog/rsyslog default traditional format (without PRI) from this:
*.* -/var/log/messages.log
to this:
*.* -/var/log/messages.log;TraditionalFormatWithPRI

The docs caution that I need to define the template before I use it in a selector line.  

With the new template, the output in /var/log/syslog would look like this:
syslog.info<46>: Jun 18 10:55:52 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2367"] exiting on signal 2.

with the <PRI> in brackets following the warning level. Before the warning level, sample entries are kernel, local7, user and syslog: these seem unrelated to my syslog entries, which show:
Timestamp, server name, application (postfix/smtpd, postfix/qmgr, postfix/local and pop3d-ssl) for example.  Here's a current syslog entry:
Oct 23 21:33:25 myserver pop3d-ssl: LOGOUT, user=myuser, ip=[::ffff:internal-ip], port=[44424], top=0, retr=0, rcvd=24, sent=96, time=0, stls=1

I could try adding a new template definition at the top of the default rsyslog config file and change the selector line accordingly, but I need to know whether this even makes sense, since I don't want to break syslog reporting.

There is no line in the sample rsyslog config file with:
local0.crit     |/etc/syslog.pipes/criticalMessages
so I'm now not sure where I got the idea to put it in there. The sample config contains a number of  template definitions which include database formats, but nothing addressing email notifications per se, so I do wonder if I'm looking to add this capability in the wrong place. I did find this:
# Templates are also fully supportd for forwarding:
*.*             @172.19.2.16;RFC3164fmt
# you can also execute a script. Let's assume, for example, you need
# to execute "turn-diesel-generator-on" when "power failed" is contained
# in a message... ;)
:msg, contains, "power failed"  ^turn-diesel-generator-on

Let me know your thoughts, thanks.

Hate those typos - /etc/syslog.cons is mistyped for /etc/syslog.conf

Thanks Duncan, again this was helpful! I don't know enough to answer your question directly but will report my status so far:

I restarted rsyslog.d with this entry (I stayed with local0 because I had used it before):
local0.info     |/path/to/myFifo

Then I ran the bash script, which stays open (and looks like it hangs):
while read -r x <myFifo;do echo $x|mail -s "Log Message" user@localhost;done

Then I ran this at the command prompt:
logger -p local0.info "Hello are you on the planet?" and another one

And yes, I do get Log Message emails asking me if I'm on the planet etc. But leaving a bash script running??
I can terminate the bash script with Ctrl + C, but then I get no email, as expected. A script will terminate by default once I log out, which is not very helpful - for it to run all the time, doesn't it have to be a daemon?

As an experiment, you can simply append an ampersand to the script

while read -r x <myFifo;do echo $x|mail -s "Log Message" user@localhost;done&

Select allOpen in new window

The script will run in the background and you can continue working in your shell and exit when done. The script will however exit if syslogd is restarted (actually I haven't tried that but perhaps you could - pkill -HUP rsyslogd should do it). You need to cater for this because logrotate usually restarts rsyslogd to make it close rotated-out files. A neat way to do it might be to have a respawn entry in /etc/inittab. That way, init will shut down the script when you halt the system, or go to single-user &c.

Thanks - the script runs in the background now. It executes from /etc/init.d and terminates when rsyslog is restarted - I added a link from /etc/rc2.d so it should restart automatically on startup. When I restart rsyslog, I also restart the bash script manually.

My problem now is with the facility: with local0 or local3, no matter what the priority, the only emails I receive are from manually issued logger -p local3.info "my sample string" commands.

In the "Logging for the mail system" section of the rsyslog config file, I commented out
#mail.warn                       -/var/log/mail.warn  to pipe output to my mailbox:          
mail.warn                      |/path/to/myFifo

I don't get enough warning messages to get any email, so I tried the same process with mail.info and got 699 messages, each containing the same server timestamp, but with different pid numbers and processes. None of those show up in /var/log/mail.log as expected. Also, normal log entries show timestamps that change once a pid reaches the next second on the clock.

Rsyslog should be able to write messages to more than one facility at the same time, but I haven't figured out how to make it work. I'm surprised that the documentation is so quiet about this - I thought this would be a common requirement, provided in scripts found in standard distributions.

This is genius, thank you!! It works, so to wrap things up, I disabled the above and added this line at the top of the default config file:
local0.*     |/path/to/myFifo
Then just above this line (for emergency files sent to everybody logged in):
*.emerg                         *
I added this line:
*.crit;*.emerg;local0.*        |/path/to/myFifo

I restarted rsyslog and the bash script but it could be a while before I see a warning in my mailbox - so let me know if I've got this right, thanks.
:)

The trouble with starting from init.d is that your script stops on rsyslogd reset as you have found. That is why I recommended starting from inittab. Specify the levels at which you want to run and specify respawn. Remove any reference to it from init.d

Thanks very much, everything seems to be working :)

There's no inittab running in the default version of Ubuntu (10.04). Links to programs exist under the appropriate /etc/rcX.d/  directory, with X representing run levels 0 - 6

I don't know how to respawn a run level under these circumstances but I did find http://manpages.ubuntu.com/manpages/hardy/man5/inittab.5.html 
which talks about respawing - will have to study this.

You might have the new-fangled replacement for /etc/inittab - can't remember what it's called. man init may tell you

Let me understand how this works. As I mentioned earlier, I have an instruction to start the bash mail script on boot, just after the syslog daemon:

# ls -l /etc/rc2.d/

lrwxrwxrwx 1 root root  18 2009-03-10 01:46 S10sysklogd -> ../init.d/sysklogd
lrwxrwxrwx 1 root root  15 2009-03-10 01:46 S11klogd -> ../init.d/klogd
lrwxrwxrwx 1 root root  19 2011-10-25 10:03 S12my_script.sh -> ../init.d/my_script.sh

By default, there's no link in here to rsyslog (which is where I have the instruction to pipe critical messages to email) so a link here to my bash script may be irrelevant or not good - another reason to remove it.

The rsyslog daemon starts up differently from the syslog daemon: I copy /etc/init/rsyslog.conf below.

So what you're advising me to do (I think) is to remove the link to the bash script in /etc/rc2.d because it is not helpful, and instead run a script under /etc/init that mirrors the way the rsyslog daemon is started. I took a stab at it, and attach that too.

/etc/init# cat rsyslog.conf
# rsyslog - system logging daemon
#
# rsyslog is an enhanced multi-threaded replacement for the traditional
# syslog daemon, logging messages from applications

description	"system logging daemon"

start on filesystem
stop on runlevel [06]

expect fork
respawn

script
    . /etc/default/rsyslog
    exec rsyslogd $RSYSLOGD_OPTIONS
end script

Select allOpen in new window

# config file for bash script in /etc/init/

# script_name.sh - launch after rsyslog daemon
#
description	"pipe critical / warning messages to mailbox"

start on runlevel [2345]
stop on runlevel [06]

expect fork
respawn

script
    . /etc/init.d/script_name.sh
    
end script

Select allOpen in new window

Looks reasonable to me, but  I have no experience of this new init system. Certainly get rid of the rc2.d link. Personally I think I would move /etc/init.d/script_name.sh to /etc/default/ because script_name.sh has no connection to the other rc scripts.
I imagine /etc/default/rsyslog sets up environment variables only, in preparation for rsyslogd - is that correct? Otherwise I'm not understanding how this init works.
Also, what does expect fork mean? The init man page should tell you, but my guess is that it tells init that rsyslogd will fork itself into background, and init should monitor that background process for respawning. A matching script might be

script
    exec /etc/default/script_name.sh &   
end script

Select allOpen in new window

Or it might be that init will fork the process itself if you omit expect fork, in which case you would also omit &

I've done as much research on these questions as I trust myself to do, given that this is all somewhat above my head.

/etc/default/rsyslog contains only this line:
RSYSLOGD_OPTIONS="-c4"
where 'c' is the compatibility mode and '4' disables inet domain sockets to limit risk to the local machine. So yes, this does I believe set up environment variables only.

I found that a relatively small fraction of config files in /etc/init/ specify any type of fork, and most of these appear to be daemons. Therefore I think it safe to omit the 'expect fork' line.

Putting together the above, I plan to move the bash script to /etc/default/, remove the '&' from the bash script and write /etc/init/mail-notify.conf with these lines:

start on runlevel [2345]
stop on runlevel [06]

respawn

script
    exec /etc/default/script_name.sh &   
end script

I'll try this and let you know how it goes.

I would point out though that /etc/init.d/ contains mostly executable scripts and some links to executables, although only one of the scripts is a bash script (umountnfs.sh).

All files under /etc/default/ have rw-r-r / 644 permissions and no filename extensions,  so putting an executable (bash) script there looks odd...

I made no change to rsyslog but restarted it and then found that the bash script had stopped as expected.  If I start it manually without the '&' the script will hang, so presumably /etc/init/mail-notify.conf will start/respawn the bash script automatically. I have no idea how long that process will take, if it does at all, because I don't quite understand what triggers a respawn or how long it takes for /etc/init to read those instructions.

I restored the bash script to the /etc/init.d/ path for reasons explained above, and modified mail-notify.conf accordingly. Will check on the status tomorrow.

If I start it manually without the '&' the script will hang no - it will run but not return. You are right, most processes are started from /etc/init.d/. But processes that need to be respawned are not so started - they are started by init directly (init is process 1). You should find that the getty processes are started by init - from some file in /etc/init.
After adding / modifying files in /etc/init, you may have to tell init to re-read them. In the old days, you issued the command telinit q - that may still work. Or kill -HUP 1 may do it.

If respawn is going to happen, it is instantaneous. No point waiting. I would not expect /etc/init/mail-notify.conf to do it for you.

I'm guessing from https://help.ubuntu.com/community/UbuntuBootupHowto that Ubuntu expects new services to use the Upstart method, which I'm studying. It appears that environment variables for the new service should be specified under /etc/default if they apply, but only some services require that.

The /etc/init/*.conf file may contain (bash) script instructions also, in addition to linking to executable files. For example, mysql.conf has a pre-start script, exec /usr/sbin/mysqld, and a post-start script. There is a symbolic link /etc/init.d/mysql pointing to /lib/init/upstart-job. This allows the daemon to restart with 'service mysql restart' .

I've tried to mimic that behavior for mail-notify, but so far I get only this:
# service mail-notify start
mail-notify start/running, process 3329

which looks nice but there is no such pid and mail-notify is not running. I see this in the terminal only because I created a symbolic link /etc/init.d/mail-notify pointing to /lib/init/upstart-job just to see what would happen.

I've learned a lot so far (I think) from this question and I may close it just to end the guessing, but not today.

Yes they may say all that but this looks to me like something you have to start from inittab or, in your case, inittab's replacement.
You could avoid the closure of your process on rsyslogd restart by prepending to your existing script

while true; do

Select allOpen in new window

and appending

done

Select allOpen in new window

but then you would have the problem of how to shut it down. Perhaps it would have to write a pid file then stop would know which process to kill. By the way there is a convention with many rc scripts that they accept start, stop and restart as arguments (and generally don't work without one of them).

Hang on, I found something:
# cat /var/log/syslog |grep mail-notify
Oct 31 12:47:59 orion init: mail-notify main process ended, respawning
Oct 31 12:47:59 orion init: last message repeated 9 times
Oct 31 12:47:59 orion init: mail-notify respawning too fast, stopped

So I tried different options with the "respawn" line and the "&" settings, and came up with something that works, which I post below.

I tried including the bash instruction /etc/init/mail-notify.conf instead of calling "exec /usr/bin/mail-notify", but on restart the mail-notify pid is still associated with /usr/bin/mail-notify.

So ubuntu looks to the /usr/bin/ directory by default (the upstart process requires that the same filename be used for every directory where it appears). To restart mail-notify, I use  "service mail-notify restart"

Interestingly, I restarted rsyslog and ran ps -ef |grep mail-notify and saw that the bash script was still running. Presumably, the respawn instruction in mail-notify.conf takes care of that :)

I will wait for your comments to close this out. Thanks very much for your patience!

# /etc/init/mail-notify.conf

start on runlevel [2345]
stop on runlevel [06]

respawn

script

user@localhost;done
exec /usr/bin/mail-notify     

end script

# /usr/bin/mail-notify

#!/bin/bash

FILE="/etc/syslog.pipes/mail-notify"

while read -r x <$FILE;do echo $x|mail -s "Log Message" user@localhost;done

Select allOpen in new window

Looks good. You might like to check that the pid of the bash script changes on rsyslogd restart - it should.

Yes it does, thanks :)

Learned a lot about the init process!!