Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.
Filter entries in /var/log/messages on CentOS
All,
I need to know if there is a way to filter entries from being logged in /var/log/messages. I see alot or repetitive, un-important things being logged that i don't need logged and it makes it hard to review the log file.
I tried to add an entry in /etc/syslog.conf, for example, snmpd entries,
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.
This stopped logging to the messages file altogether so i assume it broke it because when i removed the entry, things started being logged again.
I am running CentOS 5.5 64-bit so anyone with experience in this or RHEL may know how to do this.
I need to know if there is a way to filter entries from being logged in /var/log/messages. I see alot or repetitive, un-important things being logged that i don't need logged and it makes it hard to review the log file.
I tried to add an entry in /etc/syslog.conf, for example, snmpd entries,
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.
This stopped logging to the messages file altogether so i assume it broke it because when i removed the entry, things started being logged again.
I am running CentOS 5.5 64-bit so anyone with experience in this or RHEL may know how to do this.
Who is Participating?
"syslog" logging system has a limited number of message classes (mail, authpriv etc.), and 'snmp' isn't one of them.
you should probably use another system logging daemon, one that allows fine-grained filtering of messages (syslog-ng is one example).
you should probably use another system logging daemon, one that allows fine-grained filtering of messages (syslog-ng is one example).
Can you provide some more information for syslog-ng, it sounds like your taking a shot in the dark with your suggestion and im looking for a solid solution. I would also need to have the ability to filter other overly repetitive entries from future programs i would install that would clog up the messages log file with useless entries.
If anyone else has additional solid solutions, please let me know. As much specifics as possible please.
Thanks!
If anyone else has additional solid solutions, please let me know. As much specifics as possible please.
Thanks!
Unfortunately i cannot install any additional software so the solution i require is to be able to use what CentOS already has to effectively filter these selected messages.
So, again, if anyone else has additional solid solutions, please let me know. As much specifics as possible please
So, again, if anyone else has additional solid solutions, please let me know. As much specifics as possible please
You need to figure out what the class and level of the messages you want to filer out are. You can then redirect those messages. IIRC, info is not used much, so if I wanted to move messages to another file, I moved the info logging to another file and set that as the level at which that daemon logged.
You need to schedule a script to clean up the redundant entries periodically. For that you would need to know the entries that are unwanted and establish a pattern. For example if two consecutive lines are identical, I want only the latest one -- make sure that doesn't mess anything up. Once you figure that out, you can keep a cleanup script.
Second one is regular log rotation. I don't know you want that or not. It keeps records for last so many days.
Second one is regular log rotation. I don't know you want that or not. It keeps records for last so many days.
As i mentioned, the level or type of messages are to begin with snmp entries. I was hoping not to have to go the script route, but if someone knows of a script that cron can run that will clean up specific entries in messages, please let me know. I was hoping it was just an entry in /etc/syslog.conf that could do the job.
Is this not possible?
Is this not possible?
If it is mainly snmpd messages you are looking to suppress, then you can look at changing the way snmpd logs rather than changing the syslog configuration.
Look at the OPTIONS line in /etc/snmp/snmpd.options and change the "-Lsd" to "-LS 0-4 d"
(See: http://raetsel.wordpress.com/2008/02/15/snmpd-filling-up-varlogmessages/)
Look at the OPTIONS line in /etc/snmp/snmpd.options and change the "-Lsd" to "-LS 0-4 d"
(See: http://raetsel.wordpress.com/2008/02/15/snmpd-filling-up-varlogmessages/)
Thanks McCracky, but snmpd messages were just an example, as i said in prior posts, the solutions would have to be applicable to any numerous entries i would want to filter.
The problem is that you are trying to get syslog to do something that it wasn't meant to do. Think of it like a few notepads in the server room. The people who enter decide what to write in the logs. If one person is filling up the notepad, then tell that person to stop writing so much.
In standard syslog there are 16 facilities and 8 levels:
* The facility field can contain only 16 codes:
o kern Messages generated by the kernel.
o user Messages generated by user processes.
o mail The mail system.
o daemon System daemons, such as the in.ftpd and the telnetd daemons.
o auth The authorization system, including the login and su commands.
o syslog Messages generated internally by the syslogd daemon.
o lpr The line printer spooling system, such as the lpr and lpc commands.
o news Files reserved for the USENET network news system.
o uucp (obsolete) The UNIX-to-UNIX copy (UUCP) system does not use the syslog function.
o cron The cron and at facilities, including crontab, at, and cron.
o local0-7 Eight user-defined codes.
* the level selector specifies the severity or importance of the message. Each level includes all the levels above (of a higher severity).
o emerg 0 Panic conditions that are normally broadcast to all users
o alert 1 Conditions that should be corrected immediately, such as a corrupted system database. Only sysadmin of a particular server needs to be informed by mail or paged.
o crit 2 Warnings about critical conditions, such as hard device errors.
o err 3 Errors other than hard device errors
o warning 4 Warning messages, that generally does not interfere with normal operation.
o notice 5 Non-error conditions that might require special handling
o info 6 Purely informational messages (usually does not require any handling)
o debug 7 Messages that are normally used only when debugging a program
o none 8 Messages are not sent from the indicated facility to the selected file
That is what you have to work with. You need to tell the daemon (the person) where to write the messages (which notepad-controlled by syslog and the daemon) and how much to write (the daemon itself controls that).
Or I think rsyslog is available for CentOS. It might be better for what you want to do: http://www.rsyslog.com/doc/manual.html
http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story
http://openskill.info/infobox.php?ID=1475
http://aaronwalrath.wordpress.com/2010/09/02/set-up-rsyslog-and-loganalyzer-on-centos-linux-5-5-for-centralized-logging/
In standard syslog there are 16 facilities and 8 levels:
* The facility field can contain only 16 codes:
o kern Messages generated by the kernel.
o user Messages generated by user processes.
o mail The mail system.
o daemon System daemons, such as the in.ftpd and the telnetd daemons.
o auth The authorization system, including the login and su commands.
o syslog Messages generated internally by the syslogd daemon.
o lpr The line printer spooling system, such as the lpr and lpc commands.
o news Files reserved for the USENET network news system.
o uucp (obsolete) The UNIX-to-UNIX copy (UUCP) system does not use the syslog function.
o cron The cron and at facilities, including crontab, at, and cron.
o local0-7 Eight user-defined codes.
* the level selector specifies the severity or importance of the message. Each level includes all the levels above (of a higher severity).
o emerg 0 Panic conditions that are normally broadcast to all users
o alert 1 Conditions that should be corrected immediately, such as a corrupted system database. Only sysadmin of a particular server needs to be informed by mail or paged.
o crit 2 Warnings about critical conditions, such as hard device errors.
o err 3 Errors other than hard device errors
o warning 4 Warning messages, that generally does not interfere with normal operation.
o notice 5 Non-error conditions that might require special handling
o info 6 Purely informational messages (usually does not require any handling)
o debug 7 Messages that are normally used only when debugging a program
o none 8 Messages are not sent from the indicated facility to the selected file
That is what you have to work with. You need to tell the daemon (the person) where to write the messages (which notepad-controlled by syslog and the daemon) and how much to write (the daemon itself controls that).
Or I think rsyslog is available for CentOS. It might be better for what you want to do: http://www.rsyslog.com/doc/manual.html
http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story
http://openskill.info/infobox.php?ID=1475
http://aaronwalrath.wordpress.com/2010/09/02/set-up-rsyslog-and-loganalyzer-on-centos-linux-5-5-for-centralized-logging/
In your logout bash script .bash_logout, put a simple script to clean it up or make a cron job. Whats wrong with that
Actually, it can, but you have self-limited the constraining parameters.
syslog itself has ways to separate out things into different files, but you don't want to use those facilities with the constraints within syslog.
rsyslog or syslog-ng has been suggested, but you don't want to use those programs.
Bash scripts or log rotation have been suggested, but you don't want to use those.
Modifying the program's logging behavior has been suggested, but you don't want to do that.
So, what do you want to do?
syslog itself has ways to separate out things into different files, but you don't want to use those facilities with the constraints within syslog.
rsyslog or syslog-ng has been suggested, but you don't want to use those programs.
Bash scripts or log rotation have been suggested, but you don't want to use those.
Modifying the program's logging behavior has been suggested, but you don't want to do that.
So, what do you want to do?
According to the guidelines on the site, you shouldn't just delete something because it wasn't what you wanted. There where several solutions to the situation given.
Step 4: Assign Points
Points are your advertising; the more points you assign, the more likely you are to get a quick response. Points are not the reward you give for your answer. For example, if you ask how to do something, and an Expert tells you that you can't, that's the answer, and all of the points should be awarded, even if you don't like the answer.
(From: http://www.experts-exchange.com/help.jsp#hs=29&hi=396)
Comment 35097299 suggested a solution of syslog-ng
Comment 35112776 suggested reducing the logging from the problematic programs
Comment 35113840 showed the full array of options for syslog that were available.
Comment 35110662 suggested redefining the levels of the logging to help filter with syslog.
Comment 35110769 suggested a script to help filter the logs.
Split the points between the above.
Comment 35112776 suggested reducing the logging from the problematic programs
Comment 35113840 showed the full array of options for syslog that were available.
Comment 35110662 suggested redefining the levels of the logging to help filter with syslog.
Comment 35110769 suggested a script to help filter the logs.
Split the points between the above.
Mccracky,
When someone posts a question with specific guidelines that need to be followed to form a specific solution to particular problem and then all you have to suggest are anything but, that's not a solution. I already knew about syslog-ng, and everything else you suggested was ridiculous. You have to read what the person is specifically asking and if you know of a solution to help that problem, then say so, dont post random answers just to try and get points.
If its that easy, everyone would do it. Its like someone asking is anyone knows how to draw a circle and you tell them to draw a square instead because its a shape too.
When someone posts a question with specific guidelines that need to be followed to form a specific solution to particular problem and then all you have to suggest are anything but, that's not a solution. I already knew about syslog-ng, and everything else you suggested was ridiculous. You have to read what the person is specifically asking and if you know of a solution to help that problem, then say so, dont post random answers just to try and get points.
If its that easy, everyone would do it. Its like someone asking is anyone knows how to draw a circle and you tell them to draw a square instead because its a shape too.
The better analogy is a question asking how to fit a square peg in a round hole without changing anything on the peg or the hole.
With the contraints you put on the question, it isn't possible (as mentioned in comment 35113840).
You can reduce the logging the programs do (as I mentioned) which, in the analogy, shrinks the size of the square peg so that the corners still fit within the diameter of the round hole.
You can change the program to syslog-ng or rsyslog, in effect changing the hole to a square one.
You can write your own scripts to filter the logs, in effect filing the round hole into a square shape.
But, without changing something of the peg or the hole, it can't be done, as you finally acknowledged in comment 35245090.
It's like asking what 2 + 2 is but putting on the constraint that the answer can't be higher than 3.
That it can't be done is an acceptable answer according to the FAQ, too.
sjm
P.S. They weren't just "random answers" in order to get points. Everything mentioned was a viable solution to the problem.
With the contraints you put on the question, it isn't possible (as mentioned in comment 35113840).
You can reduce the logging the programs do (as I mentioned) which, in the analogy, shrinks the size of the square peg so that the corners still fit within the diameter of the round hole.
You can change the program to syslog-ng or rsyslog, in effect changing the hole to a square one.
You can write your own scripts to filter the logs, in effect filing the round hole into a square shape.
But, without changing something of the peg or the hole, it can't be done, as you finally acknowledged in comment 35245090.
It's like asking what 2 + 2 is but putting on the constraint that the answer can't be higher than 3.
That it can't be done is an acceptable answer according to the FAQ, too.
sjm
P.S. They weren't just "random answers" in order to get points. Everything mentioned was a viable solution to the problem.
Question has a verified solution.
Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.
Have a better answer? Share it in a comment.
All Courses
From novice to tech pro — start learning today.
Basically, instead of simple rules that syslog has, you can construct complex "filters" that discard messages from any number of unwanted programs.