[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1529
  • Last Modified:

Filter entries in /var/log/messages on CentOS

All,

I need to know if there is a way to filter entries from being logged in /var/log/messages. I see alot or repetitive, un-important things being logged that i don't need logged and it makes it hard to review the log file.

I tried to add an entry in /etc/syslog.conf, for example, snmpd entries,

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;snmp.none;cron.none                /var/log/messages


This stopped logging to the messages file altogether so i assume it broke it because when i removed the entry, things started being logged again.

I am running CentOS 5.5 64-bit so anyone with experience in this or RHEL may know how to do this.

0
linuxpig
Asked:
linuxpig
  • 7
  • 6
  • 3
  • +1
5 Solutions
 
svsCommented:
"syslog" logging system has a limited number of message classes (mail, authpriv etc.), and 'snmp' isn't one of them.

you should probably use another system logging daemon, one that allows fine-grained filtering of messages (syslog-ng is one example).
0
 
linuxpigAuthor Commented:
Can you provide some more information for syslog-ng, it sounds like your taking a shot in the dark with your suggestion and im looking for a solid solution. I would also need to have the ability to filter other overly repetitive entries from future programs i would install that would clog up the messages log file with useless entries.

If anyone else has additional solid solutions, please let me know. As much specifics as possible please.

Thanks!
0
 
svsCommented:
There's a lot of information on syslog-ng here: http://www.balabit.com/support/documentation ("The syslog-ng Open Source Edition 3.1 Administrator Guide" in particular).

Basically, instead of simple rules that syslog has, you can construct complex "filters" that discard messages from any number of unwanted programs.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
linuxpigAuthor Commented:
Unfortunately i cannot install any additional software so the solution i require is to be able to use what CentOS already has to effectively filter these selected messages.

So, again, if anyone else has additional solid solutions, please let me know. As much specifics as possible please
0
 
mccrackyCommented:
You need to figure out what the class and level of the messages you want to filer out are.  You can then redirect those messages.  IIRC, info is not used much, so if I wanted to move messages to another file, I moved the info logging to another file and set that as the level at which that daemon logged.
0
 
farzanjCommented:
You need to schedule a script to clean up the redundant entries periodically.  For that you would need to know the entries that are unwanted and establish a pattern.  For example if two consecutive lines are identical, I want only the latest one -- make sure that doesn't mess anything up.  Once you figure that out, you can keep a cleanup script.

Second one is regular log rotation.  I don't know you want that or not.  It keeps records for last so many days.
0
 
linuxpigAuthor Commented:
As i mentioned, the level or type of messages are to begin with snmp entries. I was hoping not to have to go the script route, but if someone knows of a script that cron can run that will clean up specific entries in messages, please let me know. I was hoping it was just an entry in /etc/syslog.conf that could do the job.

Is this not possible?
0
 
mccrackyCommented:
If it is mainly snmpd messages you are looking to suppress, then you can look at changing the way snmpd logs rather than changing the syslog configuration.  

Look at the OPTIONS line in /etc/snmp/snmpd.options and change the "-Lsd" to "-LS 0-4 d"

(See: http://raetsel.wordpress.com/2008/02/15/snmpd-filling-up-varlogmessages/)

0
 
linuxpigAuthor Commented:
Thanks McCracky, but snmpd messages were just an example, as i said in prior posts, the solutions would have to be applicable to any numerous entries i would want to filter.
0
 
mccrackyCommented:
The problem is that you are trying to get syslog to do something that it wasn't meant to do.  Think of it like a few notepads in the server room.  The people who enter decide what to write in the logs.  If one person is filling up the notepad, then tell that person to stop writing so much.

In standard syslog there are 16 facilities and 8 levels:
    *  The facility field can contain only 16 codes:
          o kern Messages generated by the kernel.
          o user Messages generated by user processes.
          o mail The mail system.
          o daemon System daemons, such as the in.ftpd and the telnetd daemons.
          o auth The authorization system, including the login and su commands.
          o syslog Messages generated internally by the syslogd daemon.
          o lpr The line printer spooling system, such as the lpr and lpc commands.
          o news Files reserved for the USENET network news system.
          o uucp  (obsolete) The UNIX-to-UNIX copy (UUCP) system does not use the syslog function.
          o cron The cron and at facilities, including crontab, at, and cron.
          o local0-7 Eight user-defined codes.
    * the level selector specifies the severity or importance of the message. Each level includes all the levels above (of a higher severity).  
          o emerg 0 Panic conditions that are normally broadcast to all users
          o alert 1 Conditions that should be corrected immediately, such as a corrupted system database. Only sysadmin of a particular server needs to be informed by mail or paged.
          o crit 2 Warnings about critical conditions, such as hard device errors.
          o err 3 Errors other than hard device errors
          o warning 4 Warning messages, that generally does not interfere with normal operation.
          o notice 5 Non-error conditions that might require special handling
          o info 6 Purely informational messages (usually does not require any handling)
          o debug 7 Messages that are normally used only when debugging a program
          o none 8 Messages are not sent from the indicated facility to the selected file

That is what you have to work with.  You need to tell the daemon (the person) where to write the messages (which notepad-controlled by syslog and the daemon) and how much to write (the daemon itself controls that).

Or I think rsyslog is available for CentOS.  It might be better for what you want to do: http://www.rsyslog.com/doc/manual.html

http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story
http://openskill.info/infobox.php?ID=1475
http://aaronwalrath.wordpress.com/2010/09/02/set-up-rsyslog-and-loganalyzer-on-centos-linux-5-5-for-centralized-logging/
0
 
linuxpigAuthor Commented:
I guess this cant be done as i was looking for
0
 
farzanjCommented:
In your logout bash script .bash_logout, put a simple script to clean it up or make a cron job.  Whats wrong with that
0
 
mccrackyCommented:
Actually, it can, but you have self-limited the constraining parameters.  

syslog itself has ways to separate out things into different files, but you don't want to use those facilities with the constraints within syslog.  

rsyslog or syslog-ng has been suggested, but you don't want to use those programs.  

Bash scripts or log rotation have been suggested, but you don't want to use those.  

Modifying the program's logging behavior has been suggested, but you don't want to do that.

So, what do you want to do?
0
 
mccrackyCommented:

According to the guidelines on the site, you shouldn't just delete something because it wasn't what you wanted.  There where several solutions to the situation given.

Step 4: Assign Points
    Points are your advertising; the more points you assign, the more likely you are to get a quick response. Points are not the reward you give for your answer. For example, if you ask how to do something, and an Expert tells you that you can't, that's the answer, and all of the points should be awarded, even if you don't like the answer.

(From: http://www.experts-exchange.com/help.jsp#hs=29&hi=396)
0
 
mccrackyCommented:
Comment 35097299 suggested a solution of syslog-ng

Comment 35112776 suggested reducing the logging from the problematic programs

Comment 35113840 showed the full array of options for syslog that were available.

Comment 35110662 suggested redefining the levels of the logging to help filter with syslog.

Comment 35110769 suggested a script to help filter the logs.

Split the points between the above.
0
 
linuxpigAuthor Commented:
Mccracky,

When someone posts a question with specific guidelines that need to be followed to form a specific solution to particular problem and then all you have to suggest are anything but, that's not a solution. I already knew about syslog-ng, and everything else you suggested was ridiculous. You have to read what the person is specifically asking and if you know of a solution to help that problem, then say so, dont post random answers just to try and get points.

If its that easy, everyone would do it. Its like someone asking is anyone knows how to draw a circle and you tell them to draw a square instead because its a shape too.
0
 
mccrackyCommented:
The better analogy is a question asking how to fit a square peg in a round hole without changing anything on the peg or the hole.  

With the contraints you put on the question, it isn't possible (as mentioned in comment 35113840).

You can reduce the logging the programs do (as I mentioned) which, in the analogy, shrinks the size of the square peg so that the corners still fit within the diameter of the round hole.

You can change the program to syslog-ng or rsyslog, in effect changing the hole to a square one.

You can write your own scripts to filter the logs, in effect filing the round hole into a square shape.

But, without changing something of the peg or the hole, it can't be done, as you finally acknowledged in comment 35245090.

It's like asking what 2 + 2 is but putting on the constraint that the answer can't be higher than 3.

That it can't be done is an acceptable answer according to the FAQ, too.

sjm

P.S. They weren't just "random answers" in order to get points.  Everything mentioned was a viable solution to the problem.
0
 
farzanjCommented:
linuxpig:
>> everything else you suggested was ridiculous.

You have the right not to use any suggested solution, but you have NO right to call the precious effort and time of serious professionals as "ridiculous".
0

Featured Post

Vote for the Most Valuable Expert

It’s time to recognize experts that go above and beyond with helpful solutions and engagement on site. Choose from the top experts in the Hall of Fame or on the right rail of your favorite topic page. Look for the blue “Nominate” button on their profile to vote.

  • 7
  • 6
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now