Solved

Filter entries in /var/log/messages on CentOS

Posted on 2011-03-09
20
1,303 Views
Last Modified: 2012-05-11
All,

I need to know if there is a way to filter entries from being logged in /var/log/messages. I see alot or repetitive, un-important things being logged that i don't need logged and it makes it hard to review the log file.

I tried to add an entry in /etc/syslog.conf, for example, snmpd entries,

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;snmp.none;cron.none                /var/log/messages


This stopped logging to the messages file altogether so i assume it broke it because when i removed the entry, things started being logged again.

I am running CentOS 5.5 64-bit so anyone with experience in this or RHEL may know how to do this.

0
Comment
Question by:linuxpig
  • 7
  • 6
  • 3
  • +1
20 Comments
 
LVL 9

Expert Comment

by:svs
ID: 35086279
"syslog" logging system has a limited number of message classes (mail, authpriv etc.), and 'snmp' isn't one of them.

you should probably use another system logging daemon, one that allows fine-grained filtering of messages (syslog-ng is one example).
0
 

Author Comment

by:linuxpig
ID: 35087600
Can you provide some more information for syslog-ng, it sounds like your taking a shot in the dark with your suggestion and im looking for a solid solution. I would also need to have the ability to filter other overly repetitive entries from future programs i would install that would clog up the messages log file with useless entries.

If anyone else has additional solid solutions, please let me know. As much specifics as possible please.

Thanks!
0
 
LVL 9

Accepted Solution

by:
svs earned 100 total points
ID: 35097299
There's a lot of information on syslog-ng here: http://www.balabit.com/support/documentation ("The syslog-ng Open Source Edition 3.1 Administrator Guide" in particular).

Basically, instead of simple rules that syslog has, you can construct complex "filters" that discard messages from any number of unwanted programs.
0
 

Author Comment

by:linuxpig
ID: 35108835
Unfortunately i cannot install any additional software so the solution i require is to be able to use what CentOS already has to effectively filter these selected messages.

So, again, if anyone else has additional solid solutions, please let me know. As much specifics as possible please
0
 
LVL 12

Assisted Solution

by:mccracky
mccracky earned 300 total points
ID: 35110662
You need to figure out what the class and level of the messages you want to filer out are.  You can then redirect those messages.  IIRC, info is not used much, so if I wanted to move messages to another file, I moved the info logging to another file and set that as the level at which that daemon logged.
0
 
LVL 31

Assisted Solution

by:farzanj
farzanj earned 100 total points
ID: 35110769
You need to schedule a script to clean up the redundant entries periodically.  For that you would need to know the entries that are unwanted and establish a pattern.  For example if two consecutive lines are identical, I want only the latest one -- make sure that doesn't mess anything up.  Once you figure that out, you can keep a cleanup script.

Second one is regular log rotation.  I don't know you want that or not.  It keeps records for last so many days.
0
 

Author Comment

by:linuxpig
ID: 35112071
As i mentioned, the level or type of messages are to begin with snmp entries. I was hoping not to have to go the script route, but if someone knows of a script that cron can run that will clean up specific entries in messages, please let me know. I was hoping it was just an entry in /etc/syslog.conf that could do the job.

Is this not possible?
0
 
LVL 12

Assisted Solution

by:mccracky
mccracky earned 300 total points
ID: 35112776
If it is mainly snmpd messages you are looking to suppress, then you can look at changing the way snmpd logs rather than changing the syslog configuration.  

Look at the OPTIONS line in /etc/snmp/snmpd.options and change the "-Lsd" to "-LS 0-4 d"

(See: http://raetsel.wordpress.com/2008/02/15/snmpd-filling-up-varlogmessages/)

0
 

Author Comment

by:linuxpig
ID: 35113038
Thanks McCracky, but snmpd messages were just an example, as i said in prior posts, the solutions would have to be applicable to any numerous entries i would want to filter.
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 12

Assisted Solution

by:mccracky
mccracky earned 300 total points
ID: 35113840
The problem is that you are trying to get syslog to do something that it wasn't meant to do.  Think of it like a few notepads in the server room.  The people who enter decide what to write in the logs.  If one person is filling up the notepad, then tell that person to stop writing so much.

In standard syslog there are 16 facilities and 8 levels:
    *  The facility field can contain only 16 codes:
          o kern Messages generated by the kernel.
          o user Messages generated by user processes.
          o mail The mail system.
          o daemon System daemons, such as the in.ftpd and the telnetd daemons.
          o auth The authorization system, including the login and su commands.
          o syslog Messages generated internally by the syslogd daemon.
          o lpr The line printer spooling system, such as the lpr and lpc commands.
          o news Files reserved for the USENET network news system.
          o uucp  (obsolete) The UNIX-to-UNIX copy (UUCP) system does not use the syslog function.
          o cron The cron and at facilities, including crontab, at, and cron.
          o local0-7 Eight user-defined codes.
    * the level selector specifies the severity or importance of the message. Each level includes all the levels above (of a higher severity).  
          o emerg 0 Panic conditions that are normally broadcast to all users
          o alert 1 Conditions that should be corrected immediately, such as a corrupted system database. Only sysadmin of a particular server needs to be informed by mail or paged.
          o crit 2 Warnings about critical conditions, such as hard device errors.
          o err 3 Errors other than hard device errors
          o warning 4 Warning messages, that generally does not interfere with normal operation.
          o notice 5 Non-error conditions that might require special handling
          o info 6 Purely informational messages (usually does not require any handling)
          o debug 7 Messages that are normally used only when debugging a program
          o none 8 Messages are not sent from the indicated facility to the selected file

That is what you have to work with.  You need to tell the daemon (the person) where to write the messages (which notepad-controlled by syslog and the daemon) and how much to write (the daemon itself controls that).

Or I think rsyslog is available for CentOS.  It might be better for what you want to do: http://www.rsyslog.com/doc/manual.html

http://wiki.rsyslog.com/index.php/Rsyslog_on_CentOS_success_story
http://openskill.info/infobox.php?ID=1475
http://aaronwalrath.wordpress.com/2010/09/02/set-up-rsyslog-and-loganalyzer-on-centos-linux-5-5-for-centralized-logging/
0
 

Author Comment

by:linuxpig
ID: 35245090
I guess this cant be done as i was looking for
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35245151
In your logout bash script .bash_logout, put a simple script to clean it up or make a cron job.  Whats wrong with that
0
 
LVL 12

Expert Comment

by:mccracky
ID: 35245719
Actually, it can, but you have self-limited the constraining parameters.  

syslog itself has ways to separate out things into different files, but you don't want to use those facilities with the constraints within syslog.  

rsyslog or syslog-ng has been suggested, but you don't want to use those programs.  

Bash scripts or log rotation have been suggested, but you don't want to use those.  

Modifying the program's logging behavior has been suggested, but you don't want to do that.

So, what do you want to do?
0
 
LVL 12

Expert Comment

by:mccracky
ID: 35346739

According to the guidelines on the site, you shouldn't just delete something because it wasn't what you wanted.  There where several solutions to the situation given.

Step 4: Assign Points
    Points are your advertising; the more points you assign, the more likely you are to get a quick response. Points are not the reward you give for your answer. For example, if you ask how to do something, and an Expert tells you that you can't, that's the answer, and all of the points should be awarded, even if you don't like the answer.

(From: http://www.experts-exchange.com/help.jsp#hs=29&hi=396)
0
 
LVL 12

Expert Comment

by:mccracky
ID: 35363630
Comment 35097299 suggested a solution of syslog-ng

Comment 35112776 suggested reducing the logging from the problematic programs

Comment 35113840 showed the full array of options for syslog that were available.

Comment 35110662 suggested redefining the levels of the logging to help filter with syslog.

Comment 35110769 suggested a script to help filter the logs.

Split the points between the above.
0
 

Author Comment

by:linuxpig
ID: 35394893
Mccracky,

When someone posts a question with specific guidelines that need to be followed to form a specific solution to particular problem and then all you have to suggest are anything but, that's not a solution. I already knew about syslog-ng, and everything else you suggested was ridiculous. You have to read what the person is specifically asking and if you know of a solution to help that problem, then say so, dont post random answers just to try and get points.

If its that easy, everyone would do it. Its like someone asking is anyone knows how to draw a circle and you tell them to draw a square instead because its a shape too.
0
 
LVL 12

Expert Comment

by:mccracky
ID: 35395081
The better analogy is a question asking how to fit a square peg in a round hole without changing anything on the peg or the hole.  

With the contraints you put on the question, it isn't possible (as mentioned in comment 35113840).

You can reduce the logging the programs do (as I mentioned) which, in the analogy, shrinks the size of the square peg so that the corners still fit within the diameter of the round hole.

You can change the program to syslog-ng or rsyslog, in effect changing the hole to a square one.

You can write your own scripts to filter the logs, in effect filing the round hole into a square shape.

But, without changing something of the peg or the hole, it can't be done, as you finally acknowledged in comment 35245090.

It's like asking what 2 + 2 is but putting on the constraint that the answer can't be higher than 3.

That it can't be done is an acceptable answer according to the FAQ, too.

sjm

P.S. They weren't just "random answers" in order to get points.  Everything mentioned was a viable solution to the problem.
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35397039
linuxpig:
>> everything else you suggested was ridiculous.

You have the right not to use any suggested solution, but you have NO right to call the precious effort and time of serious professionals as "ridiculous".
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

This document is written for Red Hat Enterprise Linux AS release 4 and ORACLE 10g.  Earlier releases can be installed using this document as well however there are some additional steps for packages to be installed see Metalink. Disclaimer: I hav…
Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now