asked on

Generate the nice report for Tomcat error log

I try to parse the tomcat error log to generate short error report which only show the unique error messages.
For example, the error messages in tomcat.error.log.1 look like:
===========================
$tomcat > grep -A1 ERROR tomcat.error.log.1
--
2008-09-20 02:33:14,937 [ERROR] http-8080-Processor21 -- Servlet.service() for servlet PixelServlet threw exception
java.lang.StackOverflowError
--
2008-09-20 02:33:46,583 [ERROR] http-8080-Processor23 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
--
2008-09-20 02:33:47,229 [ERROR] http-8080-Processor13 -- Exception in pixel logic:
java.lang.Exception: Token String not specified: t=s&p=0&ct=1: xx.xx.xx.xx
--
2008-09-20 02:33:47,547 [ERROR] http-8080-Processor15 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
--
2008-09-20 02:33:48,841 [ERROR] http-8080-Processor13 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
--
2008-09-20 02:33:50,258 [ERROR] http-8080-Processor17 -- Error in serving ClickTracking:
java.lang.Exception: invalid line item: nt=1&li=0&cr=0
--
2008-09-20 02:34:49,391 [ERROR] http-8080-Processor18 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
--
2008-09-20 02:34:50,281 [ERROR] http-8080-Processor18 -- Error in serving ClickTracking:
java.lang.Exception: invalid line item: nt=1&li=0&cr=0
--
2008-09-20 02:36:13,012 [ERROR] http-8080-Processor21 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
--
2008-09-20 02:36:15,091 [ERROR] http-8080-Processor21 -- Servlet.service() for servlet PixelServlet threw exception
java.lang.StackOverflowError
--
2008-09-20 02:38:47,032 [ERROR] http-8080-Processor16 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
--
2008-09-20 02:38:47,489 [ERROR] http-8080-Processor16 -- Exception in pixel logic:
java.lang.Exception: Token String not specified: t=s&p=0&ct=1: xx.xx.xx.xx
--
2008-09-20 02:38:49,159 [ERROR] http-8080-Processor16 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
--
2008-09-20 02:38:49,979 [ERROR] http-8080-Processor16 -- Error in serving ImpressionTracking:
java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified
===================================================

I would like to generate the report to show the unique error message like:
====================================================
2008-09-20 02:33:14,937 [ERROR] http-8080-Processor21 -- Servlet.service() for servlet PixelServlet threw exception -- java.lang.StackOverflowError -- Count (1)
.....
2008-09-20 02:33:46,583 [ERROR] http-8080-Processor23 -- Error in serving ImpressionTracking: -- java.lang.Exception: invalid line item: nt=1&li=&cr=0&sli=no_code_specified&pr=no_amount_specified -- Count (12)
.....
2008-09-20 02:33:47,229 [ERROR] http-8080-Processor13 -- Exception in pixel logic: -- java.lang.Exception: Token String not specified: t=s&p=0&ct=1: xx.xx.xx -- Count (2)
.....
2008-09-20 02:33:50,258 [ERROR] http-8080-Processor17 -- Error in serving ClickTracking: -- java.lang.Exception: invalid line item: nt=1&li=0&cr=0 -- Count (3)
=====================================================

The original log format have two lines for each error. Base on the error message on the first line, would like to parse those error messages to
1. show only one error messages for each type. Say "Error in serving ImpressionTracking" shows up in error log 12 times and I would like to show the first time only.
2. Combine those two lines into one line with separator "--"
3. Count how many repeat error messages for each type and put to the end of combine line with " -- Count (number of this type of error message)

This error log is fixed size because it have been rotated.

Perl or shell script are welcomed.

Adam314

Save this to a file. Call it passing it the name of the log. The output will be written to stdout. You can redirect this to a file. eg:
./tomcatunique.pl input.log >output.log

#!/usr/bin/perl
use strict;
use warnings;
 
 
my %errs;
my %errc;
local $/ = "--\n";
while(<>) {
	s/-*$//;
	
	next unless /(.*?)\s+--\s+(.*)[\r\n]+(.*)/m;
	
	$errc{$2}++;
	$errs{$2} = "$1 -- $2 -- $3" unless $errs{$2};
}
 
foreach (keys %errs) {
	print "$errs{$_} -- Count ($errc{$_})\n";
}

Open in new window

wesly_chen

ASKER

Hi Adam314,

Thanks for your script. Could you give me some explanation about your script so I can modify latter on?

ASKER CERTIFIED SOLUTION

Adam314

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

wesly_chen

ASKER

Thanks for the comments.
Please bear me as a Perl beginner, could you explain more on this line
----------
next unless /(.*?)\s+--\s+(.*)[\r\n]+(.*)/m;
----------
This line of code break the errs into $1, $2, and $3, right?
\s+-- newline followed by space followed by --
and ?

The original log is in size 10MB and I will test it today when I can access the original log.

Adam314

This is a regular expression. It will use the string in $_ unless another variable is specified - so in this case, it uses $_.

(.*?) This matches any characters except newline, the smallest amount possible, and saves it to $1
\s+ This matches 1 or more spaces, the most amount possible
-- This matches the literal characters dash dash
\s+ This matches 1 or more spaces, the most amount possible
(.*) This matches any characters except newline, the most amount possible, and saved it to $2
[\r\n]+ This matches the carriage return and linefeed characters, as many as possible
(.*) This matches any characters except newline, the most amount possible, and saved it to $3

If the string doesn't match this pattern that line (actually 3 lines ending with -- in this case) is skipped.

wesly_chen

ASKER

Thanks again for the explanation. I'm reading Perl book to digest those comment.

wesly_chen

ASKER

This solution resolve my issue with the explanation. Thanks.

wesly_chen

ASKER

Hi Adam314,

Could you help to modify this script so it show the last entry (not the first repeat entry)? Because, the most recent occurred error time make more sense to us.

I can create another question if you like.

Thanks

Adam314

On line 29, remove the unless part, so this:
$errs{$2} = "$1 -- $2 -- $3" unless $errs{$2};
becomes this:
$errs{$2} = "$1 -- $2 -- $3";

wesly_chen

ASKER

Thank you very much. You are the best.

wesly_chen

ASKER

Hi Adam314,

Based on this question, I need to show both the first time and last happen time of the same repeated error message.

I've created another question as
https://www.experts-exchange.com/questions/23773076/Parse-the-Tomcat-error-log-to-get-a-range-of-time-period-of-the-repeated-error.html

Could you help on it?

Thanks a log.

Wesly