log parsing

Hi

I will start to work on this script from now on.. so thought lets bug some  Gurus  to make it faster as always!!!!

basically , here I need some idea first, how will you parse the log to get few informations:

(a)Get the Error code as example bellow :403 Forbidden  ( which is under "F" section)
(b) Get the offender IP ~(xx.xx.xx.xx) which normaly comes on under "A"
(c) why its blocked from SEction "H"
(d) get the host name of offender ( i have some extra idea on this)
There is no rush.. long way to go... its just a start

my main purpose is to monitor  false positive .. as i am  one man army in this company!!,still  trying to use as good staff as i can to protect my network but its hard to go all the logs always.......and having some bad dream over it!!! so i need a script to monitor

thanks for  your help as always



 



--3f76833f-A--
[24/Mar/2010:13:03:47 +0000] S6oNs1nIjD4AAGWUf-IAAAAA xx.xx.xx.xx 30531 aa.bb.cc.dd 443
--3f76833f-B--
POST /ipn_main_handler.php HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Host: www.ourdomain.co.uk
Content-Length: 1279

--3f76833f-F--
HTTP/1.1 403 Forbidden
Content-Length: 222
Connection: close
Content-Type: text/html; charset=iso-8859-1

--3f76833f-H--
Message: Access denied with code 403 (phase 2). [file "/opt/modsecurity/etc/mod_rule/modsecurity_crs_21_protocol_anomalies.conf"] [line "46"] [id "960015"] [rev "2.0.5"] [msg "Re
quest Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"]
Action: Intercepted (phase 2)
Apache-Handler: application/x-httpd-php
Stopwatch: 1269435827266944 175040 (174501 174613 -)
Producer: ModSecurity for Apache/2.5.12 (http://www.modsecurity.org/).
Server: Apache/2.2.15 (Unix) mod_ssl/2.2.15 OpenSSL/0.9.8e-fips-rhel5 PHP/5.3.1

--3f76833f-Z--

--38ee3c5b-A--
[24/Mar/2010:13:56:45 +0000] S6oaHVnIjD4AAGdMI6EAAAAC xx.xx.xx.xx 46960 aa.bb.cc.dd 443
--38ee3c5b-B--
POST /ipn_main_handler.php HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Host: www.ourdomain.co.uk
Content-Length: 1273

--38ee3c5b-F--
HTTP/1.1 403 Forbidden
Content-Length: 222
Connection: close
Content-Type: text/html; charset=iso-8859-1

--38ee3c5b-H--
Message: Access denied with code 403 (phase 2). [file "/opt/modsecurity/etc/mod_rule/modsecurity_crs_21_protocol_anomalies.conf"] [line "46"] [id "960015"] [rev "2.0.5"] [msg "Re
quest Missing an Accept Header"] [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/MISSING_HEADER"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"]
Action: Intercepted (phase 2)
Apache-Handler: application/x-httpd-php
Stopwatch: 1269439005709693 171359 (170931 171028 -)
Producer: ModSecurity for Apache/2.5.12 (http://www.modsecurity.org/).
Server: Apache/2.2.15 (Unix) mod_ssl/2.2.15 OpenSSL/0.9.8e-fips-rhel5 PHP/5.3.1

--38ee3c5b-Z--

Open in new window

LVL 29
fosiul01Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

cjl7freelance for hireCommented:
Why don't you install splunk and then you can do lots more with your logs.

There is an open-source version (community) that will do all that you need and then some.

http://www.splunk.com/download?r=header

Much better way to get to know your systems!!!


//jonas
0
arnoldCommented:
Use perl, but then again....:)
Is this the format of the messages in the log file?
You can use awk/sed/cut etc. to deal with the multiline.
I think awk can have a begin/end block where you work on many lines a at a time.
Or you have to carry the ID of the event as a match mechanism.

i.e. 3f76833f is the transaction identifier and has subsection A-Z.
Section A contains the IP
Section F contains the error
Section H contains the reason for the error

at this time presuming you are catting the log into the perl script below to extract the sessionID and the subsection/subcategory.
#!/usr/bin/perl

while (<STDIN>) {
chomp();
if ( /^\-\-([0-9a-f]+)\-([A-Z]+)/i ) { # this should be the last check
    $sessionID=$1;
    $subsection=$2;
     print "Got the session ID: $sessionID Subsection: $subsection\n";
    next; #there is nothing more that can be done with this line.
}
#here you will have the logic to test based on the subcategory what information you want to extract
}

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
fosiul01Author Commented:
I am downloading splunk, but i have doubt that it can parse mod security log, but i will give it a try


abt the script :

-3f76833f-A-- B,F,H, Z is the most common, but there could be few others which is not that uses

but before A the value 3f76833f, it will be change for each transaction.

other then that the block should be same

though I am very new to perl still i am trying to understand the logic


sessionID=$1;
    $subsection=$2;

what you are trying to insert into this sessionID ??
0
arnoldCommented:
The items in the if statement that are surrounded by parenthesis () is a match reques and get store in the order

i.e. the first thing I am looking for the session ID that is made up of numbers and letters 0-9 and a-f the /i means that it does not matter what case is the letter.
[] means literal match.
The plus sign means one or more of the items
escaping the minus sign is to avoid its interpretation as an operand.

The sessionID will have 3f76833f if the line matches
The subsection will have an A,B,F,H,Z or any other Letter.

Many things depend on how and where you want the data.
One can use a the sessionID to build an Array of events or array of hases, or hashes of arrays.
perl is very versatile and fairly simple to use.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Shell Scripting

From novice to tech pro — start learning today.