Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 224
  • Last Modified:

Perl solution to find similar patterns in log files

I would like to have a script which captures similar patterns in a file. For example I would like to see in my web log if the request is coming from a certain IP address more than 10 times.  I like to make that no configurable and also run this script as a cron job.  My error log looks like follows
Mon May 01 13:48:11 CDT 2006
 ipaddr=111.111.11.90;path=/index.jsp;sessionid=MVU1DEQBFFS4VMD:

If the above pattern that includes the ip address occurs more than 10 times, I would like to be alerted through the output and also possibly send out a email (through unix mail feature). Essentially I need the script to do the following:
a. Able to take in a string to look for the similar patterns
b. Ability to change the threshold to alert when the similar patterns crosses that threshold on a given day
c. Ability to search by date

Note: I want to have the option of being alerted
a. Only when the session id's are different
b. Only when the session id's are same.
0
arunhem
Asked:
arunhem
  • 4
  • 3
1 Solution
 
Adam314Commented:
Have you got the script started?  Questions on a particular piece? Are you looking for someone to develop the entire script from scratch?
0
 
arunhemAuthor Commented:
Actually i am not very familiar with perl. I am writing a shell script and using awk to this. However it is little cumbersome as I first do a grep which takes some time for a large file and then using awk I get the ipaddr  and then I sort that and use uniq. This is quite cumbersome and I was told that Perl this could be done very elegantly.   If you could give me me a basic script with some pointers,  I could work with that and build on it. I would be happy to give you the points for such a script. Thanks.
0
 
Adam314Commented:
When you say pattern that includes the ip address, what pattern are you looking for.  Just the IP address?  

You can loop through a file and get diff fields like this:

open(IN,"<filename.log") or die "couldn't open filename.log: $!\n";
while($Line=<IN>){
($month,$mday,$hour,$min,$sec,$tz,$year,$ipaddr,$path,$sess) = $Line =~ m/\w+\s(\w+)\s(\d+)\s(\d+):(\d+):(\d+)\s(\w+)\s(\d+)\sipaddr=([\d.]+);path=([^;]+);sessionid=(\w+)/;

#count by IP address
$IPs{$ipaddr}++;
}
close(IN);

#check if any IP is more than thres
foreach $ip (keys $IPs){
  if($IPs{$ip} > 10){
    print "IP: $ip\n";
  }
}
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
arunhemAuthor Commented:
I am mainly looking for cases where IP addresses are same and also session id string is different. I guess one would have to modify that for loop.
0
 
arunhemAuthor Commented:
I changed ip to ipaddre here and ran the script:
#!/usr/bin/perl
open(IN,"<filename.log") or die "couldn't open filename.log: $!\n";
while($Line=<IN>){
($month,$mday,$hour,$min,$sec,$tz,$year,$ipaddr,$path,$sess) = $Line =~ m/\w+\s(
\w+)\s(\d+)\s(\d+):(\d+):(\d+)\s(\w+)\s(\d+)\sipaddr=([\d.]+);path=([^;]+);sessi
onid=(\w+)/;

#count by IP address
$IPs{$ipaddr}++;
}
close(IN);

#check if any IP is more than thres
foreach $ipaddr (keys $IPs){
  if($IPs{$ipaddr} > 10){
    print "IP: $ipaddr\n";
  }
}

It gives me the error:Type of arg 1 to keys must be hash (not scalar dereference) line 12
I changed it to:
#!/usr/bin/perl
open(IN,"<filename.log") or die "couldn't open filename.log: $!\n";
while($Line=<IN>){
($month,$mday,$hour,$min,$sec,$tz,$year,$ipaddr,$path,$sess) = $Line =~ m/\w+\s(
\w+)\s(\d+)\s(\d+):(\d+):(\d+)\s(\w+)\s(\d+)\sipaddr=([\d.]+);path=([^;]+);sessi
onid=(\w+)/;

#count by IP address
$IPs{$ipaddr}++;
}
close(IN);

#check if any IP is more than thres
foreach $ipaddr ( $IPs){
  if($IPs{$ipaddr} > 10){
    print "IP: $ipaddr\n";
  }
}


Is it correct to remove keys?
It coimpiles fine
0
 
arunhemAuthor Commented:
It does not give me the correct output though...
0
 
Adam314Commented:
This line
    foreach $ipaddr ( $IPs){
should be
    foreach $ipaddr (keys %IPs){

It doesn't check if session ids are the same or different, just counts number of times each IP address is listed.
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now