Solved

How to calculate between two given datetime in Perl

Posted on 2008-10-09
12
760 Views
Last Modified: 2013-11-17
Let me explain further:

I have this file as a result of a query about more than a thousand lines.  It has a date and time on each line entry and stored in a file called maillog that looks like the following:

Sep  5 00:00:30ns2 pop3-login: Login: jflanagan [192.168.30.20]....
Sep  5 01:00:02 ns2 pop3-login: Login: vismail [192.168.70.7]...
Sep  5 00:20:28 ns2 sendmail[16116]: m8570ItF016116: Milter add: header: ...
Sep  5 00:00:30 ns2 pop3-login: Login: atinbound [192.168.50.37]...
Sep  5 00:01:35 ns2 pop3-login: Login: pyen [192.168.10.87]....
Sep  5 00:00:36 ns2 sendmail[16143]: m8570Zu2016143: ......

My task is to capture all the data against the current local datetime.  That is, write a Perl script to capture only lines entries  that are less than 30 minutes old compared to  the current local time.   Let say, if I ran a Perl script at local time (e.g. "Sep 5 00:01:00), only get the entries within 30 minutes of the local time.  
How can I represent the date in the maillog file and compare it with the current date.  Now, you have to keep in mind that the datetime is a flat string format, a flat file does not have the same format as the current datetime.  That is, you can manipulate the localtime() into different formats, but how can you manipulate that date in the maillog so that I can compare the two dates (the localtime and the date in the maillog file)?   Can you give me some answers and detailed example script please!!???

0
Comment
Question by:grazal
  • 7
  • 5
12 Comments
 
LVL 39

Expert Comment

by:Adam314
ID: 22682820
You can use the Time::Local module to convert the file time into a unix time (epoch seconds).  Then compare to 30*60 (for 30 minutes).

#!/usr/bin/perl

use strict;

use warnings;

use Time::Local;

my %Months = (Jan => 0, Feb=>1, Mar=>2, Apr=>3, May=>4, Jun=>5, Jul=>6, Aug=>7, Sep=>8, Oct=>9, Nov=>11, Dec=>12);
 

my $year = (localtime())[7];

while(<>) {

	next unless /^(\w+)\s+(\d+)\s+(\d+):(\d+):(\d+)/;

	my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);

	next if time()-$linetime>30*60;

	print $_;

}

Open in new window

0
 

Author Comment

by:grazal
ID: 22689085
I copied the codes you gave me, but it gave me no output.  It should have had given me hundreds of lines of output, but in this case nothing came up!  Any suggestions please!

Here's a few excerpt from my input file "maillog":

Oct 15 04:02:56 ns2 pop3-login: Login: mlavarias [192.168.70.53]..
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: from=<dwvelvinm@velvin.com>, size=764, ..
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: Milter: data, discard...
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: discarded...
Oct 15 04:02:58 ns2 pop3-login: Login: pyen [192.168.10.87]...
Oct 15 04:03:01 ns2 sendmail[9686]: m95B31l9009686: mta-pva.cluster2.convio.net [66.45.103.48] did ...
Oct 15 04:03:02 ns2 sendmail[9635]: m95B2oof009635: ruleset=check_mail, .....

My perl script file contains:

#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
opendir(DIR, "/var/log");
open(READFILE,"/var/log/maillog");
my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6, Aug => 7, Sep => 8, Oct => 9, Nov => 11, Dec => 12);
my $year = (localtime()) [7];
while (<READFILE>)
{
   next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;
   my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
   next if time()-$linetime>30*60;
  print $_;
}
0
 
LVL 39

Expert Comment

by:Adam314
ID: 22689565
There were a few bugs in my previous post... try this:
#!/usr/bin/perl

use strict;

use warnings;

use Time::Local;

open(READFILE,"/var/log/maillog");

my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6, Aug => 7, Sep => 8, Oct => 9, Nov => 10, Dec => 11);

my $year = (localtime())[5];

while (<READFILE>)

{

    next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;

    my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);

    next if time()-$linetime>30*60;

    print $_;

}

close(READFILE);

Open in new window

0
 

Author Comment

by:grazal
ID: 22691021
Here's what happened:  I revised my script per your instruction.  The script ran, but no output.  So, what I did was I commented lines 11 and 13 just to troubleshoot.  
As a result I got the following Error:

Day '' out of range 1..31 at test2.pl line 12

Here's the revised script:
#!usr/bin/perl
use strict;
use warnings;
use Time::Local;
opendir(DIR, "/var/log");
open(READFILE,"/var/log/maillog");
my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6,
                         Aug => 7, Sep => 8, Oct => 9, Nov => 10, Dec => 11);
my $year = (localtime()) [5];
while (<READFILE>)
{
        # next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;
   my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
       # next if time()-$linetime>30*60;
  print $_;
}
close(READFILE);

I'm clueless!  
0
 
LVL 39

Expert Comment

by:Adam314
ID: 22691050
It looks like you commented lines 10 and 12 in the version I posted... line 10 is what parses the line from your file, getting the date from it.  Without that line, it won't work at all.  Line 12 is what skips the lines older than 30 minutes.  Without that, you will get all lines printed.

When I run it on your sample data, I get the output as expected.

If you get no output, it could be your file doesn't have any entries within the last 30 minutes.  If you post your actual file that you are using, I'll take a look.
0
 

Author Comment

by:grazal
ID: 22691157
To Adam314:

Please explain to me lines 11 thru 13 of the While loop above?  what does those numbers (e.g. $5, $4,..\W, \d+, etc. supposed to mean?  Sorry for my ignorance...but I'm so new to this!

Thanks
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:grazal
ID: 22691220
Here's some actual contents of the file "maillog".  I use the grep command to capture them, because the actual file contains hundreds of line input.  In fact, lines of input are added every second to this file.

 Oct 10 15:19:33 ns2 pop3-login: Login: maryp [192.168.80.31]
 Oct 10 15:19:33 ns2 pop3-login: Login: dku [192.168.193.124]
 Oct 10 15:19:34 ns2 pop3-login: Login: kmaclean [192.168.5.208]
 Oct 10 15:19:35 ns2 sendmail[19380]: m9AMJW7G019380: from=<>, size=2420, class=0, nrcpts=1, msgid=<01ce01c92b26$45cb1ae0$e905a8c0@LA46>, proto=ESMTP, daemon=MTA, relay=[192.168.5.233]
 Oct 10 15:19:35 ns2 pop3-login: Login: pyen [192.168.10.87]
 Oct 10 15:19:35 ns2 sendmail[19392]: m9AMJW7G019380: to=<maryp@coasteramer.com>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=32594, dsn=2.0.0, stat=Sent
 Oct 10 15:19:36 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:37 ns2 last message repeated 2 times
 Oct 10 15:19:37 ns2 pop3-login: Login: pmohabbat [192.168.70.28]
 Oct 10 15:19:37 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:39 ns2 last message repeated 3 times
 Oct 10 15:19:39 ns2 pop3-login: Login: vismail [192.168.70.7]
 Oct 10 15:19:39 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:40 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:40 ns2 pop3-login: Login: srosenthal [192.168.30.54]
 Oct 10 15:19:40 ns2 sendmail[19287]: m9AMJex9019287: [87.126.124.96] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA
 Oct 10 15:19:40 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:41 ns2 sendmail[19436]: m9AMJevw019436:
 Oct 10 15:20:10 ns2 pop3-login: Login: kmaclean [192.168.5.208]
 Oct 10 15:20:10 ns2 pop3-login: Login: lhernandez [192.168.15.208]
 Oct 10 15:20:11 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 pop3-login: Login: lcreceiving [192.168.101.155]
 Oct 10 15:20:12 ns2 pop3-login: Login: acaluag [192.168.10.222]
 Oct 10 15:20:12 ns2 sendmail[19589]: m9AMK41G019589: from=<kmaclean@coasteramer.com>, size=24137, class=0, nrcpts=1, msgid=<!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAk6DvyRr5zEK67tyYpWXWL8KAAAAQAAAAqWVaNEnJ7k2m1Ba4K5TB, proto=ESMTP, daemon=MTA, relay=[192.168.5.208]
 Oct 10 15:20:12 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 sendmail[19607]: m9AMK8AO019607: from=<BuyCostumes.com@nintrilab.com>, size=1378, class=0, nrcpts=1, msgid=<127533888.4755685@datinggathering.com>, bodytype=8BITMIME, proto=SMTP, daemon=MTA, relay=72.37.221.139.rdns.ubiquityservers.com [72.37.221.139] (may be forged)
 Oct 10 15:20:13 ns2 sendmail[19607]: m9AMK8AO019607: Milter: data, discard
 Oct 10 15:20:13 ns2 sendmail[19607]: m9AMK8AO019607: discarded
 Oct 10 15:20:13 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:13 ns2 pop3-login: Login: vismail [192.168.60.7]
 Oct 10 15:20:14 ns2 pop3-login: Login: agutierrez [192.168.70.47]
 Oct 10 15:20:14 ns2 pop3-login: Login: tkonetzny [70.0.132.37]
 Oct 10 15:20:14 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:14 ns2 pop3-login: Login: nweltz [192.168.15.143]
 Oct 10 15:20:15 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:15 ns2 sendmail[19635]: m9AMKFE8019635: <dortega@coasteramer.com>... User unknown

0
 
LVL 39

Accepted Solution

by:
Adam314 earned 500 total points
ID: 22694919
The part between the two forward slashes is a regular expression (abbreviated as regex or RE).  It is a way to specify a pattern in a string.  See here for details:
    http://perldoc.perl.org/perlretut.html
This allows you to look for a pattern in a string, and save parts of that string.  The saved parts are saved to the variables $1 (for the first thing), $2 (for the second things), and so on.

I also noticed the code you posted is slightly different from what I posted.  The case of the characters in the regex is important.  (which I copied your code and reposted incorrectly).  See below for updated version.

In this case, this is what it means
    ^              Start of line

    (              Start of group - save what is found to $1
    \w            A word character (MUST be lowercase)
    +             One ore more of that character
    )              End of group - save everything from ( to ) as $1

    \s             Whitespace (space or tab)
    +              One or more of that

    (               Start of group - save what is found to $2
    \d             A digit character (MUST be lowercase)
    +              One or more of that
    )              End of group - save everything from ( to ) as $1
   
    \s+            see above
    (\d+)          see above
    :                the colon character
    (\d+)          see above


So, this saves the month as $1, the day as $2, the hour as $3, the minute as $4, and the second as $5.  The timelocal function takes a sec,min,hour,day,month,year and turns it into epoch seconds (the time format returned by the time() function).  Then 30*60 is the number of seconds in 30 minutes, so it compares the time in the file to the current time.  If the file time is more than 30 minutes old, it skips this line.


In the part of the file you posted, every line begins with a space.  I'm not sure if the real file is this way, or it is just showing up that way here.  Either way, I've updated the regex to allow for spaces at the beginning.


#!/usr/bin/perl

use strict;

use warnings;

use Time::Local;

my %Months = (Jan => 0, Feb=>1, Mar=>2, Apr=>3, May=>4, Jun=>5, Jul=>6, Aug=>7, Sep=>8, Oct=>9, Nov=>10, Dec=>11);
 

open(my $file, "</var/log/maillog") or die "open: $!\n";
 

my $year = (localtime())[5];

while(<$file>) {

	next unless /^\s*(\w+)\s+(\d+)\s+(\d+):(\d+):(\d+)/;

	my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);

	next if time()-$linetime>30*60;

	print $_;

}

close($file);

Open in new window

0
 

Author Closing Comment

by:grazal
ID: 31504838
This time, it worked!  Thank you so much for your patience and understanding.  Also, thank you for pointing me to the perldoc.perl.org site.  I will use this to learn more on my own.  .....till next time.
0
 

Author Comment

by:grazal
ID: 23281714
Hi, this solution failed to work at the turn of the new year.   The $linetime value is blank.   This may have something to do with the year (it's now 2009).  Please how do I fix this bug?????
0
 
LVL 39

Expert Comment

by:Adam314
ID: 23315348
Has the format of your log changed?  The script gets the time from the system, so as long as your system time is correct, the year being 2009 should not cause any problems.
0
 

Author Comment

by:grazal
ID: 23381115
You know what?  I just realized that there's no problem with my code.  The problem was the data file.  The data was mixed in with the 2008 and 2009 data.  So, when my code/program encountered the 2008 data, the results became srewy.  Now that all the data are in the same year (2009), the results are now back to normal.  

Thank you very much for your time.  Sincerely...
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now