Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 800
  • Last Modified:

How to calculate between two given datetime in Perl

Let me explain further:

I have this file as a result of a query about more than a thousand lines.  It has a date and time on each line entry and stored in a file called maillog that looks like the following:

Sep  5 00:00:30ns2 pop3-login: Login: jflanagan [192.168.30.20]....
Sep  5 01:00:02 ns2 pop3-login: Login: vismail [192.168.70.7]...
Sep  5 00:20:28 ns2 sendmail[16116]: m8570ItF016116: Milter add: header: ...
Sep  5 00:00:30 ns2 pop3-login: Login: atinbound [192.168.50.37]...
Sep  5 00:01:35 ns2 pop3-login: Login: pyen [192.168.10.87]....
Sep  5 00:00:36 ns2 sendmail[16143]: m8570Zu2016143: ......

My task is to capture all the data against the current local datetime.  That is, write a Perl script to capture only lines entries  that are less than 30 minutes old compared to  the current local time.   Let say, if I ran a Perl script at local time (e.g. "Sep 5 00:01:00), only get the entries within 30 minutes of the local time.  
How can I represent the date in the maillog file and compare it with the current date.  Now, you have to keep in mind that the datetime is a flat string format, a flat file does not have the same format as the current datetime.  That is, you can manipulate the localtime() into different formats, but how can you manipulate that date in the maillog so that I can compare the two dates (the localtime and the date in the maillog file)?   Can you give me some answers and detailed example script please!!???

0
grazal
Asked:
grazal
  • 7
  • 5
1 Solution
 
Adam314Commented:
You can use the Time::Local module to convert the file time into a unix time (epoch seconds).  Then compare to 30*60 (for 30 minutes).

#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
my %Months = (Jan => 0, Feb=>1, Mar=>2, Apr=>3, May=>4, Jun=>5, Jul=>6, Aug=>7, Sep=>8, Oct=>9, Nov=>11, Dec=>12);
 
my $year = (localtime())[7];
while(<>) {
	next unless /^(\w+)\s+(\d+)\s+(\d+):(\d+):(\d+)/;
	my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
	next if time()-$linetime>30*60;
	print $_;
}

Open in new window

0
 
grazalAuthor Commented:
I copied the codes you gave me, but it gave me no output.  It should have had given me hundreds of lines of output, but in this case nothing came up!  Any suggestions please!

Here's a few excerpt from my input file "maillog":

Oct 15 04:02:56 ns2 pop3-login: Login: mlavarias [192.168.70.53]..
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: from=<dwvelvinm@velvin.com>, size=764, ..
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: Milter: data, discard...
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: discarded...
Oct 15 04:02:58 ns2 pop3-login: Login: pyen [192.168.10.87]...
Oct 15 04:03:01 ns2 sendmail[9686]: m95B31l9009686: mta-pva.cluster2.convio.net [66.45.103.48] did ...
Oct 15 04:03:02 ns2 sendmail[9635]: m95B2oof009635: ruleset=check_mail, .....

My perl script file contains:

#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
opendir(DIR, "/var/log");
open(READFILE,"/var/log/maillog");
my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6, Aug => 7, Sep => 8, Oct => 9, Nov => 11, Dec => 12);
my $year = (localtime()) [7];
while (<READFILE>)
{
   next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;
   my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
   next if time()-$linetime>30*60;
  print $_;
}
0
 
Adam314Commented:
There were a few bugs in my previous post... try this:
#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
open(READFILE,"/var/log/maillog");
my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6, Aug => 7, Sep => 8, Oct => 9, Nov => 10, Dec => 11);
my $year = (localtime())[5];
while (<READFILE>)
{
    next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;
    my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
    next if time()-$linetime>30*60;
    print $_;
}
close(READFILE);

Open in new window

0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
grazalAuthor Commented:
Here's what happened:  I revised my script per your instruction.  The script ran, but no output.  So, what I did was I commented lines 11 and 13 just to troubleshoot.  
As a result I got the following Error:

Day '' out of range 1..31 at test2.pl line 12

Here's the revised script:
#!usr/bin/perl
use strict;
use warnings;
use Time::Local;
opendir(DIR, "/var/log");
open(READFILE,"/var/log/maillog");
my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6,
                         Aug => 7, Sep => 8, Oct => 9, Nov => 10, Dec => 11);
my $year = (localtime()) [5];
while (<READFILE>)
{
        # next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;
   my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
       # next if time()-$linetime>30*60;
  print $_;
}
close(READFILE);

I'm clueless!  
0
 
Adam314Commented:
It looks like you commented lines 10 and 12 in the version I posted... line 10 is what parses the line from your file, getting the date from it.  Without that line, it won't work at all.  Line 12 is what skips the lines older than 30 minutes.  Without that, you will get all lines printed.

When I run it on your sample data, I get the output as expected.

If you get no output, it could be your file doesn't have any entries within the last 30 minutes.  If you post your actual file that you are using, I'll take a look.
0
 
grazalAuthor Commented:
To Adam314:

Please explain to me lines 11 thru 13 of the While loop above?  what does those numbers (e.g. $5, $4,..\W, \d+, etc. supposed to mean?  Sorry for my ignorance...but I'm so new to this!

Thanks
0
 
grazalAuthor Commented:
Here's some actual contents of the file "maillog".  I use the grep command to capture them, because the actual file contains hundreds of line input.  In fact, lines of input are added every second to this file.

 Oct 10 15:19:33 ns2 pop3-login: Login: maryp [192.168.80.31]
 Oct 10 15:19:33 ns2 pop3-login: Login: dku [192.168.193.124]
 Oct 10 15:19:34 ns2 pop3-login: Login: kmaclean [192.168.5.208]
 Oct 10 15:19:35 ns2 sendmail[19380]: m9AMJW7G019380: from=<>, size=2420, class=0, nrcpts=1, msgid=<01ce01c92b26$45cb1ae0$e905a8c0@LA46>, proto=ESMTP, daemon=MTA, relay=[192.168.5.233]
 Oct 10 15:19:35 ns2 pop3-login: Login: pyen [192.168.10.87]
 Oct 10 15:19:35 ns2 sendmail[19392]: m9AMJW7G019380: to=<maryp@coasteramer.com>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=32594, dsn=2.0.0, stat=Sent
 Oct 10 15:19:36 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:37 ns2 last message repeated 2 times
 Oct 10 15:19:37 ns2 pop3-login: Login: pmohabbat [192.168.70.28]
 Oct 10 15:19:37 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:39 ns2 last message repeated 3 times
 Oct 10 15:19:39 ns2 pop3-login: Login: vismail [192.168.70.7]
 Oct 10 15:19:39 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:40 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:40 ns2 pop3-login: Login: srosenthal [192.168.30.54]
 Oct 10 15:19:40 ns2 sendmail[19287]: m9AMJex9019287: [87.126.124.96] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA
 Oct 10 15:19:40 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:41 ns2 sendmail[19436]: m9AMJevw019436:
 Oct 10 15:20:10 ns2 pop3-login: Login: kmaclean [192.168.5.208]
 Oct 10 15:20:10 ns2 pop3-login: Login: lhernandez [192.168.15.208]
 Oct 10 15:20:11 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 pop3-login: Login: lcreceiving [192.168.101.155]
 Oct 10 15:20:12 ns2 pop3-login: Login: acaluag [192.168.10.222]
 Oct 10 15:20:12 ns2 sendmail[19589]: m9AMK41G019589: from=<kmaclean@coasteramer.com>, size=24137, class=0, nrcpts=1, msgid=<!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAk6DvyRr5zEK67tyYpWXWL8KAAAAQAAAAqWVaNEnJ7k2m1Ba4K5TB, proto=ESMTP, daemon=MTA, relay=[192.168.5.208]
 Oct 10 15:20:12 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 sendmail[19607]: m9AMK8AO019607: from=<BuyCostumes.com@nintrilab.com>, size=1378, class=0, nrcpts=1, msgid=<127533888.4755685@datinggathering.com>, bodytype=8BITMIME, proto=SMTP, daemon=MTA, relay=72.37.221.139.rdns.ubiquityservers.com [72.37.221.139] (may be forged)
 Oct 10 15:20:13 ns2 sendmail[19607]: m9AMK8AO019607: Milter: data, discard
 Oct 10 15:20:13 ns2 sendmail[19607]: m9AMK8AO019607: discarded
 Oct 10 15:20:13 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:13 ns2 pop3-login: Login: vismail [192.168.60.7]
 Oct 10 15:20:14 ns2 pop3-login: Login: agutierrez [192.168.70.47]
 Oct 10 15:20:14 ns2 pop3-login: Login: tkonetzny [70.0.132.37]
 Oct 10 15:20:14 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:14 ns2 pop3-login: Login: nweltz [192.168.15.143]
 Oct 10 15:20:15 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:15 ns2 sendmail[19635]: m9AMKFE8019635: <dortega@coasteramer.com>... User unknown

0
 
Adam314Commented:
The part between the two forward slashes is a regular expression (abbreviated as regex or RE).  It is a way to specify a pattern in a string.  See here for details:
    http://perldoc.perl.org/perlretut.html
This allows you to look for a pattern in a string, and save parts of that string.  The saved parts are saved to the variables $1 (for the first thing), $2 (for the second things), and so on.

I also noticed the code you posted is slightly different from what I posted.  The case of the characters in the regex is important.  (which I copied your code and reposted incorrectly).  See below for updated version.

In this case, this is what it means
    ^              Start of line

    (              Start of group - save what is found to $1
    \w            A word character (MUST be lowercase)
    +             One ore more of that character
    )              End of group - save everything from ( to ) as $1

    \s             Whitespace (space or tab)
    +              One or more of that

    (               Start of group - save what is found to $2
    \d             A digit character (MUST be lowercase)
    +              One or more of that
    )              End of group - save everything from ( to ) as $1
   
    \s+            see above
    (\d+)          see above
    :                the colon character
    (\d+)          see above


So, this saves the month as $1, the day as $2, the hour as $3, the minute as $4, and the second as $5.  The timelocal function takes a sec,min,hour,day,month,year and turns it into epoch seconds (the time format returned by the time() function).  Then 30*60 is the number of seconds in 30 minutes, so it compares the time in the file to the current time.  If the file time is more than 30 minutes old, it skips this line.


In the part of the file you posted, every line begins with a space.  I'm not sure if the real file is this way, or it is just showing up that way here.  Either way, I've updated the regex to allow for spaces at the beginning.


#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
my %Months = (Jan => 0, Feb=>1, Mar=>2, Apr=>3, May=>4, Jun=>5, Jul=>6, Aug=>7, Sep=>8, Oct=>9, Nov=>10, Dec=>11);
 
open(my $file, "</var/log/maillog") or die "open: $!\n";
 
my $year = (localtime())[5];
while(<$file>) {
	next unless /^\s*(\w+)\s+(\d+)\s+(\d+):(\d+):(\d+)/;
	my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
	next if time()-$linetime>30*60;
	print $_;
}
close($file);

Open in new window

0
 
grazalAuthor Commented:
This time, it worked!  Thank you so much for your patience and understanding.  Also, thank you for pointing me to the perldoc.perl.org site.  I will use this to learn more on my own.  .....till next time.
0
 
grazalAuthor Commented:
Hi, this solution failed to work at the turn of the new year.   The $linetime value is blank.   This may have something to do with the year (it's now 2009).  Please how do I fix this bug?????
0
 
Adam314Commented:
Has the format of your log changed?  The script gets the time from the system, so as long as your system time is correct, the year being 2009 should not cause any problems.
0
 
grazalAuthor Commented:
You know what?  I just realized that there's no problem with my code.  The problem was the data file.  The data was mixed in with the 2008 and 2009 data.  So, when my code/program encountered the 2008 data, the results became srewy.  Now that all the data are in the same year (2009), the results are now back to normal.  

Thank you very much for your time.  Sincerely...
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 7
  • 5
Tackle projects and never again get stuck behind a technical roadblock.
Join Now