Solved

How to calculate between two given datetime in Perl

Posted on 2008-10-09
12
752 Views
Last Modified: 2013-11-17
Let me explain further:

I have this file as a result of a query about more than a thousand lines.  It has a date and time on each line entry and stored in a file called maillog that looks like the following:

Sep  5 00:00:30ns2 pop3-login: Login: jflanagan [192.168.30.20]....
Sep  5 01:00:02 ns2 pop3-login: Login: vismail [192.168.70.7]...
Sep  5 00:20:28 ns2 sendmail[16116]: m8570ItF016116: Milter add: header: ...
Sep  5 00:00:30 ns2 pop3-login: Login: atinbound [192.168.50.37]...
Sep  5 00:01:35 ns2 pop3-login: Login: pyen [192.168.10.87]....
Sep  5 00:00:36 ns2 sendmail[16143]: m8570Zu2016143: ......

My task is to capture all the data against the current local datetime.  That is, write a Perl script to capture only lines entries  that are less than 30 minutes old compared to  the current local time.   Let say, if I ran a Perl script at local time (e.g. "Sep 5 00:01:00), only get the entries within 30 minutes of the local time.  
How can I represent the date in the maillog file and compare it with the current date.  Now, you have to keep in mind that the datetime is a flat string format, a flat file does not have the same format as the current datetime.  That is, you can manipulate the localtime() into different formats, but how can you manipulate that date in the maillog so that I can compare the two dates (the localtime and the date in the maillog file)?   Can you give me some answers and detailed example script please!!???

0
Comment
Question by:grazal
  • 7
  • 5
12 Comments
 
LVL 39

Expert Comment

by:Adam314
Comment Utility
You can use the Time::Local module to convert the file time into a unix time (epoch seconds).  Then compare to 30*60 (for 30 minutes).

#!/usr/bin/perl

use strict;

use warnings;

use Time::Local;

my %Months = (Jan => 0, Feb=>1, Mar=>2, Apr=>3, May=>4, Jun=>5, Jul=>6, Aug=>7, Sep=>8, Oct=>9, Nov=>11, Dec=>12);
 

my $year = (localtime())[7];

while(<>) {

	next unless /^(\w+)\s+(\d+)\s+(\d+):(\d+):(\d+)/;

	my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);

	next if time()-$linetime>30*60;

	print $_;

}

Open in new window

0
 

Author Comment

by:grazal
Comment Utility
I copied the codes you gave me, but it gave me no output.  It should have had given me hundreds of lines of output, but in this case nothing came up!  Any suggestions please!

Here's a few excerpt from my input file "maillog":

Oct 15 04:02:56 ns2 pop3-login: Login: mlavarias [192.168.70.53]..
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: from=<dwvelvinm@velvin.com>, size=764, ..
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: Milter: data, discard...
Oct 15 04:02:58 ns2 sendmail[9396]: m95B2lPd009396: discarded...
Oct 15 04:02:58 ns2 pop3-login: Login: pyen [192.168.10.87]...
Oct 15 04:03:01 ns2 sendmail[9686]: m95B31l9009686: mta-pva.cluster2.convio.net [66.45.103.48] did ...
Oct 15 04:03:02 ns2 sendmail[9635]: m95B2oof009635: ruleset=check_mail, .....

My perl script file contains:

#!/usr/bin/perl
use strict;
use warnings;
use Time::Local;
opendir(DIR, "/var/log");
open(READFILE,"/var/log/maillog");
my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6, Aug => 7, Sep => 8, Oct => 9, Nov => 11, Dec => 12);
my $year = (localtime()) [7];
while (<READFILE>)
{
   next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;
   my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
   next if time()-$linetime>30*60;
  print $_;
}
0
 
LVL 39

Expert Comment

by:Adam314
Comment Utility
There were a few bugs in my previous post... try this:
#!/usr/bin/perl

use strict;

use warnings;

use Time::Local;

open(READFILE,"/var/log/maillog");

my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6, Aug => 7, Sep => 8, Oct => 9, Nov => 10, Dec => 11);

my $year = (localtime())[5];

while (<READFILE>)

{

    next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;

    my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);

    next if time()-$linetime>30*60;

    print $_;

}

close(READFILE);

Open in new window

0
 

Author Comment

by:grazal
Comment Utility
Here's what happened:  I revised my script per your instruction.  The script ran, but no output.  So, what I did was I commented lines 11 and 13 just to troubleshoot.  
As a result I got the following Error:

Day '' out of range 1..31 at test2.pl line 12

Here's the revised script:
#!usr/bin/perl
use strict;
use warnings;
use Time::Local;
opendir(DIR, "/var/log");
open(READFILE,"/var/log/maillog");
my %Months = (Jan => 0, Feb => 1, Mar => 2, Apr =>3, May => 4, Jun => 5, Jul => 6,
                         Aug => 7, Sep => 8, Oct => 9, Nov => 10, Dec => 11);
my $year = (localtime()) [5];
while (<READFILE>)
{
        # next unless /^(\W+)\s+(\d+)\s+(\d+):(\d+)/;
   my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);
       # next if time()-$linetime>30*60;
  print $_;
}
close(READFILE);

I'm clueless!  
0
 
LVL 39

Expert Comment

by:Adam314
Comment Utility
It looks like you commented lines 10 and 12 in the version I posted... line 10 is what parses the line from your file, getting the date from it.  Without that line, it won't work at all.  Line 12 is what skips the lines older than 30 minutes.  Without that, you will get all lines printed.

When I run it on your sample data, I get the output as expected.

If you get no output, it could be your file doesn't have any entries within the last 30 minutes.  If you post your actual file that you are using, I'll take a look.
0
 

Author Comment

by:grazal
Comment Utility
To Adam314:

Please explain to me lines 11 thru 13 of the While loop above?  what does those numbers (e.g. $5, $4,..\W, \d+, etc. supposed to mean?  Sorry for my ignorance...but I'm so new to this!

Thanks
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:grazal
Comment Utility
Here's some actual contents of the file "maillog".  I use the grep command to capture them, because the actual file contains hundreds of line input.  In fact, lines of input are added every second to this file.

 Oct 10 15:19:33 ns2 pop3-login: Login: maryp [192.168.80.31]
 Oct 10 15:19:33 ns2 pop3-login: Login: dku [192.168.193.124]
 Oct 10 15:19:34 ns2 pop3-login: Login: kmaclean [192.168.5.208]
 Oct 10 15:19:35 ns2 sendmail[19380]: m9AMJW7G019380: from=<>, size=2420, class=0, nrcpts=1, msgid=<01ce01c92b26$45cb1ae0$e905a8c0@LA46>, proto=ESMTP, daemon=MTA, relay=[192.168.5.233]
 Oct 10 15:19:35 ns2 pop3-login: Login: pyen [192.168.10.87]
 Oct 10 15:19:35 ns2 sendmail[19392]: m9AMJW7G019380: to=<maryp@coasteramer.com>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=32594, dsn=2.0.0, stat=Sent
 Oct 10 15:19:36 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:37 ns2 last message repeated 2 times
 Oct 10 15:19:37 ns2 pop3-login: Login: pmohabbat [192.168.70.28]
 Oct 10 15:19:37 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:39 ns2 last message repeated 3 times
 Oct 10 15:19:39 ns2 pop3-login: Login: vismail [192.168.70.7]
 Oct 10 15:19:39 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:40 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:40 ns2 pop3-login: Login: srosenthal [192.168.30.54]
 Oct 10 15:19:40 ns2 sendmail[19287]: m9AMJex9019287: [87.126.124.96] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA
 Oct 10 15:19:40 ns2 pop3-login: Login: rbhatti [192.168.70.25]
 Oct 10 15:19:41 ns2 sendmail[19436]: m9AMJevw019436:
 Oct 10 15:20:10 ns2 pop3-login: Login: kmaclean [192.168.5.208]
 Oct 10 15:20:10 ns2 pop3-login: Login: lhernandez [192.168.15.208]
 Oct 10 15:20:11 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 pop3-login: Login: lcreceiving [192.168.101.155]
 Oct 10 15:20:12 ns2 pop3-login: Login: acaluag [192.168.10.222]
 Oct 10 15:20:12 ns2 sendmail[19589]: m9AMK41G019589: from=<kmaclean@coasteramer.com>, size=24137, class=0, nrcpts=1, msgid=<!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAk6DvyRr5zEK67tyYpWXWL8KAAAAQAAAAqWVaNEnJ7k2m1Ba4K5TB, proto=ESMTP, daemon=MTA, relay=[192.168.5.208]
 Oct 10 15:20:12 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:12 ns2 sendmail[19607]: m9AMK8AO019607: from=<BuyCostumes.com@nintrilab.com>, size=1378, class=0, nrcpts=1, msgid=<127533888.4755685@datinggathering.com>, bodytype=8BITMIME, proto=SMTP, daemon=MTA, relay=72.37.221.139.rdns.ubiquityservers.com [72.37.221.139] (may be forged)
 Oct 10 15:20:13 ns2 sendmail[19607]: m9AMK8AO019607: Milter: data, discard
 Oct 10 15:20:13 ns2 sendmail[19607]: m9AMK8AO019607: discarded
 Oct 10 15:20:13 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:13 ns2 pop3-login: Login: vismail [192.168.60.7]
 Oct 10 15:20:14 ns2 pop3-login: Login: agutierrez [192.168.70.47]
 Oct 10 15:20:14 ns2 pop3-login: Login: tkonetzny [70.0.132.37]
 Oct 10 15:20:14 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:14 ns2 pop3-login: Login: nweltz [192.168.15.143]
 Oct 10 15:20:15 ns2 pop3-login: Login: maybelle [192.168.70.35]
 Oct 10 15:20:15 ns2 sendmail[19635]: m9AMKFE8019635: <dortega@coasteramer.com>... User unknown

0
 
LVL 39

Accepted Solution

by:
Adam314 earned 500 total points
Comment Utility
The part between the two forward slashes is a regular expression (abbreviated as regex or RE).  It is a way to specify a pattern in a string.  See here for details:
    http://perldoc.perl.org/perlretut.html
This allows you to look for a pattern in a string, and save parts of that string.  The saved parts are saved to the variables $1 (for the first thing), $2 (for the second things), and so on.

I also noticed the code you posted is slightly different from what I posted.  The case of the characters in the regex is important.  (which I copied your code and reposted incorrectly).  See below for updated version.

In this case, this is what it means
    ^              Start of line

    (              Start of group - save what is found to $1
    \w            A word character (MUST be lowercase)
    +             One ore more of that character
    )              End of group - save everything from ( to ) as $1

    \s             Whitespace (space or tab)
    +              One or more of that

    (               Start of group - save what is found to $2
    \d             A digit character (MUST be lowercase)
    +              One or more of that
    )              End of group - save everything from ( to ) as $1
   
    \s+            see above
    (\d+)          see above
    :                the colon character
    (\d+)          see above


So, this saves the month as $1, the day as $2, the hour as $3, the minute as $4, and the second as $5.  The timelocal function takes a sec,min,hour,day,month,year and turns it into epoch seconds (the time format returned by the time() function).  Then 30*60 is the number of seconds in 30 minutes, so it compares the time in the file to the current time.  If the file time is more than 30 minutes old, it skips this line.


In the part of the file you posted, every line begins with a space.  I'm not sure if the real file is this way, or it is just showing up that way here.  Either way, I've updated the regex to allow for spaces at the beginning.


#!/usr/bin/perl

use strict;

use warnings;

use Time::Local;

my %Months = (Jan => 0, Feb=>1, Mar=>2, Apr=>3, May=>4, Jun=>5, Jul=>6, Aug=>7, Sep=>8, Oct=>9, Nov=>10, Dec=>11);
 

open(my $file, "</var/log/maillog") or die "open: $!\n";
 

my $year = (localtime())[5];

while(<$file>) {

	next unless /^\s*(\w+)\s+(\d+)\s+(\d+):(\d+):(\d+)/;

	my $linetime = timelocal($5, $4, $3, $2, $Months{$1}, $year);

	next if time()-$linetime>30*60;

	print $_;

}

close($file);

Open in new window

0
 

Author Closing Comment

by:grazal
Comment Utility
This time, it worked!  Thank you so much for your patience and understanding.  Also, thank you for pointing me to the perldoc.perl.org site.  I will use this to learn more on my own.  .....till next time.
0
 

Author Comment

by:grazal
Comment Utility
Hi, this solution failed to work at the turn of the new year.   The $linetime value is blank.   This may have something to do with the year (it's now 2009).  Please how do I fix this bug?????
0
 
LVL 39

Expert Comment

by:Adam314
Comment Utility
Has the format of your log changed?  The script gets the time from the system, so as long as your system time is correct, the year being 2009 should not cause any problems.
0
 

Author Comment

by:grazal
Comment Utility
You know what?  I just realized that there's no problem with my code.  The problem was the data file.  The data was mixed in with the 2008 and 2009 data.  So, when my code/program encountered the 2008 data, the results became srewy.  Now that all the data are in the same year (2009), the results are now back to normal.  

Thank you very much for your time.  Sincerely...
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now