Solved

Reading log file in Perl

Posted on 2011-09-29
16
191 Views
Last Modified: 2012-05-12
Hi,
Can anyone please help me with the code snippet for the below issue:

Need to the read the attached "Test_log" file on a daily basis based on date  and generate the output as shown below into a log file. Next login to Db and run a select query statement and generate a log file with the query results.
Finally send an email with both the log files attachment and the  total count of results in the log files  as subject line

[b]output for the log file attached should be:[/b]
HEFFILE0004/02720220R
HEFFILE0004/60530610R
HEFFILE0004/60530700R
HEFFILE0004/60531070R

Open in new window

Test-log.txt
0
Comment
Question by:new_perl_user
  • 10
  • 6
16 Comments
 
LVL 10

Expert Comment

by:jeromee
ID: 36817189
you would need to be more specific but in term of the rules governing what to generate in the log file and more importantly "the login to Db and run a select query statement and generate a log file with the query results". This is so vague that it makes my head spin.
0
 

Author Comment

by:new_perl_user
ID: 36817494
I  am sorry.

1)Below is the  file I need to read daily and grab yesterday's data from that log file based on date and then write the ouput  to  daily.txt.

Test_log sample  Data:

##################### SYNC  STARTED: Mon Sep 26 03:10:05
 #####################
receiving file list ... done
.d..t.... HEFFILE0004/02720220R/METADATA/
.d..t.... HEFFILE0004/60530610R/METADATA/
.d..t.... HEFFILE0004/60530700R/METADATA/
.d..t.... HEFFILE0004/60531070R/METADATA/
.d..t.... HEFFILE0004/60910750R/METADATA/
##################### SYNC FINISHED: Mon Sep 26 05:15:39 EDT 2011  #####################

Open in new window


Output Format for daily.txt :
HEFFILE0004/02720220R
HEFFILE0004/60530610R
HEFFILE0004/60530700R
HEFFILE0004/60531070R


2) After that login to Oracle Database and the run the below query:
Select a,b from table1 where date =to_date(SYSDATE, 'DD.MM.YY')-1;  and write the output to a Db.txt


3) finally send an email with both the logs attached (daily.txt , Db.txt). If possible can we get the count  of data in each log file as subject line of an email  like( daily.txt = 10 files , db.txt = 20 files)

If I am not clear yet please let me know. Thank you for  the  help.
0
 

Author Comment

by:new_perl_user
ID: 36818901
can anyone please help me on this..
0
 

Author Comment

by:new_perl_user
ID: 36891477
any suggestions will be very helpful

Thanks,
0
 

Author Comment

by:new_perl_user
ID: 36894368
Hi Jeromee,
If possible can you please help me with #1. I am able to do the #2  and will try out #3.

Thanks,
0
 
LVL 10

Expert Comment

by:jeromee
ID: 36894577
Here's #1 only...

run as
   perl thisscript.pl  Test.log.txt

Please note that I made some assumption re: what the filename must look like (HEFFILEnnnn/yyyyyy/)
use strict;

my $yesterday = scalar localtime(time()-24*60*60);	# Thu Sep 29 14:59:20 2011
substr($yesterday, 11, 9) = "";				# remove the time
my $date = '';
while( <> ) {
   if( /SYNC  STARTED: (.*) \d\d:\d\d:\d\d .* (\d{4})/ ) {	# Extract the date of each sectio
      $date = "$1 $2";
   }

   if( $yesterday eq $date && m{(HEFFILE\d+/[^/]+)} ) {		# if the same date as yesterday and has the right filename signature
      print "$1\n";
   }
}

Open in new window

0
 

Author Comment

by:new_perl_user
ID: 36894718
Hi,

Thank you so much for the response.
 
I tried the above code but it is not printing anything. Can you please take a look. Below is the sample Test.log.txt I am trying to read.

##################### SYNC  STARTED: Thu Sep 29 03:10:05 EDT 2011 #####################
receiving file list ... done
.d..t.... HEFFILE0004/02720220R/METADATA/
.d..t.... HEFFILE0004/60530610R/METADATA/
.d..t.... HEFFILE0004/60530700R/METADATA/
.d..t.... HEFFILE0004/60531070R/METADATA/
.d..t.... HEFFILE0004/60910750R/METADATA/
##################### SYNC FINISHED: Thu Sep 29 05:15:39 EDT 2011 #####################

Open in new window

0
 
LVL 10

Expert Comment

by:jeromee
ID: 36894734
It works for me:

HEFFILE0004/02720220R
HEFFILE0004/60530610R
HEFFILE0004/60530700R
HEFFILE0004/60531070R
HEFFILE0004/60910750R

How do you run the script?
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:new_perl_user
ID: 36894891
My bad it is working now. Executed in a wrong way.  One small change needed if possible. Right now it is printing all the files based on date, even if there are repetitive filenames like

HEFFILE0004/02720220R
HEFFILE0004/02720220R
HEFFILE0004/60530610R
HEFFILE0004/60530700R

 So it is possible to get just only one entry for repetitive data and write the output to a log file.

Thanks
0
 
LVL 10

Accepted Solution

by:
jeromee earned 500 total points
ID: 36894916

use strict;

my $yesterday = localtime(time()-24*60*60);	# Thu Sep 29 14:59:20 2011
substr($yesterday, 11, 9) = "";			# remove the time
my $date = '';
my %output;
while( <> ) {
   if( /SYNC  STARTED: (.*) \d\d:\d\d:\d\d .* (\d{4})/ ) {	# Extract the date of each sectio
      $date = "$1 $2";
   }

   if( $yesterday eq $date && m{(HEFFILE\d+/[^/]+)} ) {		# if the same date as yesterday and has the right filename signature
      $output{"$1\n"}++;
   }
}
print join("",sort keys %output);

Open in new window

0
 

Author Comment

by:new_perl_user
ID: 36894994
can we write the output to a log file.
0
 
LVL 10

Expert Comment

by:jeromee
ID: 36895130
perl thisscript.pl  Test.log.txt > output.txt
0
 

Author Comment

by:new_perl_user
ID: 36898165
Thanks a ton for the help.
0
 
LVL 10

Expert Comment

by:jeromee
ID: 36898577
no problem.
Happy Perling!
0
 

Author Comment

by:new_perl_user
ID: 36903780
Hi Jeromee,

I was running the script again today to grab for data from yesterday(oct 2) but it is not working . Any ideas or suggestion on why??
0
 

Author Comment

by:new_perl_user
ID: 36904037
Never mind.. It is working, there were spaces near the date so it failed. Removed those and it is fine..

Thanks,
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
use perl to insert into MySQL database 9 128
Existing perl code to be changed for ftp to sftp handling 14 160
perl search and replace 6 165
what are these modules in perl script 4 93
Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
When you create an app prototype with Adobe XD, you can insert system screens -- sharing or Control Center, for example -- with just a few clicks. This video shows you how. You can take the full course on Experts Exchange at http://bit.ly/XDcourse.

914 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now