?
Solved

How to parse server logs and send it back to server as a request in PERL?

Posted on 2011-09-30
4
Medium Priority
?
175 Views
Last Modified: 2012-05-12
This is an example of my server logs.

"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:48:32 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:53:32 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:58:34 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:03:40 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /srcsearch/ HTTP/1.1" 200 12931
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:23 -0500" "GET /srcsearch/SearchResults.do?searchTerm=sdk&searchField=TEXT&sort=PATH&fileType=M&fileType=Model&fileType=Java&fileType=C%2B%2B&fileType=Fortran&fileType=Header&fileType=C&fileType=XML&fileType=Resource&fileType=TLC&fileType=Makefile&fileType=MTF&fileType=Requirements&fileType=Shell+Scripts&fileType=Perl&fileType=Message+Table&fileType=Translation+Table&fileType=C%23&fileType=JavaScript&fileType=Visual+Basic&fileType=Chart&fileType=Configuration&sourceDir=java%2Fsrc&sourceDir=src&sourceDir=simulink%2Fsrc&sourceDir=test&sourceDir=toolbox&sourceDir=standalone&sourceDir=makerules&sourceDir=extern%2Finclude&sourceDir=resources&sourceDir=rtw&sourceDir=stateflow&sourceDir=config&sourceDir=foundation_libraries&sourceDir=install&sourceDir=pbr&indexId=0&indexDir= HTTP/1.1" 200 1232330
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:54 -0500" "GET /srcsearch/SearchResults.do?searchTerm=sdk&searchField=FILENAME&sort=PATH&fileType=M&fileType=Model&fileType=Java&fileType=C%2B%2B&fileType=Fortran&fileType=Header&fileType=C&fileType=XML&fileType=Resource&fileType=TLC&fileType=Makefile&fileType=MTF&fileType=Requirements&fileType=Shell+Scripts&fileType=Perl&fileType=Message+Table&fileType=Translation+Table&fileType=C%23&fileType=JavaScript&fileType=Visual+Basic&fileType=Chart&fileType=Configuration&sourceDir=java%2Fsrc&sourceDir=src&sourceDir=simulink%2Fsrc&sourceDir=test&sourceDir=toolbox&sourceDir=standalone&sourceDir=makerules&sourceDir=extern%2Finclude&sourceDir=resources&sourceDir=rtw&sourceDir=stateflow&sourceDir=config&sourceDir=foundation_libraries&sourceDir=install&sourceDir=pbr&indexId=1&indexDir= HTTP/1.1" 200 16379
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:06:35 -0500" "GET /srcsearch/ HTTP/1.1" 200 12851
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:08:41 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:12:11 -0500" "GET /srcsearch/SearchResults.do?searchTerm=blkexist HTTP/1.1" 200 23033
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:12:11 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:13:42 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:18:37 -0500" "GET /srcsearch/ HTTP/1.1" 200 11271
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:18:49 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:18:58 -0500" "GET /srcsearch/SearchResults.do?searchTerm=qeVerifyWithPolling&searchField=TEXT&sort=PATH&fileType=M&sourceDir=test&indexId=0&indexDir= HTTP/1.1" 200 1140275
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:19:06 -0500" "GET /srcsearch/Preview.do?file=test%2Fdata%2Ftestmeas%2Finstrument%2Finterface%2FhSharedFcnPropsAsync.m&searchString=qeverifywithpolling&caseSensitive=false&indexId=17&indexDir= HTTP/1.1" 200 0

Open in new window



I want to write a parser that will extract only the part that is after GET till the first space character. Example from the first one is:
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:48:32 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554

Open in new window


Extracted part:

/srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java

Open in new window


and the time stamp for the first one can be assumed as time=0

Then the others will be with respect to it. Which means:

The time stamp for the first request is :

29/Sep/2011:04:48:32

Open in new window


The time stamp for the second request is :

29/Sep/2011:04:53:32

Open in new window


So the delta is exactly 5 minutes. We will use this delta while sending request to the server.

Then append this URL to something like:

http://myapp.mycmp.com:8080 

Open in new window


Once I get this data I want to send these URLs to the server with the time difference which we calculated.

NOTE: As far as I know, CURL can be used for this purpose. I work on UNIX environment.

Or if there is an easier way to do it rather than CURL, that would be fine too.

Let me summarize it as a sudo code:

For each line that starts with "xxx.xx.xxx.xx"  (x is a number and it can be anything. and its format may change like xxx.x.xxxxx.xx) extract the time stamp and find the difference between the previous one.

If it is the first one then assign the current time or zero to it.

Then extract the part which is right after GET till the next space. But don't get the space before and after the extracted part. 

then append this URL to something like:

http://myapp.mycmp.com:8080 

and send this entire thing to the server.

Open in new window



As a full short example:

send this request to the server
http://myapp.mycmp.com:8080/srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java

Open in new window


wait for 5 minutes

(because there is a 5 minute difference between the first two requests)

then send this one
http://myapp.mycmp.com:8080/srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java

Open in new window


and the process continues till the end of the file
0
Comment
Question by:Tolgar
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
4 Comments
 
LVL 9

Expert Comment

by:parparov
ID: 36894775
If the delta is zero, should the refetch be done at once?

0
 
LVL 9

Accepted Solution

by:
parparov earned 2000 total points
ID: 36894825
Assuming a 'Yes' on the previous Q, here is the code.
You will need to install LWP::Simple perl module if you haven't it already.

#!/usr/bin/perl

use strict;
use warnings;

use LWP::Simple;
use POSIX qw(mktime);

my $file;
$file = shift || die "Can't read file $file: $!";

open(F, $file);
my $BASE_URL = "http://myapp.mycmp.com:8080";
my %MONTHS = qw(
	Jan 0 Feb 1 Mar 2 Apr 3 May 4 Jun 5 Jul 6 Aug 7 Sep 8 Oct 9 Nov 10 Dec 11
);

sub get_timestamp ($$$$$$) {

	my ($sec, $min, $hou, $day, $mon, $yea) = @_;
	$yea -= 1900;
	$mon = $MONTHS{$mon};
	return mktime($sec, $min, $hou, $day, $mon, $yea, 0, 0);
}
my $ts0 = 0;

while (<F>) {
	if (/^\"\w+\.\w+\.\w+\.\w+\"\s+.*\s+\"(\d+)\/(\w{3})\/(\d+):(\d+):(\d+):(\d+)\s+.*\s+\"GET (\S+)/) {
#		print;
		my $day = $1;
		my $mon = $2;
		my $yea = $3;
		my $hou = $4;
		my $min = $5;
		my $sec = $6;
		my $url = $7;
		my $ts1 = get_timestamp($sec, $min, $hou, $day, $mon, $yea);
		my $to_sleep = $ts0 ? $ts1 - $ts0 : 0;
		print "Sleeping: $to_sleep\n";
		sleep($to_sleep);
		$ts0 = $ts1;
		print "At ", scalar(localtime), " get(\"$BASE_URL$url\");\n";
		get("$BASE_URL$url");
	}
}

Open in new window

0
 

Author Comment

by:Tolgar
ID: 36894894
@parparov: my answer to your first question is yes. and thank you for your quick and prompt response.

I will try your code.

Thanks,
0
 

Author Comment

by:Tolgar
ID: 36898428
I haven't checked it yet but let's say I don't have LWP::Simple module

But let's say  I found it and download it to my computer. Then how can I direct my Perl code to see this module?

Because, the perl is not locally installed on my machine.

Thanks,
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this tutorial I will show you how to provide a dynamic RTF document on your website generated with data from your database. For this tutorial you will need Microsoft Word or WordPad, WhizBase and Microsoft Access. In this tutorial I will show …
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)
Suggested Courses

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question