Tolgar
asked on
How to parse server logs and send it back to server as a request in PERL?
This is an example of my server logs.
I want to write a parser that will extract only the part that is after GET till the first space character. Example from the first one is:
Extracted part:
and the time stamp for the first one can be assumed as time=0
Then the others will be with respect to it. Which means:
The time stamp for the first request is :
The time stamp for the second request is :
So the delta is exactly 5 minutes. We will use this delta while sending request to the server.
Then append this URL to something like:
Once I get this data I want to send these URLs to the server with the time difference which we calculated.
NOTE: As far as I know, CURL can be used for this purpose. I work on UNIX environment.
Or if there is an easier way to do it rather than CURL, that would be fine too.
Let me summarize it as a sudo code:
As a full short example:
send this request to the server
wait for 5 minutes
(because there is a 5 minute difference between the first two requests)
then send this one
and the process continues till the end of the file
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:48:32 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:53:32 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:58:34 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:03:40 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /srcsearch/ HTTP/1.1" 200 12931
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:17 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:23 -0500" "GET /srcsearch/SearchResults.do?searchTerm=sdk&searchField=TEXT&sort=PATH&fileType=M&fileType=Model&fileType=Java&fileType=C%2B%2B&fileType=Fortran&fileType=Header&fileType=C&fileType=XML&fileType=Resource&fileType=TLC&fileType=Makefile&fileType=MTF&fileType=Requirements&fileType=Shell+Scripts&fileType=Perl&fileType=Message+Table&fileType=Translation+Table&fileType=C%23&fileType=JavaScript&fileType=Visual+Basic&fileType=Chart&fileType=Configuration&sourceDir=java%2Fsrc&sourceDir=src&sourceDir=simulink%2Fsrc&sourceDir=test&sourceDir=toolbox&sourceDir=standalone&sourceDir=makerules&sourceDir=extern%2Finclude&sourceDir=resources&sourceDir=rtw&sourceDir=stateflow&sourceDir=config&sourceDir=foundation_libraries&sourceDir=install&sourceDir=pbr&indexId=0&indexDir= HTTP/1.1" 200 1232330
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:05:54 -0500" "GET /srcsearch/SearchResults.do?searchTerm=sdk&searchField=FILENAME&sort=PATH&fileType=M&fileType=Model&fileType=Java&fileType=C%2B%2B&fileType=Fortran&fileType=Header&fileType=C&fileType=XML&fileType=Resource&fileType=TLC&fileType=Makefile&fileType=MTF&fileType=Requirements&fileType=Shell+Scripts&fileType=Perl&fileType=Message+Table&fileType=Translation+Table&fileType=C%23&fileType=JavaScript&fileType=Visual+Basic&fileType=Chart&fileType=Configuration&sourceDir=java%2Fsrc&sourceDir=src&sourceDir=simulink%2Fsrc&sourceDir=test&sourceDir=toolbox&sourceDir=standalone&sourceDir=makerules&sourceDir=extern%2Finclude&sourceDir=resources&sourceDir=rtw&sourceDir=stateflow&sourceDir=config&sourceDir=foundation_libraries&sourceDir=install&sourceDir=pbr&indexId=1&indexDir= HTTP/1.1" 200 16379
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:06:35 -0500" "GET /srcsearch/ HTTP/1.1" 200 12851
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:08:41 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:12:11 -0500" "GET /srcsearch/SearchResults.do?searchTerm=blkexist HTTP/1.1" 200 23033
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:12:11 -0500" "GET /favicon.ico HTTP/1.1" 404 1022
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:13:42 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:18:37 -0500" "GET /srcsearch/ HTTP/1.1" 200 11271
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:18:49 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:18:58 -0500" "GET /srcsearch/SearchResults.do?searchTerm=qeVerifyWithPolling&searchField=TEXT&sort=PATH&fileType=M&sourceDir=test&indexId=0&indexDir= HTTP/1.1" 200 1140275
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:05:19:06 -0500" "GET /srcsearch/Preview.do?file=test%2Fdata%2Ftestmeas%2Finstrument%2Finterface%2FhSharedFcnPropsAsync.m&searchString=qeverifywithpolling&caseSensitive=false&indexId=17&indexDir= HTTP/1.1" 200 0I want to write a parser that will extract only the part that is after GET till the first space character. Example from the first one is:
"xxx.xx.xxx.xx" "NULL-AUTH-USER" "29/Sep/2011:04:48:32 -0500" "GET /srcsearch/SearchResults.do?searchTerm=newyork&fileType=Java HTTP/1.0" 200 27554Extracted part:
/srcsearch/SearchResults.do?searchTerm=newyork&fileType=Javaand the time stamp for the first one can be assumed as time=0
Then the others will be with respect to it. Which means:
The time stamp for the first request is :
29/Sep/2011:04:48:32The time stamp for the second request is :
29/Sep/2011:04:53:32So the delta is exactly 5 minutes. We will use this delta while sending request to the server.
Then append this URL to something like:
http://myapp.mycmp.com:8080 Once I get this data I want to send these URLs to the server with the time difference which we calculated.
NOTE: As far as I know, CURL can be used for this purpose. I work on UNIX environment.
Or if there is an easier way to do it rather than CURL, that would be fine too.
Let me summarize it as a sudo code:
For each line that starts with "xxx.xx.xxx.xx" (x is a number and it can be anything. and its format may change like xxx.x.xxxxx.xx) extract the time stamp and find the difference between the previous one.
If it is the first one then assign the current time or zero to it.
Then extract the part which is right after GET till the next space. But don't get the space before and after the extracted part.
then append this URL to something like:
http://myapp.mycmp.com:8080
and send this entire thing to the server.As a full short example:
send this request to the server
http://myapp.mycmp.com:8080/srcsearch/SearchResults.do?searchTerm=newyork&fileType=Javawait for 5 minutes
(because there is a 5 minute difference between the first two requests)
then send this one
http://myapp.mycmp.com:8080/srcsearch/SearchResults.do?searchTerm=newyork&fileType=Javaand the process continues till the end of the file
parparov
If the delta is zero, should the refetch be done at once?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
@parparov: my answer to your first question is yes. and thank you for your quick and prompt response.
I will try your code.
Thanks,
I will try your code.
Thanks,
ASKER
I haven't checked it yet but let's say I don't have LWP::Simple module
But let's say I found it and download it to my computer. Then how can I direct my Perl code to see this module?
Because, the perl is not locally installed on my machine.
Thanks,
But let's say I found it and download it to my computer. Then how can I direct my Perl code to see this module?
Because, the perl is not locally installed on my machine.
Thanks,