Avatar of jayatallen
jayatallen

asked on 

how to grep urls from apache access logs and again run them using wget - for testing purpose

hi Folks,

i want to run few urls against my application server for testing purpose.Basically these are urls from access logs and we want to test if these urls causing application slowness.
Is there any way i can grab the hits from access log and again invoke those urls using any kind of program or script.

Below is snippet from access log:

Open in new window




155.180.105.36 - - [09/Jul/2011:11:46:50 -0400] "GET /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29811
155.180.105.36 - - [09/Jul/2011:11:46:52 -0400] "GET /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
155.180.105.36 - - [09/Jul/2011:11:46:52 -0400] "GET /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
155.180.105.36 - - [09/Jul/2011:11:46:52 -0400] "GET /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
155.180.105.36 - - [09/Jul/2011:11:46:52 -0400] "GET /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732

Open in new window


I was thinking to do awk and grep the url but it missing the hostname.
so, is there any way to append hostname and wget before the url and then put it in a shell script and run  it .
something like

Open in new window

wget hostname /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
wget hostname /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
wget hostname /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
wget hostname /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
wget hostname  /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732
wget hostname  /cm/Satellite?pagename=CYQ/Href&urlname=cyqorgan/am/wateress/aboutus HTTP/1.1" 200 29732

Open in new window

Apache Web ServerShell Scripting

Avatar of undefined
Last Comment
jayatallen
Avatar of Kamaraj Subramanian
Kamaraj Subramanian
Flag of Singapore image

Try this.. not tested..

awk -F"\"" {printf("wget hostname %s %s\n",$2,$3)}' logfile

Open in new window

Avatar of jayatallen
jayatallen

ASKER

thank you your reply.
copied few lines from access log to a file named it test and then used given command

bash-3.00$ awk -F"\"" {printf("wget hostname %s %s\n",$2,$3)}' test
bash: syntax error near unexpected token `('
bash-3.00$

Please help.
ASKER CERTIFIED SOLUTION
Avatar of Kamaraj Subramanian
Kamaraj Subramanian
Flag of Singapore image

Blurred text
THIS SOLUTION IS ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
Avatar of jayatallen
jayatallen

ASKER

tried ...very close..


after runing given command, one of line from output  pasted below:
wget hostname GET /cm/Satellite?blobcol=urldata&blobheader=image%2Fjpeg&blobkey=id&blobtable=MungoBlobs&blobwhere=11586

any way we can change it to like:

wget hostname/cm/Satellite?blobcol=urldata&blobheader=image%2Fjpeg&blobkey=id&blobtable=MungoBlobs&blobwhere=11586

I mean how i can  remove GET and no spaces between hostname/cm.

thank you very much.
SOLUTION
Avatar of jayatallen
jayatallen

Blurred text
THIS SOLUTION IS ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
Good, if you got the answer, then please close this question :)
SOLUTION
Avatar of skullnobrains
skullnobrains

Blurred text
THIS SOLUTION IS ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
Avatar of jayatallen
jayatallen

ASKER

answer provided by Guru helped..i just copied it.
Apache Web Server
Apache Web Server

The Apache HTTP Server is a secure, efficient and extensible server that provides HTTP services in sync with the current HTTP standards. Typically Apache is run on a Unix-like operating system, but it is available for a wide variety of operating systems, including Linux, Novell NetWare, Mac OS-X and Windows. Released under the Apache License, Apache is open-source software.

21K
Questions
--
Followers
--
Top Experts
Get a personalized solution from industry experts
Ask the experts
Read over 600 more reviews

TRUSTED BY

IBM logoIntel logoMicrosoft logoUbisoft logoSAP logo
Qualcomm logoCitrix Systems logoWorkday logoErnst & Young logo
High performer badgeUsers love us badge
LinkedIn logoFacebook logoX logoInstagram logoTikTok logoYouTube logo