We help IT Professionals succeed at work.

Extract Log Information with programming

Last Modified: 2010-04-20
I am running Squid Guard on a Debian server.  I inherited this configuration and am not a programmer, although I understand some basic concepts.  I am hoping to search a log and count the number of times a website was visited.  Two typical log entries looks like:

2007-01-30 18:06:00 [3688] Request(computers/movies/-) http://disney.go.com/ - GET
2007-01-31 13:32:05 [13114] Request(computers/movies/-) http://video.msn.com/js/vp.js - GET

Ideally the result would be a list of websites with a number next to it for number of visists.  I do not care what exactly was visited after the ".com".  For the above example, the output file I'm looking for is:

disney.go.com - 1
video.msn.com - 1

Thanks for your help.
Watch Question

This one is on us!
(Get your first solution completely free - no credit card required)

I'd try the searching externally, from a shell prompt...

 perl -e 'while (<>) { $v{$6}++ if /^(\S+)\s* (\S+)\s* (\S+)\s* (\S+)\(([^)]+)\)\ \s*[a-z]+:
\/\/([^ \/]+)\//; }  printf "  %-30s  - %-3d\n", $_, $v{$_} for(keys %v);'     PATH_TO_LOGFILE

Where   PATH_TO_LOGFILE   is replaced with the path to the actual logfile
(this is all on one line)


The perl script better captures the format I am after.

For the perl script, two questions:

How would I put this is in a file that I could call upon by name, like countlog.pl?

Can it sort the output greatest to least for the most visited sites as part of the script?



Well, without more info on perl I used veedar's answer to get the job done.  Thanks for the help.

Geeze.  Give some time to respond.  You can just throw the contents inside the quotation
marks into a file or save this to a filename.pl and then   chmod +x  filename.pl...


while (<>) {
$v{$6}++ if /^(\S+)\s* (\S+)\s* (\S+)\s* (\S+)\(([^)]+)\)\ \s*[a-z]+:\/\/([^ \/]+)\//;
printf "  %-30s  - %-3d\n", $_, $v{$_} for(keys %v);
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.