Solved

grep a log file within a specific time range

Posted on 2014-02-20
7
1,654 Views
Last Modified: 2014-03-01
Hi,
my O.S. is Linux and I have a log file (aud.log), each line in the log is like this (I've more 10000 lines):

02\/11\/2014 12:01:36 +0200 - SUCCESS - gmail.com  - 12:01:36 - http - Lifer -  - id=aaaaa
02\/11\/2014 12:01:37 +0200 - SUCCESS - gmail.com  - 12:01:37 - http - Lifer -  - id=bbbb
02\/11\/2014 12:11:36 +0200 - FAIL - gmail.com  - 12:11:36 - http - Lifer -  - id=bbbb
02\/11\/2014 12:21:39 +0200 - SUCCESS - gmail.com  - 12:21:39 - http - Lifer -  - id=cc
02\/11\/2014 12:51:45 +0200 - SUCCESS - gmail.com  - 12:51:45 - http - Lifer -  - id=tt
.........................................................................................
.........................................................................................
02\/11\/2014 14:01:37 +0200 - SUCCESS - gmail.com  - 14:01:37 - http - Lifer -  - id=bb
02\/11\/2014 14:11:37 +0200 - SUCCESS - gmail.com  - 14:11:37 - http - Lifer -  - id=cccccccc
02\/11\/2014 14:31:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:02:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=xxxx
02\/11\/2014 16:12:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=ss
02\/11\/2014 16:22:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:32:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=tt
02\/11\/2014 17:52:37 +0200 - FAIL - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 20:52:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb

Open in new window


I want to count the number of occurrences with string "SUCCESS" happening from:
 02\/11\/2014 12:00 to 02\/11\/2014 19:00

 02\/11\/2014 16:02 to end of the file


Also I would like to count the number of occurrences in the file grouped by id, for example:

bb 4
tt 2
ss 1
....
....

I am having difficulty writing these grep, can you help me?

Thanks in advance!
0
Comment
Question by:ralph_rea
  • 3
  • 2
  • 2
7 Comments
 
LVL 31

Accepted Solution

by:
farzanj earned 300 total points
Comment Utility
To
grep -P '02\\\/11\\\/2014 (?:1[2-8]:\d\d|19:00).*SUCCESS' aud.log

Open in new window


And you can use c to count, making your option -Pc


For second one, you need sed or perl.  If you try to do it with grep, regex would become too complicated.

You can do like
sed -n '/02\\\/11\\\/2014 16:02/, $'p aud.log | grep 'SUCCESS'

Open in new window



For your last request, you will be really better off with Perl
perl -ne '($id)=/id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 

Author Comment

by:ralph_rea
Comment Utility
So if I want to count from 16:02 to 19:05 can I use sed command like this:

sed -n '/02\\\/11\\\/2014 16:02/, /02\\\/11\\\/2014 19:05/'p aud.log | grep 'SUCCESS'

Open in new window



correct?
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
Yes, but make sure both of these date values actually exist in the file.  For example, if you don't have 19:05, and have 19:04 and then 19:06, it would not know where to stop.  It works on exact match.
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 

Author Comment

by:ralph_rea
Comment Utility
Ok, thanks
only one thing on perl script, I'd like to count the number of UID with string "SUCCESS", in your script you count any uid.

Thanks
0
 
LVL 31

Expert Comment

by:farzanj
Comment Utility
Here you go

perl -ne '($id)=/SUCCESS.*id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 
LVL 26

Expert Comment

by:skullnobrains
Comment Utility
your last query in shell
grep '^02\/11\/2014 1[2-8].*SUCCESS' | cut -d ' ' -f 16 | sort | uniq -c

there may be more efficient ways, namely replacing the grep+cut with a sed like mentioned above

if you are fluent in sed, you can easily produce outputs with a column for SUCCESS/FAILURE and a second for the id and sort+uniq them as you wish

sort can also act on a specific column using the -k parameter and is also able to sort numerically
0
 
LVL 26

Assisted Solution

by:skullnobrains
skullnobrains earned 200 total points
Comment Utility
examples using your input as /tmp/x

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\1 \2/p' | sort | uniq -c | sort -n
      1 FAIL bb
      1 SUCCESS ss
      1 SUCCESS tt
      1 SUCCESS xxxx
      2 SUCCESS bb

Open in new window


or present the failures and then successed ordered by number of occurences

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\2 \1/p' | sort | uniq -c | sort -n | sed 's/[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)/\3 \1 \2/' | sort
FAIL 1 bb
SUCCESS 1 ss
SUCCESS 1 tt
SUCCESS 1 xxxx
SUCCESS 2 bb

Open in new window


... i guess you can figure out how to do what you need easily

note that awk is probably also a good tool for such things
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Suggested Solutions

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now