Go Premium for a chance to win a PS4. Enter to Win

x
?
Solved

grep a log file within a specific time range

Posted on 2014-02-20
7
Medium Priority
?
2,827 Views
Last Modified: 2014-03-01
Hi,
my O.S. is Linux and I have a log file (aud.log), each line in the log is like this (I've more 10000 lines):

02\/11\/2014 12:01:36 +0200 - SUCCESS - gmail.com  - 12:01:36 - http - Lifer -  - id=aaaaa
02\/11\/2014 12:01:37 +0200 - SUCCESS - gmail.com  - 12:01:37 - http - Lifer -  - id=bbbb
02\/11\/2014 12:11:36 +0200 - FAIL - gmail.com  - 12:11:36 - http - Lifer -  - id=bbbb
02\/11\/2014 12:21:39 +0200 - SUCCESS - gmail.com  - 12:21:39 - http - Lifer -  - id=cc
02\/11\/2014 12:51:45 +0200 - SUCCESS - gmail.com  - 12:51:45 - http - Lifer -  - id=tt
.........................................................................................
.........................................................................................
02\/11\/2014 14:01:37 +0200 - SUCCESS - gmail.com  - 14:01:37 - http - Lifer -  - id=bb
02\/11\/2014 14:11:37 +0200 - SUCCESS - gmail.com  - 14:11:37 - http - Lifer -  - id=cccccccc
02\/11\/2014 14:31:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:02:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=xxxx
02\/11\/2014 16:12:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=ss
02\/11\/2014 16:22:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:32:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=tt
02\/11\/2014 17:52:37 +0200 - FAIL - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 20:52:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb

Open in new window


I want to count the number of occurrences with string "SUCCESS" happening from:
 02\/11\/2014 12:00 to 02\/11\/2014 19:00

 02\/11\/2014 16:02 to end of the file


Also I would like to count the number of occurrences in the file grouped by id, for example:

bb 4
tt 2
ss 1
....
....

I am having difficulty writing these grep, can you help me?

Thanks in advance!
0
Comment
Question by:ralph_rea
  • 3
  • 2
  • 2
7 Comments
 
LVL 31

Accepted Solution

by:
farzanj earned 1200 total points
ID: 39873545
To
grep -P '02\\\/11\\\/2014 (?:1[2-8]:\d\d|19:00).*SUCCESS' aud.log

Open in new window


And you can use c to count, making your option -Pc


For second one, you need sed or perl.  If you try to do it with grep, regex would become too complicated.

You can do like
sed -n '/02\\\/11\\\/2014 16:02/, $'p aud.log | grep 'SUCCESS'

Open in new window



For your last request, you will be really better off with Perl
perl -ne '($id)=/id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 

Author Comment

by:ralph_rea
ID: 39874005
So if I want to count from 16:02 to 19:05 can I use sed command like this:

sed -n '/02\\\/11\\\/2014 16:02/, /02\\\/11\\\/2014 19:05/'p aud.log | grep 'SUCCESS'

Open in new window



correct?
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874024
Yes, but make sure both of these date values actually exist in the file.  For example, if you don't have 19:05, and have 19:04 and then 19:06, it would not know where to stop.  It works on exact match.
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 

Author Comment

by:ralph_rea
ID: 39874218
Ok, thanks
only one thing on perl script, I'd like to count the number of UID with string "SUCCESS", in your script you count any uid.

Thanks
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874641
Here you go

perl -ne '($id)=/SUCCESS.*id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 
LVL 27

Expert Comment

by:skullnobrains
ID: 39886425
your last query in shell
grep '^02\/11\/2014 1[2-8].*SUCCESS' | cut -d ' ' -f 16 | sort | uniq -c

there may be more efficient ways, namely replacing the grep+cut with a sed like mentioned above

if you are fluent in sed, you can easily produce outputs with a column for SUCCESS/FAILURE and a second for the id and sort+uniq them as you wish

sort can also act on a specific column using the -k parameter and is also able to sort numerically
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 800 total points
ID: 39886470
examples using your input as /tmp/x

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\1 \2/p' | sort | uniq -c | sort -n
      1 FAIL bb
      1 SUCCESS ss
      1 SUCCESS tt
      1 SUCCESS xxxx
      2 SUCCESS bb

Open in new window


or present the failures and then successed ordered by number of occurences

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\2 \1/p' | sort | uniq -c | sort -n | sed 's/[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)/\3 \1 \2/' | sort
FAIL 1 bb
SUCCESS 1 ss
SUCCESS 1 tt
SUCCESS 1 xxxx
SUCCESS 2 bb

Open in new window


... i guess you can figure out how to do what you need easily

note that awk is probably also a good tool for such things
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
Suggested Courses
Course of the Month10 days, 7 hours left to enroll

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question