?
Solved

grep a log file within a specific time range

Posted on 2014-02-20
7
Medium Priority
?
2,637 Views
Last Modified: 2014-03-01
Hi,
my O.S. is Linux and I have a log file (aud.log), each line in the log is like this (I've more 10000 lines):

02\/11\/2014 12:01:36 +0200 - SUCCESS - gmail.com  - 12:01:36 - http - Lifer -  - id=aaaaa
02\/11\/2014 12:01:37 +0200 - SUCCESS - gmail.com  - 12:01:37 - http - Lifer -  - id=bbbb
02\/11\/2014 12:11:36 +0200 - FAIL - gmail.com  - 12:11:36 - http - Lifer -  - id=bbbb
02\/11\/2014 12:21:39 +0200 - SUCCESS - gmail.com  - 12:21:39 - http - Lifer -  - id=cc
02\/11\/2014 12:51:45 +0200 - SUCCESS - gmail.com  - 12:51:45 - http - Lifer -  - id=tt
.........................................................................................
.........................................................................................
02\/11\/2014 14:01:37 +0200 - SUCCESS - gmail.com  - 14:01:37 - http - Lifer -  - id=bb
02\/11\/2014 14:11:37 +0200 - SUCCESS - gmail.com  - 14:11:37 - http - Lifer -  - id=cccccccc
02\/11\/2014 14:31:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:02:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=xxxx
02\/11\/2014 16:12:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=ss
02\/11\/2014 16:22:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:32:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=tt
02\/11\/2014 17:52:37 +0200 - FAIL - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 20:52:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb

Open in new window


I want to count the number of occurrences with string "SUCCESS" happening from:
 02\/11\/2014 12:00 to 02\/11\/2014 19:00

 02\/11\/2014 16:02 to end of the file


Also I would like to count the number of occurrences in the file grouped by id, for example:

bb 4
tt 2
ss 1
....
....

I am having difficulty writing these grep, can you help me?

Thanks in advance!
0
Comment
Question by:ralph_rea
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 31

Accepted Solution

by:
farzanj earned 1200 total points
ID: 39873545
To
grep -P '02\\\/11\\\/2014 (?:1[2-8]:\d\d|19:00).*SUCCESS' aud.log

Open in new window


And you can use c to count, making your option -Pc


For second one, you need sed or perl.  If you try to do it with grep, regex would become too complicated.

You can do like
sed -n '/02\\\/11\\\/2014 16:02/, $'p aud.log | grep 'SUCCESS'

Open in new window



For your last request, you will be really better off with Perl
perl -ne '($id)=/id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 

Author Comment

by:ralph_rea
ID: 39874005
So if I want to count from 16:02 to 19:05 can I use sed command like this:

sed -n '/02\\\/11\\\/2014 16:02/, /02\\\/11\\\/2014 19:05/'p aud.log | grep 'SUCCESS'

Open in new window



correct?
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874024
Yes, but make sure both of these date values actually exist in the file.  For example, if you don't have 19:05, and have 19:04 and then 19:06, it would not know where to stop.  It works on exact match.
0
Does Your Cloud Backup Use Blockchain Technology?

Blockchain technology has already revolutionized finance thanks to Bitcoin. Now it's disrupting other areas, including the realm of data protection. Learn how blockchain is now being used to authenticate backup files and keep them safe from hackers.

 

Author Comment

by:ralph_rea
ID: 39874218
Ok, thanks
only one thing on perl script, I'd like to count the number of UID with string "SUCCESS", in your script you count any uid.

Thanks
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874641
Here you go

perl -ne '($id)=/SUCCESS.*id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 
LVL 27

Expert Comment

by:skullnobrains
ID: 39886425
your last query in shell
grep '^02\/11\/2014 1[2-8].*SUCCESS' | cut -d ' ' -f 16 | sort | uniq -c

there may be more efficient ways, namely replacing the grep+cut with a sed like mentioned above

if you are fluent in sed, you can easily produce outputs with a column for SUCCESS/FAILURE and a second for the id and sort+uniq them as you wish

sort can also act on a specific column using the -k parameter and is also able to sort numerically
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 800 total points
ID: 39886470
examples using your input as /tmp/x

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\1 \2/p' | sort | uniq -c | sort -n
      1 FAIL bb
      1 SUCCESS ss
      1 SUCCESS tt
      1 SUCCESS xxxx
      2 SUCCESS bb

Open in new window


or present the failures and then successed ordered by number of occurences

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\2 \1/p' | sort | uniq -c | sort -n | sed 's/[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)/\3 \1 \2/' | sort
FAIL 1 bb
SUCCESS 1 ss
SUCCESS 1 tt
SUCCESS 1 xxxx
SUCCESS 2 bb

Open in new window


... i guess you can figure out how to do what you need easily

note that awk is probably also a good tool for such things
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses

718 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question