Solved

grep a log file within a specific time range

Posted on 2014-02-20
7
2,172 Views
Last Modified: 2014-03-01
Hi,
my O.S. is Linux and I have a log file (aud.log), each line in the log is like this (I've more 10000 lines):

02\/11\/2014 12:01:36 +0200 - SUCCESS - gmail.com  - 12:01:36 - http - Lifer -  - id=aaaaa
02\/11\/2014 12:01:37 +0200 - SUCCESS - gmail.com  - 12:01:37 - http - Lifer -  - id=bbbb
02\/11\/2014 12:11:36 +0200 - FAIL - gmail.com  - 12:11:36 - http - Lifer -  - id=bbbb
02\/11\/2014 12:21:39 +0200 - SUCCESS - gmail.com  - 12:21:39 - http - Lifer -  - id=cc
02\/11\/2014 12:51:45 +0200 - SUCCESS - gmail.com  - 12:51:45 - http - Lifer -  - id=tt
.........................................................................................
.........................................................................................
02\/11\/2014 14:01:37 +0200 - SUCCESS - gmail.com  - 14:01:37 - http - Lifer -  - id=bb
02\/11\/2014 14:11:37 +0200 - SUCCESS - gmail.com  - 14:11:37 - http - Lifer -  - id=cccccccc
02\/11\/2014 14:31:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:02:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=xxxx
02\/11\/2014 16:12:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=ss
02\/11\/2014 16:22:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:32:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=tt
02\/11\/2014 17:52:37 +0200 - FAIL - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 20:52:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb

Open in new window


I want to count the number of occurrences with string "SUCCESS" happening from:
 02\/11\/2014 12:00 to 02\/11\/2014 19:00

 02\/11\/2014 16:02 to end of the file


Also I would like to count the number of occurrences in the file grouped by id, for example:

bb 4
tt 2
ss 1
....
....

I am having difficulty writing these grep, can you help me?

Thanks in advance!
0
Comment
Question by:ralph_rea
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 31

Accepted Solution

by:
farzanj earned 300 total points
ID: 39873545
To
grep -P '02\\\/11\\\/2014 (?:1[2-8]:\d\d|19:00).*SUCCESS' aud.log

Open in new window


And you can use c to count, making your option -Pc


For second one, you need sed or perl.  If you try to do it with grep, regex would become too complicated.

You can do like
sed -n '/02\\\/11\\\/2014 16:02/, $'p aud.log | grep 'SUCCESS'

Open in new window



For your last request, you will be really better off with Perl
perl -ne '($id)=/id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 

Author Comment

by:ralph_rea
ID: 39874005
So if I want to count from 16:02 to 19:05 can I use sed command like this:

sed -n '/02\\\/11\\\/2014 16:02/, /02\\\/11\\\/2014 19:05/'p aud.log | grep 'SUCCESS'

Open in new window



correct?
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874024
Yes, but make sure both of these date values actually exist in the file.  For example, if you don't have 19:05, and have 19:04 and then 19:06, it would not know where to stop.  It works on exact match.
0
NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

 

Author Comment

by:ralph_rea
ID: 39874218
Ok, thanks
only one thing on perl script, I'd like to count the number of UID with string "SUCCESS", in your script you count any uid.

Thanks
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874641
Here you go

perl -ne '($id)=/SUCCESS.*id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 
LVL 27

Expert Comment

by:skullnobrains
ID: 39886425
your last query in shell
grep '^02\/11\/2014 1[2-8].*SUCCESS' | cut -d ' ' -f 16 | sort | uniq -c

there may be more efficient ways, namely replacing the grep+cut with a sed like mentioned above

if you are fluent in sed, you can easily produce outputs with a column for SUCCESS/FAILURE and a second for the id and sort+uniq them as you wish

sort can also act on a specific column using the -k parameter and is also able to sort numerically
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 200 total points
ID: 39886470
examples using your input as /tmp/x

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\1 \2/p' | sort | uniq -c | sort -n
      1 FAIL bb
      1 SUCCESS ss
      1 SUCCESS tt
      1 SUCCESS xxxx
      2 SUCCESS bb

Open in new window


or present the failures and then successed ordered by number of occurences

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\2 \1/p' | sort | uniq -c | sort -n | sed 's/[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)/\3 \1 \2/' | sort
FAIL 1 bb
SUCCESS 1 ss
SUCCESS 1 tt
SUCCESS 1 xxxx
SUCCESS 2 bb

Open in new window


... i guess you can figure out how to do what you need easily

note that awk is probably also a good tool for such things
0

Featured Post

Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
spectrum scale snapshot  resotre/mount 1 58
LINUX Field Separators 7 78
Advice on ESXi 5.1 Health / Storage 1 76
How many users could squid support? 21 57
Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses

739 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question