Solved

grep a log file within a specific time range

Posted on 2014-02-20
7
2,414 Views
Last Modified: 2014-03-01
Hi,
my O.S. is Linux and I have a log file (aud.log), each line in the log is like this (I've more 10000 lines):

02\/11\/2014 12:01:36 +0200 - SUCCESS - gmail.com  - 12:01:36 - http - Lifer -  - id=aaaaa
02\/11\/2014 12:01:37 +0200 - SUCCESS - gmail.com  - 12:01:37 - http - Lifer -  - id=bbbb
02\/11\/2014 12:11:36 +0200 - FAIL - gmail.com  - 12:11:36 - http - Lifer -  - id=bbbb
02\/11\/2014 12:21:39 +0200 - SUCCESS - gmail.com  - 12:21:39 - http - Lifer -  - id=cc
02\/11\/2014 12:51:45 +0200 - SUCCESS - gmail.com  - 12:51:45 - http - Lifer -  - id=tt
.........................................................................................
.........................................................................................
02\/11\/2014 14:01:37 +0200 - SUCCESS - gmail.com  - 14:01:37 - http - Lifer -  - id=bb
02\/11\/2014 14:11:37 +0200 - SUCCESS - gmail.com  - 14:11:37 - http - Lifer -  - id=cccccccc
02\/11\/2014 14:31:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:02:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=xxxx
02\/11\/2014 16:12:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=ss
02\/11\/2014 16:22:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 16:32:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=tt
02\/11\/2014 17:52:37 +0200 - FAIL - gmail.com  - 14:31:37 - http - Lifer -  - id=bb
02\/11\/2014 20:52:37 +0200 - SUCCESS - gmail.com  - 14:31:37 - http - Lifer -  - id=bb

Open in new window


I want to count the number of occurrences with string "SUCCESS" happening from:
 02\/11\/2014 12:00 to 02\/11\/2014 19:00

 02\/11\/2014 16:02 to end of the file


Also I would like to count the number of occurrences in the file grouped by id, for example:

bb 4
tt 2
ss 1
....
....

I am having difficulty writing these grep, can you help me?

Thanks in advance!
0
Comment
Question by:ralph_rea
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
  • 2
7 Comments
 
LVL 31

Accepted Solution

by:
farzanj earned 300 total points
ID: 39873545
To
grep -P '02\\\/11\\\/2014 (?:1[2-8]:\d\d|19:00).*SUCCESS' aud.log

Open in new window


And you can use c to count, making your option -Pc


For second one, you need sed or perl.  If you try to do it with grep, regex would become too complicated.

You can do like
sed -n '/02\\\/11\\\/2014 16:02/, $'p aud.log | grep 'SUCCESS'

Open in new window



For your last request, you will be really better off with Perl
perl -ne '($id)=/id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 

Author Comment

by:ralph_rea
ID: 39874005
So if I want to count from 16:02 to 19:05 can I use sed command like this:

sed -n '/02\\\/11\\\/2014 16:02/, /02\\\/11\\\/2014 19:05/'p aud.log | grep 'SUCCESS'

Open in new window



correct?
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874024
Yes, but make sure both of these date values actually exist in the file.  For example, if you don't have 19:05, and have 19:04 and then 19:06, it would not know where to stop.  It works on exact match.
0
 [eBook] Windows Nano Server

Download this FREE eBook and learn all you need to get started with Windows Nano Server, including deployment options, remote management
and troubleshooting tips and tricks

 

Author Comment

by:ralph_rea
ID: 39874218
Ok, thanks
only one thing on perl script, I'd like to count the number of UID with string "SUCCESS", in your script you count any uid.

Thanks
0
 
LVL 31

Expert Comment

by:farzanj
ID: 39874641
Here you go

perl -ne '($id)=/SUCCESS.*id=(\w+)/;$ct{$id}++ if($id);END{foreach (keys %ct) { print $_, " ", $ct{$_},"\n";}}' aud.log

Open in new window

0
 
LVL 27

Expert Comment

by:skullnobrains
ID: 39886425
your last query in shell
grep '^02\/11\/2014 1[2-8].*SUCCESS' | cut -d ' ' -f 16 | sort | uniq -c

there may be more efficient ways, namely replacing the grep+cut with a sed like mentioned above

if you are fluent in sed, you can easily produce outputs with a column for SUCCESS/FAILURE and a second for the id and sort+uniq them as you wish

sort can also act on a specific column using the -k parameter and is also able to sort numerically
0
 
LVL 27

Assisted Solution

by:skullnobrains
skullnobrains earned 200 total points
ID: 39886470
examples using your input as /tmp/x

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\1 \2/p' | sort | uniq -c | sort -n
      1 FAIL bb
      1 SUCCESS ss
      1 SUCCESS tt
      1 SUCCESS xxxx
      2 SUCCESS bb

Open in new window


or present the failures and then successed ordered by number of occurences

$ cat /tmp/x | sed -n '/02\\\/11\\\/2014 16:02/,/02\\\/11\\\/2014 19:05/ s/.*\(SUCCESS\|FAIL\).*id=\(.*\)$/\2 \1/p' | sort | uniq -c | sort -n | sed 's/[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)[[:space:]]*\([^[:space:]]*\)/\3 \1 \2/' | sort
FAIL 1 bb
SUCCESS 1 ss
SUCCESS 1 tt
SUCCESS 1 xxxx
SUCCESS 2 bb

Open in new window


... i guess you can figure out how to do what you need easily

note that awk is probably also a good tool for such things
0

Featured Post

NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Active Directory replication delay is the cause to many problems.  Here is a super easy script to force Active Directory replication to all sites with by using an elevated PowerShell command prompt, and a tool to verify your changes.
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Suggested Courses

636 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question