[Last Call] Learn how to a build a cloud-first strategyRegister Now

  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 234
  • Last Modified:

linux bash script problem

Hi all.

I know I should learn more bash programming and I want to and I will but right now, I'm somewhat ashamed to admit that I need help and
haven't got the time to learn what I need to know.

so, my problem:

I have an application that outputs a log file (rotated daily) of the format:

17-Apr-2006 20:16:28 [proc1] [INFO ] [mainapp] 15845 Request: xxx.xxx.xxx.xxx
17-Apr-2006 20:16:28 [proc1] [INFO ] [mainapp] 15845 Info: blah/blah1/blah2/blah3
17-Apr-2006 20:16:29 [proc1] [INFO ] [mainapp] 15845 Response: 00000000000000000000000000005Ev2 - OK blah/blah1/blah2/blah3

Each successful process on my application results in a set of three lines of code that go into /logs/blah.log, each set of three log entries is linked by the id number, in this case 15845 (names, processes and etc changed to protect the innocent and copyrighted..). However, sometimes my application crashes and in that circumstance, it only writes lines 1 and 2.  

I need a script that will tail, say, the last 50 lines of the file, check that

how the script does this, I dont know.

maybe it will get the order number from the file through grep/awk then grep the log file for that order number (surrounded by spaces) and pipe it through "wc -l" (but this would have a problem if the system was in the middle of processing something and was half way through a set of "three log entries").

I dunno.

any ideas please?

thanks v much for any help offered.

1 Solution
This could be done with a simple awk program, but your problem actually resides here:

>>this would have a problem if the system was in the middle of processing something and was half way through a set of "three log entries".

Unless you ignore the "first" group after the "tail" command and the last group at the end of the file.

perl -ane '$id{$F[5]}++;END{for(keys %id){ print "$_ only $id{$_} lines\n" if $id{$_}<3 } }' /logs/blah.log

-- OR --
tail -50 /logs/blah.log|\
awk 'BEGIN {n=0;f=0}
{ if (n == 0) k=$7;
  print k " " NR " - " $7 ":" $8 " f" f " n" n;
  if ( k == $7 ) n=n+1
  else {
    if ( f == 1 && n < 3 ) {print k " Missing line. " f;}
  k = $7;
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Are all ids with the same number guaranteed to be together in the log file, and not mixed with log entries from other ids?
darynAuthor Commented:
thats the thing, (due to my not mentioning this somewhat crucial fact) neither your perl script not the bash script from MikeOM_DBA works too well since the id numbers are not sequential, they can be mixed up, ie 2 lines re: 15845, then a line for 15846, then the final line for 15845, then two lines of 15847, then the final two lines of 15846 etc...

apologies for missing out that fact, I've got a stinking cold and no sleep.. :)

awk '(NR>6){a[$6]++}END{for(n in a){if(a[n]<3){print "bad "n}}}' /logs/blah.log

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now