linux bash script problem

Posted on 2006-04-17
Last Modified: 2010-04-22
Hi all.

I know I should learn more bash programming and I want to and I will but right now, I'm somewhat ashamed to admit that I need help and
haven't got the time to learn what I need to know.

so, my problem:

I have an application that outputs a log file (rotated daily) of the format:

17-Apr-2006 20:16:28 [proc1] [INFO ] [mainapp] 15845 Request:
17-Apr-2006 20:16:28 [proc1] [INFO ] [mainapp] 15845 Info: blah/blah1/blah2/blah3
17-Apr-2006 20:16:29 [proc1] [INFO ] [mainapp] 15845 Response: 00000000000000000000000000005Ev2 - OK blah/blah1/blah2/blah3

Each successful process on my application results in a set of three lines of code that go into /logs/blah.log, each set of three log entries is linked by the id number, in this case 15845 (names, processes and etc changed to protect the innocent and copyrighted..). However, sometimes my application crashes and in that circumstance, it only writes lines 1 and 2.  

I need a script that will tail, say, the last 50 lines of the file, check that

how the script does this, I dont know.

maybe it will get the order number from the file through grep/awk then grep the log file for that order number (surrounded by spaces) and pipe it through "wc -l" (but this would have a problem if the system was in the middle of processing something and was half way through a set of "three log entries").

I dunno.

any ideas please?

thanks v much for any help offered.

Question by:daryn
    LVL 29

    Expert Comment

    This could be done with a simple awk program, but your problem actually resides here:

    >>this would have a problem if the system was in the middle of processing something and was half way through a set of "three log entries".

    Unless you ignore the "first" group after the "tail" command and the last group at the end of the file.

    LVL 84

    Expert Comment

    perl -ane '$id{$F[5]}++;END{for(keys %id){ print "$_ only $id{$_} lines\n" if $id{$_}<3 } }' /logs/blah.log
    LVL 29

    Expert Comment


    -- OR --
    tail -50 /logs/blah.log|\
    awk 'BEGIN {n=0;f=0}
    { if (n == 0) k=$7;
      print k " " NR " - " $7 ":" $8 " f" f " n" n;
      if ( k == $7 ) n=n+1
      else {
        if ( f == 1 && n < 3 ) {print k " Missing line. " f;}
      k = $7;
    LVL 84

    Expert Comment

    Are all ids with the same number guaranteed to be together in the log file, and not mixed with log entries from other ids?

    Author Comment

    thats the thing, (due to my not mentioning this somewhat crucial fact) neither your perl script not the bash script from MikeOM_DBA works too well since the id numbers are not sequential, they can be mixed up, ie 2 lines re: 15845, then a line for 15846, then the final line for 15845, then two lines of 15847, then the final two lines of 15846 etc...

    apologies for missing out that fact, I've got a stinking cold and no sleep.. :)

    LVL 51

    Accepted Solution

    awk '(NR>6){a[$6]++}END{for(n in a){if(a[n]<3){print "bad "n}}}' /logs/blah.log

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How to improve team productivity

    Quip adds documents, spreadsheets, and tasklists to your Slack experience
    - Elevate ideas to Quip docs
    - Share Quip docs in Slack
    - Get notified of changes to your docs
    - Available on iOS/Android/Desktop/Web
    - Online/Offline

    The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
    The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
    how to add IIS SMTP to handle application/Scanner relays into office 365.
    Here's a very brief overview of the methods PRTG Network Monitor ( offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

    794 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now