Analyse a file not beginning each time at the start

Posted on 2004-11-18
Last Modified: 2010-04-22
I need a command (tail, or a shell script) to analyse a large LOG file (100 Mo).

I check this file every 5 minutes, and i don't want to re-read each time from the beginning of the file, but beginning at the last line analyzed before.

Anyone could help ?
Question by:eeolivier

    Author Comment

    Let me add that i can only read the LOG file, and that the content is increased of about 50/100 lines each seconds !
    LVL 6

    Expert Comment

    One option is to use the tail command like this:

    tail -f /your/log/file

    From the man page:
          -f, --follow[={name|descriptor}]
                  output appended data as the file grows; -f, --follow, and --fol-
                  low=descriptor are equivalent

    Author Comment

    No, that wasn't my question.

    I need a command that return me, when i call it the last lines since i check it last times.

    It's for use then in a log analyser script.
    LVL 48

    Expert Comment

    Any chance of changing the method your log analyser script uses?  Running tail -f is the easiest way if you are flexible in the way you input/analyse the data.
    LVL 6

    Expert Comment

    Well, you could do something like this:

    tail -f /your/log/file | \
    while read xl
       # code to determine outfile name here.
       echo $xl >> $outfile

    Add some code in the while loop that alters the name of the outfile every so often, say every so many lines or every so many times.
    That way, your log file is being split up into separate files that you can analyze at your leisure.
    LVL 45

    Expert Comment

    Assume that tempfile contains the last line number in the logfile that you read. You will have write the first entry yourself

    first=`cat tempfile`
    second=`wc -l logfile`

    d=`expr $second - $first`

    tail -n $d logfile

    echo $second > tempfile

    I have not tried the script but it should be pretty close to what you are looking for.

    Author Comment

    It don't work sunnycode !!

    I obtain this in my tempile : 3 /usr/logger/stat/SCVRU.all.041207.csv
    (instead of only 3), and so an error

    expr: syntax error
    tail: /usr/logger/stat/SCVRU.all.041207.csv: invalid number of lines

    it's the wc -l that return : 3 /usr/logger/stat/SCVRU.all.041207.csv
    LVL 48

    Expert Comment

    *If* you wanted to use sunnycoder's script, you'd need to correct:

    second=`wc -l <logfile`


    Accepted Solution

    PAQed with no points refunded (of 350)

    Community Support Moderator

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    IT, Stop Being Called Into Every Meeting

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    The purpose of this article is to fix the unknown display problem in Linux Mint operating system. After installing the OS if you see Display monitor is not recognized then we can install "MESA" utilities to fix this problem or we can install additio…
    The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
    how to add IIS SMTP to handle application/Scanner relays into office 365.
    This video discusses moving either the default database or any database to a new volume.

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    22 Experts available now in Live!

    Get 1:1 Help Now