[Last Call] Learn about multicloud storage options and how to improve your company's cloud strategy. Register Now


Shell Script for grepping log files.

Posted on 2010-09-16
Medium Priority
Last Modified: 2012-05-10

I have a log file like the following:

Inbound Message:

Please take a look at the initialOrderRef which I need to grep.

FrontEnd can be any of the following types


2010-09-16 00:41:16,726 INFO frontend IN: entering fromMessage [<com...ConfirmedRequest>
  <user class="com....LoggedInUser">



2010-09-16 07:31:46,430 INFO frontend IN: entering fromMessage [<com...MessageImpl>
  <payload class="com...OrderImpl">
  <user class="com....LoggedInUser">


Outbound Message from the backend will be like this:

2010-09-16 00:41:16,751 INFO backend-outbound : exiting fromMessage [[MasterTransferObject: message = <
common_fields :
order_fields :
      broker : ...
initial_order_ref : 0001945520100915234138000000000092227285
> attachments <>]]

The log mentioned above is for one message.

In between the frontend and backend I have mentioned above there can be frontend exiting message and some components exiting message and so on and so forth.

Something like this:

2010-09-16 07:31:46,440 INFO frontend : entering toMessage [[MasterTransferObject: message = <
type : NEW_ORDER
common_fields :
order_fields :
      held : false
      confirm_validation : false
      validation_mask : 1
      broker : ...
initial_order_ref : 0001945520100915234138000000000092227285
> attachments <>]]

But, what I am looking for is the time taken from frontend entering to backend exiting fromMessage based upon initial_order_ref for all the orders.

This is the order log for one message. There can be more than one order log at the same time because this is multi-threaded program.

So, there can be frontend log for one order and frontend log for 10 orders in between and backend log for the first message.

I need to calculate the time taken for each order to go from frontend to backend based upon the initial_order_ref.

How can I do that?
Please suggest Shell script required.
Question by:srikanthrad
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
LVL 80

Expert Comment

ID: 33696678
Your best might be to use perl with XMLParser module.


Or using perl to process the lines and extracting data.

Author Comment

ID: 33701515
If I were to do using Java. Can you please give me an idea of how to do it?

I am thinking like first I will grep for first occurrence of "INFO frontend IN: entering fromMessage" and then pass it as a parameter to program.

It will start scanning from the line passed as a parameter, then for each occurrence of initial_order_ref/initialOrderRef it will construct an object which contains the start time/end time while parsing the log file.

While parsing each line, if it sees frontEnd/backend it will record the time into the start/endTime and then based upon the initialOrderRef, if the object is not present with that initialOrderRef it will construct one and assign startTime to that object. If the object is already present means, startTime is already assigned and it will assign endTime to the object.

long initialOrderRef
Date startTime
Date endTime

What do you think of that idea? If you have any better ideas, please suggest.
LVL 80

Expert Comment

ID: 33702436
Unfortunately, I do not have sufficient background in java.

You do not want to use grep unless you are certain that the log entries are all one liners from date to date.
It is possible that these are one liners or perhaps an erroneous enter is included in the data which will break.
You have to sequentially get a line at a time checking whether there is a date at the beginning of the line which indicates an end to the old record and the beginning of the new record.  Do you need data from within the <> tags such as <user class=somedesignation>username</user> or the item you care about is username only?

If you doing java it is the same process, you have to treat the data as an xml stream.

XML like parser might be the best option because you have variations of how the data can be present. The other part is that the pattern might change which will break a process that uses explicit checks.
I.e. instead of two variations, you now have three. with XML parsing, provided the change conforms/maintains the XML structure, the parsing part will have an additional parameter, while an explicit processing mechanism will ignore and discard that information.
cat logfile | perl -e '$variable=''; while (<STDIN>) {
chomp(); #strip the crlf from the line if any
if (length($variable)>=1 && /^[0-9]+\-[[0-9]+/) {  #the inclusion of the date in the log means the end of the prior record.
#the variable $variable contains the entire data
crunch_data($variable); #process the data
$variable .=$_;

sub crunch_data {
 my (scalar $data)=@_;
#here you have to extract the data from the string


Open in new window


Accepted Solution

rumi78 earned 2000 total points
ID: 33708983
Here you have your solution script:

time_pattern="\([0-9]\{4\}\).\([0-9][0-9]\).\([0-9][0-9]\) \([0-9][0-9]\).\([0-9][0-9]\).\([0-9][0-9]\).\([0-9][0-9][0-9]\) "
start_pattern=" frontend IN: entering fromMessage "
end_pattern=" backend-outbound : exiting fromMessage "

#append newline mark on single log entry
sed 's/\('^"$time_pattern"'.*\)/'$newLine'\1/' | \
        #remove all new line chars
        tr -d '\n' | \
        #replace newline mark with new line (single entry is now single line)
        tr "$newLine" '\n' | \
        #remove other than start/end entries
        egrep "$start_pattern|$end_pattern" | \
        #convert date and time to seconds
        sed 's/^'"$time_pattern"'\(.*\)/\1 \2 \3 \4 \5 \6 \7 \0/g' | \
        awk '{"date +%s -d "$1"/"$2"/"$3"\\ "$4":"$5":"$6|getline x ; close("date +%s -d "$1"/"$2"/"$3"\\ "$4":"$5":"$6); x+=$7/1000;printf("%0.3f ", x);for(i=8;i<=NF;i++){printf("%s ",$i)}printf("\n")}' | \
        #format line: <identifier> <time> start/end <original line>
        sed 's/^\([^ ]\+\) .*'"$start_pattern"'.*'"$identifier_pattern"'.*/\3 \1 start \0/g' | \
        sed 's/^\([^ ]\+\) .*'"$end_pattern"'.*'"$identifier_pattern"'.*/\3 \1 end \0/g' | \
        #print time difference
        awk '$3=="start"{line[$1]=$0;start[$1]=$2}$3=="end"{printf("%0.3f %s %s\n", $2-start[$1],line[$1],$0);delete start[$1];delete line[$1];}'

You have to redirect log to the script, e.g.
cat your_log_file.txt | ./times.sh

Adjust start_pattern/end_pattern/identifier_patter if you need.

The script is extremely fast. You will have to buy me a beer ;)
The output contains the time difference between start and end operation in seconds as a first column, and full log entries to allow you check start and end entries matches.
Of course you can then sort the result by first column to find the longest operation, count statistics for average operation time, etc.

If there is no output, or other problems put line:
to see the output at chosen level, e.g. after tr -d '\n' | \


Author Closing Comment

ID: 33709810
If you are in Chicago, I will definitely buy you a beer. Those runes in the script are working fine and Gods are pleased.

I am trying to understand the runes that you have written, but absolutely makes no sense or I am feeling really sleepy and need to look into those runes in the morning.

I am trying echoes after every line.

Thank you very much.

Featured Post

Learn how to optimize MySQL for your business need

With the increasing importance of apps & networks in both business & personal interconnections, perfor. has become one of the key metrics of successful communication. This ebook is a hands-on business-case-driven guide to understanding MySQL query parameter tuning & database perf

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
The Windows functions GetTickCount and timeGetTime retrieve the number of milliseconds since the system was started. However, the value is stored in a DWORD, which means that it wraps around to zero every 49.7 days. This article shows how to solve t…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …
Suggested Courses

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question