Solved

Need a shell script (ksh) that uses an Awk array to process an input file

Posted on 2012-03-15
4
509 Views
Last Modified: 2016-02-11
Hello Experts,

I am looking for a shell script (preferabally KSH) which processes a large input file (a log file) and provides an output file in a certain format. I want to use an Awk array to process each line of the input file and print the required output feilds to a file.

I need an array, because there are various WebContainer Types and they have different Broker transactions within them.

Thanks,

-Katewadi

####################################
Sample Input file data:

2012-03-15 05:50:09,916 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform starting command <AddUpdateProgramGroup> -------------------
 05:50:10,043 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,215 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 165 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,219 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,390 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 168 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,392 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform command <AddUpdateProgramGroup> done in 476 milliseconds -------------------

##################################
Format of the Output File:

Date|Start-Time|WebContainer-Number|Type|End-Time|Broker-Time|Total-Transaction-Time
2012-03-15|05:50:09,916|2|AddUpdateProgramGroup|05:50:10,392|333|476
0
Comment
Question by:Katewadi
  • 2
4 Comments
 
LVL 61

Assisted Solution

by:gheist
gheist earned 50 total points
ID: 37728601
GNU awk would allow to customize row and field separators so that you can have multiline records parsed. Default BWK awk in FreeBSD does not do what you like, you need to install GNU AWK via # pkg_add -r gawk
https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html
0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 450 total points
ID: 37731323
# traditional awk:
awk '/basicPerform starting command/{d=$1;t=$2;w=substr($5,1,1);y=substr($8,2,length($8)-2);b=0;next}/Broker Transaction completed in/{b+=$(NF-3);next}/basicPerform command.* done in/{e=$2;m=$(NF-2);printf "%s|%s|%s|%s|%s|%s|%s\n",d,t,w,y,e,b,m;next}'  your-file
0
 

Author Comment

by:Katewadi
ID: 37738642
Ahoffmann,

Thanks for your input...

BTW, I tried the awk command (as-is), and apparently it did not work for me.

It gave me an output like this:

ontainer|:|C|-----------------|06:15:13,220|622|in

Am I doing anything wrong? Most of the output feilds do not look as expected -- including the first and last feilds.

Could you help any furhter?

-Katewadi
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 450 total points
ID: 37738988
my script works with the payload you posted in your question, I simply c&p this data into a file and processed it with that awk script
if this does not work with the data from your file, it's most likely formated in another way

see my script, it expects the data you want in the proper fields $1 .. $8 which must be separated by white spaces

when I see
   ontainer|:|
I assume that your data contains more fields, or are arranged in other sequence
0

Featured Post

U.S. Department of Agriculture and Acronis Access

With the new era of mobile computing, smartphones and tablets, wireless communications and cloud services, the USDA sought to take advantage of a mobilized workforce and the blurring lines between personal and corporate computing resources.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…

947 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now