Solved

Need a shell script (ksh) that uses an Awk array to process an input file

Posted on 2012-03-15
4
512 Views
Last Modified: 2016-02-11
Hello Experts,

I am looking for a shell script (preferabally KSH) which processes a large input file (a log file) and provides an output file in a certain format. I want to use an Awk array to process each line of the input file and print the required output feilds to a file.

I need an array, because there are various WebContainer Types and they have different Broker transactions within them.

Thanks,

-Katewadi

####################################
Sample Input file data:

2012-03-15 05:50:09,916 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform starting command <AddUpdateProgramGroup> -------------------
 05:50:10,043 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,215 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 165 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,219 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,390 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 168 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,392 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform command <AddUpdateProgramGroup> done in 476 milliseconds -------------------

##################################
Format of the Output File:

Date|Start-Time|WebContainer-Number|Type|End-Time|Broker-Time|Total-Transaction-Time
2012-03-15|05:50:09,916|2|AddUpdateProgramGroup|05:50:10,392|333|476
0
Comment
Question by:Katewadi
  • 2
4 Comments
 
LVL 62

Assisted Solution

by:gheist
gheist earned 50 total points
ID: 37728601
GNU awk would allow to customize row and field separators so that you can have multiline records parsed. Default BWK awk in FreeBSD does not do what you like, you need to install GNU AWK via # pkg_add -r gawk
https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html
0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 450 total points
ID: 37731323
# traditional awk:
awk '/basicPerform starting command/{d=$1;t=$2;w=substr($5,1,1);y=substr($8,2,length($8)-2);b=0;next}/Broker Transaction completed in/{b+=$(NF-3);next}/basicPerform command.* done in/{e=$2;m=$(NF-2);printf "%s|%s|%s|%s|%s|%s|%s\n",d,t,w,y,e,b,m;next}'  your-file
0
 

Author Comment

by:Katewadi
ID: 37738642
Ahoffmann,

Thanks for your input...

BTW, I tried the awk command (as-is), and apparently it did not work for me.

It gave me an output like this:

ontainer|:|C|-----------------|06:15:13,220|622|in

Am I doing anything wrong? Most of the output feilds do not look as expected -- including the first and last feilds.

Could you help any furhter?

-Katewadi
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 450 total points
ID: 37738988
my script works with the payload you posted in your question, I simply c&p this data into a file and processed it with that awk script
if this does not work with the data from your file, it's most likely formated in another way

see my script, it expects the data you want in the proper fields $1 .. $8 which must be separated by white spaces

when I see
   ontainer|:|
I assume that your data contains more fields, or are arranged in other sequence
0

Featured Post

The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
looking for a CENTOS ISO to download with x window installed 2 49
android secure ftp 3 50
bash file 10 49
RHEL 6.7 Gnome Desktop on VMware 6 VM 9 66
Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question