?
Solved

Need a shell script (ksh) that uses an Awk array to process an input file

Posted on 2012-03-15
4
Medium Priority
?
518 Views
Last Modified: 2016-02-11
Hello Experts,

I am looking for a shell script (preferabally KSH) which processes a large input file (a log file) and provides an output file in a certain format. I want to use an Awk array to process each line of the input file and print the required output feilds to a file.

I need an array, because there are various WebContainer Types and they have different Broker transactions within them.

Thanks,

-Katewadi

####################################
Sample Input file data:

2012-03-15 05:50:09,916 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform starting command <AddUpdateProgramGroup> -------------------
 05:50:10,043 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,215 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 165 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,219 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,390 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 168 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,392 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform command <AddUpdateProgramGroup> done in 476 milliseconds -------------------

##################################
Format of the Output File:

Date|Start-Time|WebContainer-Number|Type|End-Time|Broker-Time|Total-Transaction-Time
2012-03-15|05:50:09,916|2|AddUpdateProgramGroup|05:50:10,392|333|476
0
Comment
Question by:Katewadi
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 62

Assisted Solution

by:gheist
gheist earned 150 total points
ID: 37728601
GNU awk would allow to customize row and field separators so that you can have multiline records parsed. Default BWK awk in FreeBSD does not do what you like, you need to install GNU AWK via # pkg_add -r gawk
https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html
0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 1350 total points
ID: 37731323
# traditional awk:
awk '/basicPerform starting command/{d=$1;t=$2;w=substr($5,1,1);y=substr($8,2,length($8)-2);b=0;next}/Broker Transaction completed in/{b+=$(NF-3);next}/basicPerform command.* done in/{e=$2;m=$(NF-2);printf "%s|%s|%s|%s|%s|%s|%s\n",d,t,w,y,e,b,m;next}'  your-file
0
 

Author Comment

by:Katewadi
ID: 37738642
Ahoffmann,

Thanks for your input...

BTW, I tried the awk command (as-is), and apparently it did not work for me.

It gave me an output like this:

ontainer|:|C|-----------------|06:15:13,220|622|in

Am I doing anything wrong? Most of the output feilds do not look as expected -- including the first and last feilds.

Could you help any furhter?

-Katewadi
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 1350 total points
ID: 37738988
my script works with the payload you posted in your question, I simply c&p this data into a file and processed it with that awk script
if this does not work with the data from your file, it's most likely formated in another way

see my script, it expects the data you want in the proper fields $1 .. $8 which must be separated by white spaces

when I see
   ontainer|:|
I assume that your data contains more fields, or are arranged in other sequence
0

Featured Post

7 Extremely Useful Linux Commands for Beginners

Just getting started with Linux? Here's a quick start guide that has 7 commands that we believe will come in handy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Setting up Secure Ubuntu server on VMware 1.      Insert the Ubuntu Server distribution CD or attach the ISO of the CD which is in the “Datastore”. Note that it is important to install the x64 edition on servers, not the X86 editions. 2.      Power on th…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses
Course of the Month8 days, 4 hours left to enroll

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question