Solved

Need a shell script (ksh) that uses an Awk array to process an input file

Posted on 2012-03-15
4
513 Views
Last Modified: 2016-02-11
Hello Experts,

I am looking for a shell script (preferabally KSH) which processes a large input file (a log file) and provides an output file in a certain format. I want to use an Awk array to process each line of the input file and print the required output feilds to a file.

I need an array, because there are various WebContainer Types and they have different Broker transactions within them.

Thanks,

-Katewadi

####################################
Sample Input file data:

2012-03-15 05:50:09,916 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform starting command <AddUpdateProgramGroup> -------------------
 05:50:10,043 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,215 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 165 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,219 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,390 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 168 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,392 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform command <AddUpdateProgramGroup> done in 476 milliseconds -------------------

##################################
Format of the Output File:

Date|Start-Time|WebContainer-Number|Type|End-Time|Broker-Time|Total-Transaction-Time
2012-03-15|05:50:09,916|2|AddUpdateProgramGroup|05:50:10,392|333|476
0
Comment
Question by:Katewadi
  • 2
4 Comments
 
LVL 62

Assisted Solution

by:gheist
gheist earned 50 total points
ID: 37728601
GNU awk would allow to customize row and field separators so that you can have multiline records parsed. Default BWK awk in FreeBSD does not do what you like, you need to install GNU AWK via # pkg_add -r gawk
https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html
0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 450 total points
ID: 37731323
# traditional awk:
awk '/basicPerform starting command/{d=$1;t=$2;w=substr($5,1,1);y=substr($8,2,length($8)-2);b=0;next}/Broker Transaction completed in/{b+=$(NF-3);next}/basicPerform command.* done in/{e=$2;m=$(NF-2);printf "%s|%s|%s|%s|%s|%s|%s\n",d,t,w,y,e,b,m;next}'  your-file
0
 

Author Comment

by:Katewadi
ID: 37738642
Ahoffmann,

Thanks for your input...

BTW, I tried the awk command (as-is), and apparently it did not work for me.

It gave me an output like this:

ontainer|:|C|-----------------|06:15:13,220|622|in

Am I doing anything wrong? Most of the output feilds do not look as expected -- including the first and last feilds.

Could you help any furhter?

-Katewadi
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 450 total points
ID: 37738988
my script works with the payload you posted in your question, I simply c&p this data into a file and processed it with that awk script
if this does not work with the data from your file, it's most likely formated in another way

see my script, it expects the data you want in the proper fields $1 .. $8 which must be separated by white spaces

when I see
   ontainer|:|
I assume that your data contains more fields, or are arranged in other sequence
0

Featured Post

U.S. Department of Agriculture and Acronis Access

With the new era of mobile computing, smartphones and tablets, wireless communications and cloud services, the USDA sought to take advantage of a mobilized workforce and the blurring lines between personal and corporate computing resources.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Bash script - Exit out of choice loop 2 55
EXCHANGE. LINUX, SYMANTEC MESSAGING GATEWAY 2 51
Syslog-ng works. Now what? How to filter and manage? 8 93
SSH in linux 9 69
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Introduction We as admins face situation where we need to redirect websites to another. This may be required as a part of an upgrade keeping the old URL but website should be served from new URL. This document would brief you on different ways ca…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question