Solved

Need a shell script (ksh) that uses an Awk array to process an input file

Posted on 2012-03-15
4
515 Views
Last Modified: 2016-02-11
Hello Experts,

I am looking for a shell script (preferabally KSH) which processes a large input file (a log file) and provides an output file in a certain format. I want to use an Awk array to process each line of the input file and print the required output feilds to a file.

I need an array, because there are various WebContainer Types and they have different Broker transactions within them.

Thanks,

-Katewadi

####################################
Sample Input file data:

2012-03-15 05:50:09,916 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform starting command <AddUpdateProgramGroup> -------------------
 05:50:10,043 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,215 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 165 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,219 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction started
2012-03-15 05:50:10,390 [WebContainer : 2] INFO  IDMODTransaction (AddUpdateProgramGroup) - ####### Broker Transaction completed in 168 milliseconds : EXSIDMOD/MFPR/NOTIFY
2012-03-15 05:50:10,392 [WebContainer : 2] INFO  BPServlet (AddUpdateProgramGroup) - ------------------- basicPerform command <AddUpdateProgramGroup> done in 476 milliseconds -------------------

##################################
Format of the Output File:

Date|Start-Time|WebContainer-Number|Type|End-Time|Broker-Time|Total-Transaction-Time
2012-03-15|05:50:09,916|2|AddUpdateProgramGroup|05:50:10,392|333|476
0
Comment
Question by:Katewadi
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 62

Assisted Solution

by:gheist
gheist earned 50 total points
ID: 37728601
GNU awk would allow to customize row and field separators so that you can have multiline records parsed. Default BWK awk in FreeBSD does not do what you like, you need to install GNU AWK via # pkg_add -r gawk
https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html
0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 450 total points
ID: 37731323
# traditional awk:
awk '/basicPerform starting command/{d=$1;t=$2;w=substr($5,1,1);y=substr($8,2,length($8)-2);b=0;next}/Broker Transaction completed in/{b+=$(NF-3);next}/basicPerform command.* done in/{e=$2;m=$(NF-2);printf "%s|%s|%s|%s|%s|%s|%s\n",d,t,w,y,e,b,m;next}'  your-file
0
 

Author Comment

by:Katewadi
ID: 37738642
Ahoffmann,

Thanks for your input...

BTW, I tried the awk command (as-is), and apparently it did not work for me.

It gave me an output like this:

ontainer|:|C|-----------------|06:15:13,220|622|in

Am I doing anything wrong? Most of the output feilds do not look as expected -- including the first and last feilds.

Could you help any furhter?

-Katewadi
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 450 total points
ID: 37738988
my script works with the payload you posted in your question, I simply c&p this data into a file and processed it with that awk script
if this does not work with the data from your file, it's most likely formated in another way

see my script, it expects the data you want in the proper fields $1 .. $8 which must be separated by white spaces

when I see
   ontainer|:|
I assume that your data contains more fields, or are arranged in other sequence
0

Featured Post

The Eight Noble Truths of Backup and Recovery

How can IT departments tackle the challenges of a Big Data world? This white paper provides a roadmap to success and helps companies ensure that all their data is safe and secure, no matter if it resides on-premise with physical or virtual machines or in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
In the first part of this tutorial we will cover the prerequisites for installing SQL Server vNext on Linux.
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

688 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question