Solved

Need a shell script that does formatting as well

Posted on 2013-12-03
17
362 Views
Last Modified: 2013-12-10
Hi,

I am running the ifconfig command and it gives output of lot of things like following but I just want this data to be formatted by doing one command line script get the output as following

so can someone share a command line script with me for doing this  


interface,packet_Type,packet_received,packet_errors,packet_dropped
bond1,rx,162,0,0
bond1,tx,38,0,0
eth0,rx,10086800,0,0
eth0,tx,2530268,0,0
eth1,rx,
eth1,tx

etc.......


bond1     Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          inet addr:10.115.206.101  Bcast:10.115.206.255  Mask:255.255.255.0
          inet6 addr: fe80::d6ae:52ff:fe8f:fb5c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)

eth0      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:10086800 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2530268 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15174241787 (14.1 GiB)  TX bytes:507714861 (484.1 MiB)
          Interrupt:36 Memory:f4000000-f4012800

eth1      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:7402 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:895359 (874.3 KiB)  TX bytes:0 (0.0 b)
          Interrupt:48 Memory:f6000000-f6012800

eth2      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6752 (6.5 KiB)  TX bytes:2660 (2.5 KiB)
          Interrupt:34 Memory:f8000000-f87fffff
0
Comment
Question by:anshuma
  • 9
  • 8
17 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39694353
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }'

Open in new window

0
 

Author Comment

by:anshuma
ID: 39697252
this script is awesome, Can you tell me if I put this in a shell script then how can I write the output of this into a CSV file in the following format

hostname_timestamp.csv

thanks
-anshu
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 39697771
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }' > $(hostname)_$(date "+%Y%m%d%H%M%S").csv

Open in new window

This will create a file "hostname_YYYYMMDDhhmmss.csv"
0
Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

 

Author Closing Comment

by:anshuma
ID: 39700028
thank you very much
0
 

Author Comment

by:anshuma
ID: 39700214
Two more modifications

add a time stamp column and a hostname column to the output data

 and output to a standard file

$(hostname)_network_ifconfigout.csv

so the output should look like

hostname, time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped
abc123,00:00:00,bond1,rx,162,0,0
abc123,00:00:00,bond1,tx,38,0,0
abc123,00:00:00,eth0,rx,10086800,0,0
abc123,00:00:00,eth0,tx,2530268,0,0
abc123,00:00:00,eth1,rx,
abc123,00:00:00,eth1,tx

and so on.

The end goal is to create a file that can be run by a cron job every 30 minutes or 1 hour and then be analysed in excel or any other visualization tool. And then do trending over time
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39700506
Here you go:
ifconfig | awk -v H=$(hostname) -v D=$(date "+%H:%M:%S") '
  BEGIN {print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $(hostname)_network_ifconfigout.csv

Open in new window

0
 

Author Comment

by:anshuma
ID: 39702576
this works great but there's a little problem, if this code is run several times then the header information is repeated. Can the logic be put that the header is only written when the script is run for the first time and then no more again
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39703016
We can test if the output file already exists and write the header only if it doesn't.
So here you go again:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

Please note that the hostname and the output file name are now in their own variables "$H" and "$OUT" (first and second line above) to avoid unneccessary repetitions of "$(hostname)" (a fork, after all) and to facilitate changing the output file name (and path), if required.
0
 

Author Comment

by:anshuma
ID: 39704430
This script runs forever and does not do anything may be I am doing something wrong here.
script.png
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704477
Sorry, I mistakenly posted my test script which is missing a small but important part ("ifconfig | " before awk). Here is the "production" version:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig | awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

0
 

Author Comment

by:anshuma
ID: 39704509
This works great. I have some more questions regarding this. Do you want me to open a new question so that I can award you further points ?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704516
If it's a new isssue or a major amendment please open a new Q, if it's an issue with the current script please continue here.

wmp
0
 

Author Comment

by:anshuma
ID: 39704708
For the following change let me know if you want me to open a new thread.

As you can see that my goal is run this script every hour and then visualize the data in Excel or Tableau. Hence the data is highly denormalized

so I want to add the date as well to the time stamp  mm-dd-yy hh:mm:ss

and then I also need to add  overruns, frame, carrier to each interface and finally the RX BYTES and TX BYTES (Just Byte Values only) I can then diviide them to get KB/GB/TB etc


RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)
 

so the output should now look like


time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes
mm-dd-yy hh:mm:ss,bond1,rx,162,0,0,0,0,0,10848
mm-dd-yy hh:mm:ss,bond1,tx,38,0,0,0,0,0,2660
mm-dd-yy hh:mm:ss,eth0,rx,10086800,0,0
mm-dd-yy hh:mm:ss,eth0,tx,2530268,0,0
mm-dd-yy hh:mm:ss,eth1,rx,
mm-dd-yy hh:mm:ss,eth1,tx
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704715
OK, that's not too hard. Please be patient for a minute or two. I'll be back here.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704733
Do you really want to omit the hostname?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704811
OK, this is without hostname:

H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

And this is with the hostname in the first column:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v H=$H -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = H "," D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = H "," D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

Please note that with RX there is no "carrier" value and that with TX there is no "frame" value, so I set those columns to "0" in their respective lines ("packets_carrier" is always "0" in RX lines and "packets_frame" is always "0" in TX lines).
0
 

Author Comment

by:anshuma
ID: 39710486
Thanks alot. You are very nice. Really appreciate your help
0

Featured Post

Live: Real-Time Solutions, Start Here

Receive instant 1:1 support from technology experts, using our real-time conversation and whiteboard interface. Your first 5 minutes are always free.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now