Solved

Need a shell script that does formatting as well

Posted on 2013-12-03
17
363 Views
Last Modified: 2013-12-10
Hi,

I am running the ifconfig command and it gives output of lot of things like following but I just want this data to be formatted by doing one command line script get the output as following

so can someone share a command line script with me for doing this  


interface,packet_Type,packet_received,packet_errors,packet_dropped
bond1,rx,162,0,0
bond1,tx,38,0,0
eth0,rx,10086800,0,0
eth0,tx,2530268,0,0
eth1,rx,
eth1,tx

etc.......


bond1     Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          inet addr:10.115.206.101  Bcast:10.115.206.255  Mask:255.255.255.0
          inet6 addr: fe80::d6ae:52ff:fe8f:fb5c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)

eth0      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:10086800 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2530268 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15174241787 (14.1 GiB)  TX bytes:507714861 (484.1 MiB)
          Interrupt:36 Memory:f4000000-f4012800

eth1      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:7402 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:895359 (874.3 KiB)  TX bytes:0 (0.0 b)
          Interrupt:48 Memory:f6000000-f6012800

eth2      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6752 (6.5 KiB)  TX bytes:2660 (2.5 KiB)
          Interrupt:34 Memory:f8000000-f87fffff
0
Comment
Question by:anshuma
  • 9
  • 8
17 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39694353
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }'

Open in new window

0
 

Author Comment

by:anshuma
ID: 39697252
this script is awesome, Can you tell me if I put this in a shell script then how can I write the output of this into a CSV file in the following format

hostname_timestamp.csv

thanks
-anshu
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 39697771
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }' > $(hostname)_$(date "+%Y%m%d%H%M%S").csv

Open in new window

This will create a file "hostname_YYYYMMDDhhmmss.csv"
0
Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 

Author Closing Comment

by:anshuma
ID: 39700028
thank you very much
0
 

Author Comment

by:anshuma
ID: 39700214
Two more modifications

add a time stamp column and a hostname column to the output data

 and output to a standard file

$(hostname)_network_ifconfigout.csv

so the output should look like

hostname, time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped
abc123,00:00:00,bond1,rx,162,0,0
abc123,00:00:00,bond1,tx,38,0,0
abc123,00:00:00,eth0,rx,10086800,0,0
abc123,00:00:00,eth0,tx,2530268,0,0
abc123,00:00:00,eth1,rx,
abc123,00:00:00,eth1,tx

and so on.

The end goal is to create a file that can be run by a cron job every 30 minutes or 1 hour and then be analysed in excel or any other visualization tool. And then do trending over time
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39700506
Here you go:
ifconfig | awk -v H=$(hostname) -v D=$(date "+%H:%M:%S") '
  BEGIN {print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $(hostname)_network_ifconfigout.csv

Open in new window

0
 

Author Comment

by:anshuma
ID: 39702576
this works great but there's a little problem, if this code is run several times then the header information is repeated. Can the logic be put that the header is only written when the script is run for the first time and then no more again
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39703016
We can test if the output file already exists and write the header only if it doesn't.
So here you go again:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

Please note that the hostname and the output file name are now in their own variables "$H" and "$OUT" (first and second line above) to avoid unneccessary repetitions of "$(hostname)" (a fork, after all) and to facilitate changing the output file name (and path), if required.
0
 

Author Comment

by:anshuma
ID: 39704430
This script runs forever and does not do anything may be I am doing something wrong here.
script.png
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704477
Sorry, I mistakenly posted my test script which is missing a small but important part ("ifconfig | " before awk). Here is the "production" version:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig | awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

0
 

Author Comment

by:anshuma
ID: 39704509
This works great. I have some more questions regarding this. Do you want me to open a new question so that I can award you further points ?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704516
If it's a new isssue or a major amendment please open a new Q, if it's an issue with the current script please continue here.

wmp
0
 

Author Comment

by:anshuma
ID: 39704708
For the following change let me know if you want me to open a new thread.

As you can see that my goal is run this script every hour and then visualize the data in Excel or Tableau. Hence the data is highly denormalized

so I want to add the date as well to the time stamp  mm-dd-yy hh:mm:ss

and then I also need to add  overruns, frame, carrier to each interface and finally the RX BYTES and TX BYTES (Just Byte Values only) I can then diviide them to get KB/GB/TB etc


RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)
 

so the output should now look like


time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes
mm-dd-yy hh:mm:ss,bond1,rx,162,0,0,0,0,0,10848
mm-dd-yy hh:mm:ss,bond1,tx,38,0,0,0,0,0,2660
mm-dd-yy hh:mm:ss,eth0,rx,10086800,0,0
mm-dd-yy hh:mm:ss,eth0,tx,2530268,0,0
mm-dd-yy hh:mm:ss,eth1,rx,
mm-dd-yy hh:mm:ss,eth1,tx
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704715
OK, that's not too hard. Please be patient for a minute or two. I'll be back here.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704733
Do you really want to omit the hostname?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704811
OK, this is without hostname:

H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

And this is with the hostname in the first column:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v H=$H -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = H "," D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = H "," D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

Please note that with RX there is no "carrier" value and that with TX there is no "frame" value, so I set those columns to "0" in their respective lines ("packets_carrier" is always "0" in RX lines and "packets_frame" is always "0" in TX lines).
0
 

Author Comment

by:anshuma
ID: 39710486
Thanks alot. You are very nice. Really appreciate your help
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space. w60 means Windows Vista or Windows Server 2008. w61 means Windows 7 or Windows Server 2008 R2. There are various …
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

808 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question