Improve company productivity with a Business Account.Sign Up

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 373
  • Last Modified:

Need a shell script that does formatting as well

Hi,

I am running the ifconfig command and it gives output of lot of things like following but I just want this data to be formatted by doing one command line script get the output as following

so can someone share a command line script with me for doing this  


interface,packet_Type,packet_received,packet_errors,packet_dropped
bond1,rx,162,0,0
bond1,tx,38,0,0
eth0,rx,10086800,0,0
eth0,tx,2530268,0,0
eth1,rx,
eth1,tx

etc.......


bond1     Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          inet addr:10.115.206.101  Bcast:10.115.206.255  Mask:255.255.255.0
          inet6 addr: fe80::d6ae:52ff:fe8f:fb5c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)

eth0      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:10086800 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2530268 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15174241787 (14.1 GiB)  TX bytes:507714861 (484.1 MiB)
          Interrupt:36 Memory:f4000000-f4012800

eth1      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:7402 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:895359 (874.3 KiB)  TX bytes:0 (0.0 b)
          Interrupt:48 Memory:f6000000-f6012800

eth2      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6752 (6.5 KiB)  TX bytes:2660 (2.5 KiB)
          Interrupt:34 Memory:f8000000-f87fffff
0
anshuma
Asked:
anshuma
  • 9
  • 8
1 Solution
 
woolmilkporcCommented:
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }'

Open in new window

0
 
anshumaEngineeringAuthor Commented:
this script is awesome, Can you tell me if I put this in a shell script then how can I write the output of this into a CSV file in the following format

hostname_timestamp.csv

thanks
-anshu
0
 
woolmilkporcCommented:
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }' > $(hostname)_$(date "+%Y%m%d%H%M%S").csv

Open in new window

This will create a file "hostname_YYYYMMDDhhmmss.csv"
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
anshumaEngineeringAuthor Commented:
thank you very much
0
 
anshumaEngineeringAuthor Commented:
Two more modifications

add a time stamp column and a hostname column to the output data

 and output to a standard file

$(hostname)_network_ifconfigout.csv

so the output should look like

hostname, time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped
abc123,00:00:00,bond1,rx,162,0,0
abc123,00:00:00,bond1,tx,38,0,0
abc123,00:00:00,eth0,rx,10086800,0,0
abc123,00:00:00,eth0,tx,2530268,0,0
abc123,00:00:00,eth1,rx,
abc123,00:00:00,eth1,tx

and so on.

The end goal is to create a file that can be run by a cron job every 30 minutes or 1 hour and then be analysed in excel or any other visualization tool. And then do trending over time
0
 
woolmilkporcCommented:
Here you go:
ifconfig | awk -v H=$(hostname) -v D=$(date "+%H:%M:%S") '
  BEGIN {print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $(hostname)_network_ifconfigout.csv

Open in new window

0
 
anshumaEngineeringAuthor Commented:
this works great but there's a little problem, if this code is run several times then the header information is repeated. Can the logic be put that the header is only written when the script is run for the first time and then no more again
0
 
woolmilkporcCommented:
We can test if the output file already exists and write the header only if it doesn't.
So here you go again:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

Please note that the hostname and the output file name are now in their own variables "$H" and "$OUT" (first and second line above) to avoid unneccessary repetitions of "$(hostname)" (a fork, after all) and to facilitate changing the output file name (and path), if required.
0
 
anshumaEngineeringAuthor Commented:
This script runs forever and does not do anything may be I am doing something wrong here.
script.png
0
 
woolmilkporcCommented:
Sorry, I mistakenly posted my test script which is missing a small but important part ("ifconfig | " before awk). Here is the "production" version:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig | awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

0
 
anshumaEngineeringAuthor Commented:
This works great. I have some more questions regarding this. Do you want me to open a new question so that I can award you further points ?
0
 
woolmilkporcCommented:
If it's a new isssue or a major amendment please open a new Q, if it's an issue with the current script please continue here.

wmp
0
 
anshumaEngineeringAuthor Commented:
For the following change let me know if you want me to open a new thread.

As you can see that my goal is run this script every hour and then visualize the data in Excel or Tableau. Hence the data is highly denormalized

so I want to add the date as well to the time stamp  mm-dd-yy hh:mm:ss

and then I also need to add  overruns, frame, carrier to each interface and finally the RX BYTES and TX BYTES (Just Byte Values only) I can then diviide them to get KB/GB/TB etc


RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)
 

so the output should now look like


time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes
mm-dd-yy hh:mm:ss,bond1,rx,162,0,0,0,0,0,10848
mm-dd-yy hh:mm:ss,bond1,tx,38,0,0,0,0,0,2660
mm-dd-yy hh:mm:ss,eth0,rx,10086800,0,0
mm-dd-yy hh:mm:ss,eth0,tx,2530268,0,0
mm-dd-yy hh:mm:ss,eth1,rx,
mm-dd-yy hh:mm:ss,eth1,tx
0
 
woolmilkporcCommented:
OK, that's not too hard. Please be patient for a minute or two. I'll be back here.
0
 
woolmilkporcCommented:
Do you really want to omit the hostname?
0
 
woolmilkporcCommented:
OK, this is without hostname:

H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

And this is with the hostname in the first column:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v H=$H -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = H "," D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = H "," D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

Please note that with RX there is no "carrier" value and that with TX there is no "frame" value, so I set those columns to "0" in their respective lines ("packets_carrier" is always "0" in RX lines and "packets_frame" is always "0" in TX lines).
0
 
anshumaEngineeringAuthor Commented:
Thanks alot. You are very nice. Really appreciate your help
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 9
  • 8
Tackle projects and never again get stuck behind a technical roadblock.
Join Now