Solved

Need a shell script that does formatting as well

Posted on 2013-12-03
17
366 Views
Last Modified: 2013-12-10
Hi,

I am running the ifconfig command and it gives output of lot of things like following but I just want this data to be formatted by doing one command line script get the output as following

so can someone share a command line script with me for doing this  


interface,packet_Type,packet_received,packet_errors,packet_dropped
bond1,rx,162,0,0
bond1,tx,38,0,0
eth0,rx,10086800,0,0
eth0,tx,2530268,0,0
eth1,rx,
eth1,tx

etc.......


bond1     Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          inet addr:10.115.206.101  Bcast:10.115.206.255  Mask:255.255.255.0
          inet6 addr: fe80::d6ae:52ff:fe8f:fb5c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)

eth0      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:10086800 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2530268 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15174241787 (14.1 GiB)  TX bytes:507714861 (484.1 MiB)
          Interrupt:36 Memory:f4000000-f4012800

eth1      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:7402 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:895359 (874.3 KiB)  TX bytes:0 (0.0 b)
          Interrupt:48 Memory:f6000000-f6012800

eth2      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6752 (6.5 KiB)  TX bytes:2660 (2.5 KiB)
          Interrupt:34 Memory:f8000000-f87fffff
0
Comment
Question by:anshuma
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 8
17 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39694353
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }'

Open in new window

0
 

Author Comment

by:anshuma
ID: 39697252
this script is awesome, Can you tell me if I put this in a shell script then how can I write the output of this into a CSV file in the following format

hostname_timestamp.csv

thanks
-anshu
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 39697771
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }' > $(hostname)_$(date "+%Y%m%d%H%M%S").csv

Open in new window

This will create a file "hostname_YYYYMMDDhhmmss.csv"
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 

Author Closing Comment

by:anshuma
ID: 39700028
thank you very much
0
 

Author Comment

by:anshuma
ID: 39700214
Two more modifications

add a time stamp column and a hostname column to the output data

 and output to a standard file

$(hostname)_network_ifconfigout.csv

so the output should look like

hostname, time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped
abc123,00:00:00,bond1,rx,162,0,0
abc123,00:00:00,bond1,tx,38,0,0
abc123,00:00:00,eth0,rx,10086800,0,0
abc123,00:00:00,eth0,tx,2530268,0,0
abc123,00:00:00,eth1,rx,
abc123,00:00:00,eth1,tx

and so on.

The end goal is to create a file that can be run by a cron job every 30 minutes or 1 hour and then be analysed in excel or any other visualization tool. And then do trending over time
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39700506
Here you go:
ifconfig | awk -v H=$(hostname) -v D=$(date "+%H:%M:%S") '
  BEGIN {print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $(hostname)_network_ifconfigout.csv

Open in new window

0
 

Author Comment

by:anshuma
ID: 39702576
this works great but there's a little problem, if this code is run several times then the header information is repeated. Can the logic be put that the header is only written when the script is run for the first time and then no more again
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39703016
We can test if the output file already exists and write the header only if it doesn't.
So here you go again:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

Please note that the hostname and the output file name are now in their own variables "$H" and "$OUT" (first and second line above) to avoid unneccessary repetitions of "$(hostname)" (a fork, after all) and to facilitate changing the output file name (and path), if required.
0
 

Author Comment

by:anshuma
ID: 39704430
This script runs forever and does not do anything may be I am doing something wrong here.
script.png
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704477
Sorry, I mistakenly posted my test script which is missing a small but important part ("ifconfig | " before awk). Here is the "production" version:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig | awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

0
 

Author Comment

by:anshuma
ID: 39704509
This works great. I have some more questions regarding this. Do you want me to open a new question so that I can award you further points ?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704516
If it's a new isssue or a major amendment please open a new Q, if it's an issue with the current script please continue here.

wmp
0
 

Author Comment

by:anshuma
ID: 39704708
For the following change let me know if you want me to open a new thread.

As you can see that my goal is run this script every hour and then visualize the data in Excel or Tableau. Hence the data is highly denormalized

so I want to add the date as well to the time stamp  mm-dd-yy hh:mm:ss

and then I also need to add  overruns, frame, carrier to each interface and finally the RX BYTES and TX BYTES (Just Byte Values only) I can then diviide them to get KB/GB/TB etc


RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)
 

so the output should now look like


time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes
mm-dd-yy hh:mm:ss,bond1,rx,162,0,0,0,0,0,10848
mm-dd-yy hh:mm:ss,bond1,tx,38,0,0,0,0,0,2660
mm-dd-yy hh:mm:ss,eth0,rx,10086800,0,0
mm-dd-yy hh:mm:ss,eth0,tx,2530268,0,0
mm-dd-yy hh:mm:ss,eth1,rx,
mm-dd-yy hh:mm:ss,eth1,tx
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704715
OK, that's not too hard. Please be patient for a minute or two. I'll be back here.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704733
Do you really want to omit the hostname?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704811
OK, this is without hostname:

H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

And this is with the hostname in the first column:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v H=$H -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = H "," D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = H "," D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

Please note that with RX there is no "carrier" value and that with TX there is no "frame" value, so I set those columns to "0" in their respective lines ("packets_carrier" is always "0" in RX lines and "packets_frame" is always "0" in TX lines).
0
 

Author Comment

by:anshuma
ID: 39710486
Thanks alot. You are very nice. Really appreciate your help
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
In a recent question (https://www.experts-exchange.com/questions/29004105/Run-AutoHotkey-script-directly-from-Notepad.html) here at Experts Exchange, a member asked how to run an AutoHotkey script (.AHK) directly from Notepad++ (aka NPP). This video…
Suggested Courses

615 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question