Solved

Need a shell script that does formatting as well

Posted on 2013-12-03
17
360 Views
Last Modified: 2013-12-10
Hi,

I am running the ifconfig command and it gives output of lot of things like following but I just want this data to be formatted by doing one command line script get the output as following

so can someone share a command line script with me for doing this  


interface,packet_Type,packet_received,packet_errors,packet_dropped
bond1,rx,162,0,0
bond1,tx,38,0,0
eth0,rx,10086800,0,0
eth0,tx,2530268,0,0
eth1,rx,
eth1,tx

etc.......


bond1     Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          inet addr:10.115.206.101  Bcast:10.115.206.255  Mask:255.255.255.0
          inet6 addr: fe80::d6ae:52ff:fe8f:fb5c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)

eth0      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:10086800 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2530268 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15174241787 (14.1 GiB)  TX bytes:507714861 (484.1 MiB)
          Interrupt:36 Memory:f4000000-f4012800

eth1      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:58
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:7402 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:895359 (874.3 KiB)  TX bytes:0 (0.0 b)
          Interrupt:48 Memory:f6000000-f6012800

eth2      Link encap:Ethernet  HWaddr D4:AE:52:8F:FB:5C
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:98 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6752 (6.5 KiB)  TX bytes:2660 (2.5 KiB)
          Interrupt:34 Memory:f8000000-f87fffff
0
Comment
Question by:anshuma
  • 9
  • 8
17 Comments
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39694353
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }'

Open in new window

0
 

Author Comment

by:anshuma
ID: 39697252
this script is awesome, Can you tell me if I put this in a shell script then how can I write the output of this into a CSV file in the following format

hostname_timestamp.csv

thanks
-anshu
0
 
LVL 68

Accepted Solution

by:
woolmilkporc earned 500 total points
ID: 39697771
ifconfig | awk '
  BEGIN {print "interface,packet_Type,packet_received,packet_errors,packet_dropped"} 
{  
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print intf ",tx," $3 "," $5 "," $7} 
 }' > $(hostname)_$(date "+%Y%m%d%H%M%S").csv

Open in new window

This will create a file "hostname_YYYYMMDDhhmmss.csv"
0
 

Author Closing Comment

by:anshuma
ID: 39700028
thank you very much
0
 

Author Comment

by:anshuma
ID: 39700214
Two more modifications

add a time stamp column and a hostname column to the output data

 and output to a standard file

$(hostname)_network_ifconfigout.csv

so the output should look like

hostname, time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped
abc123,00:00:00,bond1,rx,162,0,0
abc123,00:00:00,bond1,tx,38,0,0
abc123,00:00:00,eth0,rx,10086800,0,0
abc123,00:00:00,eth0,tx,2530268,0,0
abc123,00:00:00,eth1,rx,
abc123,00:00:00,eth1,tx

and so on.

The end goal is to create a file that can be run by a cron job every 30 minutes or 1 hour and then be analysed in excel or any other visualization tool. And then do trending over time
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39700506
Here you go:
ifconfig | awk -v H=$(hostname) -v D=$(date "+%H:%M:%S") '
  BEGIN {print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $(hostname)_network_ifconfigout.csv

Open in new window

0
 

Author Comment

by:anshuma
ID: 39702576
this works great but there's a little problem, if this code is run several times then the header information is repeated. Can the logic be put that the header is only written when the script is run for the first time and then no more again
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39703016
We can test if the output file already exists and write the header only if it doesn't.
So here you go again:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

Please note that the hostname and the output file name are now in their own variables "$H" and "$OUT" (first and second line above) to avoid unneccessary repetitions of "$(hostname)" (a fork, after all) and to facilitate changing the output file name (and path), if required.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:anshuma
ID: 39704430
This script runs forever and does not do anything may be I am doing something wrong here.
script.png
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704477
Sorry, I mistakenly posted my test script which is missing a small but important part ("ifconfig | " before awk). Here is the "production" version:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig | awk -v H=$H -v D=$(date "+%H:%M:%S") -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); print H "," D "," intf ",rx," $3 "," $5 "," $7}
  if($0~/TX packets/) {gsub(":"," ",$0); print H "," D "," intf ",tx," $3 "," $5 "," $7}
 }' >> $OUT

Open in new window

0
 

Author Comment

by:anshuma
ID: 39704509
This works great. I have some more questions regarding this. Do you want me to open a new question so that I can award you further points ?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704516
If it's a new isssue or a major amendment please open a new Q, if it's an issue with the current script please continue here.

wmp
0
 

Author Comment

by:anshuma
ID: 39704708
For the following change let me know if you want me to open a new thread.

As you can see that my goal is run this script every hour and then visualize the data in Excel or Tableau. Hence the data is highly denormalized

so I want to add the date as well to the time stamp  mm-dd-yy hh:mm:ss

and then I also need to add  overruns, frame, carrier to each interface and finally the RX BYTES and TX BYTES (Just Byte Values only) I can then diviide them to get KB/GB/TB etc


RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10848 (10.5 KiB)  TX bytes:2660 (2.5 KiB)
 

so the output should now look like


time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes
mm-dd-yy hh:mm:ss,bond1,rx,162,0,0,0,0,0,10848
mm-dd-yy hh:mm:ss,bond1,tx,38,0,0,0,0,0,2660
mm-dd-yy hh:mm:ss,eth0,rx,10086800,0,0
mm-dd-yy hh:mm:ss,eth0,tx,2530268,0,0
mm-dd-yy hh:mm:ss,eth1,rx,
mm-dd-yy hh:mm:ss,eth1,tx
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704715
OK, that's not too hard. Please be patient for a minute or two. I'll be back here.
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704733
Do you really want to omit the hostname?
0
 
LVL 68

Expert Comment

by:woolmilkporc
ID: 39704811
OK, this is without hostname:

H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

And this is with the hostname in the first column:
H=$(hostname)
OUT="${H}_network_ifconfigout.csv"
unset I; [[ ! -e $OUT ]] && I="x"
ifconfig |awk -v H=$H -v D="$(date "+%m-%d-%y %H:%M:%S")" -v I="$I"  '
  BEGIN {if(I!="") print "hostname,time_stamp,interface,packet_Type,packet_received,packet_errors,packet_dropped,packets_overrun,packets_frame,packets_carrier,bytes"}
{
  if($0~/Link encap/) {intf=$1};
  if($0~/RX packets/) {gsub(":"," ",$0); rxline = H "," D "," intf ",rx," $3 "," $5 "," $7 "," $9 "," $11 ",0" }
  if($0~/TX packets/) {gsub(":"," ",$0); txline = H "," D "," intf ",tx," $3 "," $5 "," $7 "," $9 ",0," $11}
  if($0~/RX bytes/) {gsub(":"," ",$0); rxline = rxline "," $3; txline = txline "," $8; print rxline; print txline}
 }'  >> $OUT

Open in new window

Please note that with RX there is no "carrier" value and that with TX there is no "frame" value, so I set those columns to "0" in their respective lines ("packets_carrier" is always "0" in RX lines and "packets_frame" is always "0" in TX lines).
0
 

Author Comment

by:anshuma
ID: 39710486
Thanks alot. You are very nice. Really appreciate your help
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space. w60 means Windows Vista or Windows Server 2008. w61 means Windows 7 or Windows Server 2008 R2. There are various …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
In a previous video, we went over how to export a DynamoDB table into Amazon S3.  In this video, we show how to load the export from S3 into a DynamoDB table.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now