• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 593
  • Last Modified:

Need a bash script to extract 4th argument (syslog host IP address) in a CSV file

Hi team,

I have a dissected syslog file in CSV format which contains the following fields:

Month, date, time, IP address, Syslog message.  The csv file is tens of thousands of lines long and I just have a requirement to extract the unique IPs in the entire csv file and save them in a separate text file.

A snippet of the syslog.csv file is here:

"Apr","17","06:51:01","","syslog T /emupdate/subscription?uid=3 HTTP/1.1' 200 492 "
"Apr","17","06:51:01","","local/testmachine info logger: [ssl_req][17/Apr/2011:06:51:01 +1000] TLSv1 DHE-RSA-AES256-SHA 'POST /emupdate/subscription?uid=3 HTTP/1.1' 492 "
"Apr","17","06:51:02","","test info logger: [ssl_acc] - - [17/Apr/2011:06:51:02 +1000] 'POST /emupdate/subscription?uid=3 HTTP/1.1' 200 492 "
"Apr","17","06:51:02","","test info logger: [ssl_req][17/Apr/2011:06:51:02 +1000] TLSv1 DHE-RSA-AES256-SHA 'POST /emupdate/subscription?uid=3 HTTP/1.1' 492 "

Open in new window

May i just request a simple bash script that can do the above?

Finally, does someone have a readymade "diff" script which can quickly compare two text files and extract a listing of lines (host IP addresses in this case) which are present in a text file (let's call it master) but which are not present in the extracted file above (let's call it extract).

Thanks for any help
  • 2
3 Solutions

awk -F',|"' '{print $11}' csvfile | sort -nu > textfile


comm -23 master extract


cat syslog | cut -f4 -d',' | tr -d '"' | sort -u > new_file
ad 2)

if "master is not already sorted, use this:

sort -nu master > master.sorted; comm -23 master.sorted extract; rm master.sorted
rleyba828Author Commented:
Hi Team....

sincere apologies for the late reply.  For some reason, the first reply from woolmilkporc  (the one using awk) did not print out the full list but the one from point_pleasant (using the cut script) seems to print out everything.   Not sure how/why these two different approaches would yield different results.   Anyway,   I have awarded the points.  Thanks to the contributors for the big help.

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now