Need a bash script to extract 4th argument (syslog host IP address) in a CSV file

Posted on 2011-04-27
Last Modified: 2012-05-11
Hi team,

I have a dissected syslog file in CSV format which contains the following fields:

Month, date, time, IP address, Syslog message.  The csv file is tens of thousands of lines long and I just have a requirement to extract the unique IPs in the entire csv file and save them in a separate text file.

A snippet of the syslog.csv file is here:

"Apr","17","06:51:01","","syslog T /emupdate/subscription?uid=3 HTTP/1.1' 200 492 "
"Apr","17","06:51:01","","local/testmachine info logger: [ssl_req][17/Apr/2011:06:51:01 +1000] TLSv1 DHE-RSA-AES256-SHA 'POST /emupdate/subscription?uid=3 HTTP/1.1' 492 "
"Apr","17","06:51:02","","test info logger: [ssl_acc] - - [17/Apr/2011:06:51:02 +1000] 'POST /emupdate/subscription?uid=3 HTTP/1.1' 200 492 "
"Apr","17","06:51:02","","test info logger: [ssl_req][17/Apr/2011:06:51:02 +1000] TLSv1 DHE-RSA-AES256-SHA 'POST /emupdate/subscription?uid=3 HTTP/1.1' 492 "

Open in new window

May i just request a simple bash script that can do the above?

Finally, does someone have a readymade "diff" script which can quickly compare two text files and extract a listing of lines (host IP addresses in this case) which are present in a text file (let's call it master) but which are not present in the extracted file above (let's call it extract).

Thanks for any help
Question by:rleyba828
    LVL 68

    Assisted Solution


    awk -F',|"' '{print $11}' csvfile | sort -nu > textfile


    comm -23 master extract


    LVL 8

    Accepted Solution

    cat syslog | cut -f4 -d',' | tr -d '"' | sort -u > new_file
    LVL 68

    Assisted Solution

    ad 2)

    if "master is not already sorted, use this:

    sort -nu master > master.sorted; comm -23 master.sorted extract; rm master.sorted

    Author Comment

    Hi Team....

    sincere apologies for the late reply.  For some reason, the first reply from woolmilkporc  (the one using awk) did not print out the full list but the one from point_pleasant (using the cut script) seems to print out everything.   Not sure how/why these two different approaches would yield different results.   Anyway,   I have awarded the points.  Thanks to the contributors for the big help.

    Featured Post

    What Should I Do With This Threat Intelligence?

    Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

    Join & Write a Comment

    A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (,  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
    How to remove superseded packages in windows w60 or w61 installation media (.wim) or online system to prevent unnecessary space. w60 means Windows Vista or Windows Server 2008. w61 means Windows 7 or Windows Server 2008 R2. There are various …
    Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
    Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

    729 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    15 Experts available now in Live!

    Get 1:1 Help Now