Assessment of Splunk to analyse security logs

Posted on 2014-08-21
Last Modified: 2014-09-03
I have a few million IPS security logs every month to analyse:
need to assess the top few events & assess if they are valid
concern or invalid with reasons to be given.

if the source & destination are both internal source IP
addresses, then not a concern;  
if source is an external public IP, then is it a known
malicious source ?
Is behaviour of the events malicious/suspicious?

Can Splunk perform the above functions?

Is Splunk free & what is the system requirement to run it
(Can Win XP, Win 7 run it, what's the amt of RAM, ... ?

Pls point me to a Quick Start User guide to get me
started to use & do the analysis
Question by:sunhux

    Author Comment

    Feel free to suggest any other free analytics software/tools (with intelligence
    to detect suspicious activities esp high volume ones) that could do the above
    LVL 57

    Assisted Solution

    A1:  You can get modules for Splunk that can do some of what you are asking.  Some of it you would need to figure out.  Example: I don't think Splunk maintains a data base of malicious IP addresses.  That would be something you would need to supply.

    A2:  Couple of links.  Splunk is free for indexing up to 500MB of log data per day.  If you go over 500MB or you want Enterprise you need to pay.  First link shows platforms you can run it on.  Second link talks about pricing.

    A3: Go to the splunk site.  Lots of information there.
    LVL 38

    Assisted Solution

    by:Rich Rumble
    Look into a product that is actually free (Splunk is not fully free) ELK
    Splunk is considerably hobbled in the free version
    You do need to be familiar with Linux, and that goes for Splunk as well.
    ELK for windows is somewhat trickier than it is with Splunk, but you do need a fwd'r with both (the wmi methods in splunk are not recommended)

    This subject however is like asking what your favorite icecream flavor is... everyone will have their own deeply felt opinion. I didn't like any SIEM I tried until ELK, and we had most of them at one time or another. RSA envision, Logrythim, loglogic, arcsight, Splunk(still have) and alienvault, that's pretty much the whole Gartner magic quadrant...
    LVL 10

    Assisted Solution

    by:Schuyler Dorsey
    I would check out AlienVault OSSIM. Not sure if it does everything you need but it's a great open source SIEM.
    LVL 60

    Accepted Solution

    Quick tutorial - Splunk runs with either an Enterprise license or a Free license. When you download Splunk for the first time, you get an Enterprise trial license that expires after 60 days. This trial license enables 500 MB/day indexing and all of the Enterprise features.Once you install Splunk, you can run with the Enterprise trial license until it expires, switch to the perpetual Free license (it's included!), or purchase an Enterprise license.
    Any search can be run on a schedule, and scheduled searches can be set up to trigger notifications or when specific conditions occur. This automated alerting functionality works across the wide range of components and technologies throughout your IT infrastructure--from applications to firewalls to access controls.

    You need the Splunk Forwarder with your Windows server (or security capturing device) to send event log into Splunk server fro indexing and search and monitoring.  Splunk can be used as an intrusion detection system but it is recommended here only as an enhancement to an existing one. Splunk server comes pre-installed with the
    Search App. This is where custom searches can be done across all indexed data sources. It can be accessed through the Splunk server web interface.
    Furthermore, its alerting can be based on scheduling options such as
    1) Trigger in real-time whenever a result matches
    2) Run on a schedule once every…
    3) Monitor in real-time over a rolling window of…
    See more on "Discovering Security Events of Interest Using Splunk"

    Likely the case for you is leveraging and looking out for " event correlation " e.g. group events that included the offending IP addresses in your own events. E.g. f someone has logged in using SSH and their source IP is one that is in the list of malicious IP addresses (the transaction command does the grouping) within a day’s span, and the number of sourcetypes in the grouping is greater than one so we know both sourcetypes were in in the grouping, then return results

    The flexibility comes in there are variety of Splunk Apps that offer specialized insight into your data and systems with pre-configured dashboards, reports, data inputs, and saved searches. Apps can include new views and dashboards that completely reconfigure the way Splunk looks.  Splunk Apps is like the Splunk app marketplace. You can download apps and add-ons for use in your Splunk environment, or you can create your own and share them with other members of the Splunk community.

    E.g. Community:Splunk for Network Security - offers a set of reports, saved searches, and dashboards, as well as corresponding alerts that you can use to monitor your firewalls, intrusion detection and prevention systems, as well as operating systems.
    E.g. The Splunk for Microsoft Windows add-on includes predefined inputs to collect data from Windows systems
    E.g.  Splunk App for Windows Infrastructure gives you deep visibility into the health and performance of your Microsoft Windows Server and Active Directory environments. Include monitor security events, such as virus outbreaks and anomalous logons

    LVL 60

    Assisted Solution

    may include

    AlienVault Unified Security Management -
    SAGAN - open source (GNU/GPLv2) high performance, real-time log analysis & correlation engine. It serves as real time event log monitoring system that is able to detect incidents on hosts or network and can correlate information with the snort sensor present on your network. It gathers syslog events and then correlates them with other alerts such as snort logs.
    Find the latest ruleset at

    alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.
    Fluentd  (Collecting Log Data from Windows) -
     nxlog on Windows -

    Author Comment

    So after 60 days, so long as I pump in less than 500mb per day,  the full enterprise features?  As I have 6 different IPSes I tot of 6 splunks as I m currently required to analyse the 6 groups of data logs separately anyway.   Tks for the other alternative products whih we can only look at freewares

    Author Comment

    I m linux trained so as long as the full enterprise features ( other tan 500mb/day) with community support is availble, that will be great.  Is there any feature to archive data or indexes say  more than 6 months old?  500mb per day can grow pretty fast n we are short of storage
    LVL 38

    Assisted Solution

    by:Rich Rumble
    You're going to have to try a few solutions to see what fit's your needs. I've been down this road with all my clients, and I'm converting them to ELK where I can. I do have a client that is using Hadoop to store the data and Splunk to index/search it, and that get's around the 500Mb restriction. They are filtering the events prior to being indexed by Splunk so maybe that is the key. I'm still happier with ELK, and I'm making converts daily :)
    LVL 60

    Assisted Solution

    Yes the limit is there till Enterprise trial expired as mentioned below and thereafter, it convert to Splunk free with exception to full features (e.g. no alert/monitoring and more). Also do note the violation which can cause disabled searching

    When you first install a downloaded copy of Splunk, that instance of Splunk is using a 60 day Trial Enterprise license. This license allows you to try out all of the Enterprise features in Splunk for 60 days, and to index up to 500 MB of data per day.

    Once the 60 day trial expires (and if you have not purchased and installed an Enterprise license), you are given the option to switch to Splunk Free. Splunk Free includes a subset of the features of Splunk Enterprise and is intended for use in standalone deployments and for short-term forensic investigations. It allows you to index up to 500 MB of data a day indefinitely.

    Important: Splunk Free does not include authentication or scheduled searches/alerting. This means that any user accessing your Splunk installation (via Splunk Web or the CLI) will not have to provide credentials. Additionally, scheduled saved searches/alerts will no longer fire.
    You can exceed your Enterprise license 4 times within 30 days--the 5th time, search will be disabled. You can exceed your Free licenses 2 times, and the 3rd time, search will be disabled.

    Actually the alternate have some that is opensource such as SAGAN

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Top 6 Sources for Identifying Threat Actor TTPs

    Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

    When the confidentiality and security of your data is a must, trust the highly encrypted cloud fax portfolio used by 12 million businesses worldwide, including nearly half of the Fortune 500.
    This paper addresses the security of Sennheiser DECT Contact Center and Office (CC&O) headsets. It describes the DECT security chain comprised of “Pairing”, “Per Call Authentication” and “Encryption”, which are all part of the standard DECT protocol.
    Sending a Secure fax is easy with eFax Corporate ( First, Just open a new email message.  In the To field, type your recipient's fax number You can even send a secure international fax — just include t…
    Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

    794 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    16 Experts available now in Live!

    Get 1:1 Help Now