Assessment of Splunk to analyse security logs

I have a few million IPS security logs every month to analyse:
need to assess the top few events & assess if they are valid
concern or invalid with reasons to be given.

if the source & destination are both internal source IP
addresses, then not a concern;  
if source is an external public IP, then is it a known
malicious source ?
Is behaviour of the events malicious/suspicious?

Can Splunk perform the above functions?

Is Splunk free & what is the system requirement to run it
(Can Win XP, Win 7 run it, what's the amt of RAM, ... ?

Pls point me to a Quick Start User guide to get me
started to use & do the analysis
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

sunhuxAuthor Commented:
Feel free to suggest any other free analytics software/tools (with intelligence
to detect suspicious activities esp high volume ones) that could do the above
A1:  You can get modules for Splunk that can do some of what you are asking.  Some of it you would need to figure out.  Example: I don't think Splunk maintains a data base of malicious IP addresses.  That would be something you would need to supply.

A2:  Couple of links.  Splunk is free for indexing up to 500MB of log data per day.  If you go over 500MB or you want Enterprise you need to pay.  First link shows platforms you can run it on.  Second link talks about pricing.

A3: Go to the splunk site.  Lots of information there.
Rich RumbleSecurity SamuraiCommented:
Look into a product that is actually free (Splunk is not fully free) ELK
Splunk is considerably hobbled in the free version
You do need to be familiar with Linux, and that goes for Splunk as well.
ELK for windows is somewhat trickier than it is with Splunk, but you do need a fwd'r with both (the wmi methods in splunk are not recommended)

This subject however is like asking what your favorite icecream flavor is... everyone will have their own deeply felt opinion. I didn't like any SIEM I tried until ELK, and we had most of them at one time or another. RSA envision, Logrythim, loglogic, arcsight, Splunk(still have) and alienvault, that's pretty much the whole Gartner magic quadrant...
The Lifecycle Approach to Managing Security Policy

Managing application connectivity and security policies can be achieved more effectively when following a framework that automates repeatable processes and ensures that the right activities are performed in the right order.

Schuyler DorseyCommented:
I would check out AlienVault OSSIM. Not sure if it does everything you need but it's a great open source SIEM.
btanExec ConsultantCommented:
Quick tutorial - Splunk runs with either an Enterprise license or a Free license. When you download Splunk for the first time, you get an Enterprise trial license that expires after 60 days. This trial license enables 500 MB/day indexing and all of the Enterprise features.Once you install Splunk, you can run with the Enterprise trial license until it expires, switch to the perpetual Free license (it's included!), or purchase an Enterprise license.
Any search can be run on a schedule, and scheduled searches can be set up to trigger notifications or when specific conditions occur. This automated alerting functionality works across the wide range of components and technologies throughout your IT infrastructure--from applications to firewalls to access controls.

You need the Splunk Forwarder with your Windows server (or security capturing device) to send event log into Splunk server fro indexing and search and monitoring.  Splunk can be used as an intrusion detection system but it is recommended here only as an enhancement to an existing one. Splunk server comes pre-installed with the
Search App. This is where custom searches can be done across all indexed data sources. It can be accessed through the Splunk server web interface.
Furthermore, its alerting can be based on scheduling options such as
1) Trigger in real-time whenever a result matches
2) Run on a schedule once every…
3) Monitor in real-time over a rolling window of…
See more on "Discovering Security Events of Interest Using Splunk"

Likely the case for you is leveraging and looking out for " event correlation " e.g. group events that included the offending IP addresses in your own events. E.g. f someone has logged in using SSH and their source IP is one that is in the list of malicious IP addresses (the transaction command does the grouping) within a day’s span, and the number of sourcetypes in the grouping is greater than one so we know both sourcetypes were in in the grouping, then return results

The flexibility comes in there are variety of Splunk Apps that offer specialized insight into your data and systems with pre-configured dashboards, reports, data inputs, and saved searches. Apps can include new views and dashboards that completely reconfigure the way Splunk looks.  Splunk Apps is like the Splunk app marketplace. You can download apps and add-ons for use in your Splunk environment, or you can create your own and share them with other members of the Splunk community.

E.g. Community:Splunk for Network Security - offers a set of reports, saved searches, and dashboards, as well as corresponding alerts that you can use to monitor your firewalls, intrusion detection and prevention systems, as well as operating systems.
E.g. The Splunk for Microsoft Windows add-on includes predefined inputs to collect data from Windows systems
E.g.  Splunk App for Windows Infrastructure gives you deep visibility into the health and performance of your Microsoft Windows Server and Active Directory environments. Include monitor security events, such as virus outbreaks and anomalous logons


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
btanExec ConsultantCommented:
may include

AlienVault Unified Security Management -
SAGAN - open source (GNU/GPLv2) high performance, real-time log analysis & correlation engine. It serves as real time event log monitoring system that is able to detect incidents on hosts or network and can correlate information with the snort sensor present on your network. It gathers syslog events and then correlates them with other alerts such as snort logs.
Find the latest ruleset at

alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.
Fluentd  (Collecting Log Data from Windows) -
 nxlog on Windows -
sunhuxAuthor Commented:
So after 60 days, so long as I pump in less than 500mb per day,  the full enterprise features?  As I have 6 different IPSes I tot of 6 splunks as I m currently required to analyse the 6 groups of data logs separately anyway.   Tks for the other alternative products whih we can only look at freewares
sunhuxAuthor Commented:
I m linux trained so as long as the full enterprise features ( other tan 500mb/day) with community support is availble, that will be great.  Is there any feature to archive data or indexes say  more than 6 months old?  500mb per day can grow pretty fast n we are short of storage
Rich RumbleSecurity SamuraiCommented:
You're going to have to try a few solutions to see what fit's your needs. I've been down this road with all my clients, and I'm converting them to ELK where I can. I do have a client that is using Hadoop to store the data and Splunk to index/search it, and that get's around the 500Mb restriction. They are filtering the events prior to being indexed by Splunk so maybe that is the key. I'm still happier with ELK, and I'm making converts daily :)
btanExec ConsultantCommented:
Yes the limit is there till Enterprise trial expired as mentioned below and thereafter, it convert to Splunk free with exception to full features (e.g. no alert/monitoring and more). Also do note the violation which can cause disabled searching

When you first install a downloaded copy of Splunk, that instance of Splunk is using a 60 day Trial Enterprise license. This license allows you to try out all of the Enterprise features in Splunk for 60 days, and to index up to 500 MB of data per day.

Once the 60 day trial expires (and if you have not purchased and installed an Enterprise license), you are given the option to switch to Splunk Free. Splunk Free includes a subset of the features of Splunk Enterprise and is intended for use in standalone deployments and for short-term forensic investigations. It allows you to index up to 500 MB of data a day indefinitely.

Important: Splunk Free does not include authentication or scheduled searches/alerting. This means that any user accessing your Splunk installation (via Splunk Web or the CLI) will not have to provide credentials. Additionally, scheduled saved searches/alerts will no longer fire.
You can exceed your Enterprise license 4 times within 30 days--the 5th time, search will be disabled. You can exceed your Free licenses 2 times, and the 3rd time, search will be disabled.

Actually the alternate have some that is opensource such as SAGAN
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.