Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Assessment of Splunk to analyse security logs

Posted on 2014-08-21
10
Medium Priority
?
1,380 Views
Last Modified: 2014-09-03
I have a few million IPS security logs every month to analyse:
need to assess the top few events & assess if they are valid
concern or invalid with reasons to be given.

Eg:
if the source & destination are both internal source IP
addresses, then not a concern;  
if source is an external public IP, then is it a known
malicious source ?
Is behaviour of the events malicious/suspicious?

Q1:
Can Splunk perform the above functions?

Q2:
Is Splunk free & what is the system requirement to run it
(Can Win XP, Win 7 run it, what's the amt of RAM, ... ?

Q3:
Pls point me to a Quick Start User guide to get me
started to use & do the analysis
0
Comment
Question by:sunhux
  • 3
  • 3
  • 2
  • +2
10 Comments
 

Author Comment

by:sunhux
ID: 40278322
Feel free to suggest any other free analytics software/tools (with intelligence
to detect suspicious activities esp high volume ones) that could do the above
function
0
 
LVL 57

Assisted Solution

by:giltjr
giltjr earned 200 total points
ID: 40279556
A1:  You can get modules for Splunk that can do some of what you are asking.  Some of it you would need to figure out.  Example: I don't think Splunk maintains a data base of malicious IP addresses.  That would be something you would need to supply.

A2:  Couple of links.  Splunk is free for indexing up to 500MB of log data per day.  If you go over 500MB or you want Enterprise you need to pay.  First link shows platforms you can run it on.  Second link talks about pricing.

http://www.splunk.com/download?r=header

http://www.splunk.com/view/pricing/SP-CAAADFV

A3: Go to the splunk site.  Lots of information there.
0
 
LVL 38

Assisted Solution

by:Rich Rumble
Rich Rumble earned 480 total points
ID: 40279568
Look into a product that is actually free (Splunk is not fully free) ELK
Splunk is considerably hobbled in the free version
http://www.splunk.com/view/SP-CAAAE8W
http://www.reddit.com/r/netsec/comments/27s20s/security_analyst_discuss_siems_such_as_arcsight/
http://www.networkassassin.com/elk-for-network-operations/
You do need to be familiar with Linux, and that goes for Splunk as well.
ELK for windows is somewhat trickier than it is with Splunk, but you do need a fwd'r with both (the wmi methods in splunk are not recommended)
http://www.ragingcomputer.com/2014/02/logstash-elasticsearch-kibana-for-windows-event-logs

This subject however is like asking what your favorite icecream flavor is... everyone will have their own deeply felt opinion. I didn't like any SIEM I tried until ELK, and we had most of them at one time or another. RSA envision, Logrythim, loglogic, arcsight, Splunk(still have) and alienvault, that's pretty much the whole Gartner magic quadrant...
-rich
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 10

Assisted Solution

by:Schuyler Dorsey
Schuyler Dorsey earned 120 total points
ID: 40279791
I would check out AlienVault OSSIM. Not sure if it does everything you need but it's a great open source SIEM.
0
 
LVL 65

Accepted Solution

by:
btan earned 1200 total points
ID: 40279995
Quick tutorial - Splunk runs with either an Enterprise license or a Free license. When you download Splunk for the first time, you get an Enterprise trial license that expires after 60 days. This trial license enables 500 MB/day indexing and all of the Enterprise features.Once you install Splunk, you can run with the Enterprise trial license until it expires, switch to the perpetual Free license (it's included!), or purchase an Enterprise license.
Any search can be run on a schedule, and scheduled searches can be set up to trigger notifications or when specific conditions occur. This automated alerting functionality works across the wide range of components and technologies throughout your IT infrastructure--from applications to firewalls to access controls.
http://www.nhhs.net/ourpages/auto/2011/10/7/51955419/Splunk-5_0_1-Tutoriala.pdf

You need the Splunk Forwarder with your Windows server (or security capturing device) to send event log into Splunk server fro indexing and search and monitoring.  Splunk can be used as an intrusion detection system but it is recommended here only as an enhancement to an existing one. Splunk server comes pre-installed with the
Search App. This is where custom searches can be done across all indexed data sources. It can be accessed through the Splunk server web interface.
Furthermore, its alerting can be based on scheduling options such as
1) Trigger in real-time whenever a result matches
2) Run on a schedule once every…
3) Monitor in real-time over a rolling window of…
See more on "Discovering Security Events of Interest Using Splunk"
http://www.sans.org/reading-room/whitepapers/logging/discovering-security-events-interest-splunk-34272

Likely the case for you is leveraging and looking out for " event correlation " e.g. group events that included the offending IP addresses in your own events. E.g. f someone has logged in using SSH and their source IP is one that is in the list of malicious IP addresses (the transaction command does the grouping) within a day’s span, and the number of sourcetypes in the grouping is greater than one so we know both sourcetypes were in in the grouping, then return results
http://blogs.splunk.com/2010/09/01/event-correlation/

The flexibility comes in there are variety of Splunk Apps that offer specialized insight into your data and systems with pre-configured dashboards, reports, data inputs, and saved searches. Apps can include new views and dashboards that completely reconfigure the way Splunk looks.  Splunk Apps is like the Splunk app marketplace. You can download apps and add-ons for use in your Splunk environment, or you can create your own and share them with other members of the Splunk community.
http://docs.slunk.com/Documentation/Splunkbase/latest/Splunkbase/Introduction

E.g. Community:Splunk for Network Security - offers a set of reports, saved searches, and dashboards, as well as corresponding alerts that you can use to monitor your firewalls, intrusion detection and prevention systems, as well as operating systems.
http://wiki.splunk.com/Apps:Splunk_for_Network_Security
E.g. The Splunk for Microsoft Windows add-on includes predefined inputs to collect data from Windows systems
http://docs.splunk.com/Documentation/WindowsAddOn/4.7.1/User/AbouttheSplunkAdd-onforWindows
E.g.  Splunk App for Windows Infrastructure gives you deep visibility into the health and performance of your Microsoft Windows Server and Active Directory environments. Include monitor security events, such as virus outbreaks and anomalous logons
http://docs.splunk.com/Documentation/MSApp/latest/MSInfra/AbouttheSplunkAppforMSInfrastructure

Alternatives
0
 
LVL 65

Assisted Solution

by:btan
btan earned 1200 total points
ID: 40279998
may include

AlienVault Unified Security Management - http://www.alienvault.com/solutions/siem-event-correlation
SAGAN - open source (GNU/GPLv2) high performance, real-time log analysis & correlation engine. It serves as real time event log monitoring system that is able to detect incidents on hosts or network and can correlate information with the snort sensor present on your network. It gathers syslog events and then correlates them with other alerts such as snort logs.
https://wiki.quadrantsec.com/twiki/bin/view/Main/SaganInstall
Find the latest ruleset at http://sagan.softwink.com/rules

alternative to Splunk by combining three open source projects: Elasticsearch, Kibana, and Fluentd.
http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd
Fluentd  (Collecting Log Data from Windows) - http://docs.fluentd.org/articles/windows
 nxlog on Windows - http://nxlog.org/features
0
 

Author Comment

by:sunhux
ID: 40280109
So after 60 days, so long as I pump in less than 500mb per day,  the full enterprise features?  As I have 6 different IPSes I tot of 6 splunks as I m currently required to analyse the 6 groups of data logs separately anyway.   Tks for the other alternative products whih we can only look at freewares
0
 

Author Comment

by:sunhux
ID: 40280115
I m linux trained so as long as the full enterprise features ( other tan 500mb/day) with community support is availble, that will be great.  Is there any feature to archive data or indexes say  more than 6 months old?  500mb per day can grow pretty fast n we are short of storage
0
 
LVL 38

Assisted Solution

by:Rich Rumble
Rich Rumble earned 480 total points
ID: 40280130
You're going to have to try a few solutions to see what fit's your needs. I've been down this road with all my clients, and I'm converting them to ELK where I can. I do have a client that is using Hadoop to store the data and Splunk to index/search it, and that get's around the 500Mb restriction. They are filtering the events prior to being indexed by Splunk so maybe that is the key. I'm still happier with ELK, and I'm making converts daily :)
-rich
0
 
LVL 65

Assisted Solution

by:btan
btan earned 1200 total points
ID: 40280195
Yes the limit is there till Enterprise trial expired as mentioned below and thereafter, it convert to Splunk free with exception to full features (e.g. no alert/monitoring and more). Also do note the violation which can cause disabled searching
http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/MoreaboutSplunkFree
http://www.splunk.com/view/SP-CAAAE8W
http://docs.splunk.com/Documentation/Splunk/6.1.3/Admin/HowSplunklicensingworks

When you first install a downloaded copy of Splunk, that instance of Splunk is using a 60 day Trial Enterprise license. This license allows you to try out all of the Enterprise features in Splunk for 60 days, and to index up to 500 MB of data per day.

Once the 60 day trial expires (and if you have not purchased and installed an Enterprise license), you are given the option to switch to Splunk Free. Splunk Free includes a subset of the features of Splunk Enterprise and is intended for use in standalone deployments and for short-term forensic investigations. It allows you to index up to 500 MB of data a day indefinitely.

Important: Splunk Free does not include authentication or scheduled searches/alerting. This means that any user accessing your Splunk installation (via Splunk Web or the CLI) will not have to provide credentials. Additionally, scheduled saved searches/alerts will no longer fire.
You can exceed your Enterprise license 4 times within 30 days--the 5th time, search will be disabled. You can exceed your Free licenses 2 times, and the 3rd time, search will be disabled.

Actually the alternate have some that is opensource such as SAGAN
0

Featured Post

VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Securing your business data in current era should be your biggest priority. Numerous people are unaware of the fact that insiders commit more than 60 percent of security breaches. You need to figure out the underlying cause and invoke your potential…
Spectre and Meltdown, how it affects me and my clients?
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …
Is your data getting by on basic protection measures? In today’s climate of debilitating malware and ransomware—like WannaCry—that may not be enough. You need to establish more than basics, like a recovery plan that protects both data and endpoints.…
Suggested Courses

580 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question