log aggregation server linux

hey guys i want to have a central log server, where all of the various logs from my devices go too. My servers, switches, firewalls, send all logs to this point and just collect. What is the best linux distro, or even unix, to faciliate manage this?

mrbayItAsked: 2014-02-25

Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

Experts Exchange Solution brought to you by

Enjoy your complimentary solution view.

Get every solution instantly with Premium. Start your 7-day free trial.

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

LVL 21

netcmhCommented: 2014-02-25

While the options of the software itself are plentiful - Zenoss, Logzilla, Splunk, AlienVault, Octopussy etc, the OS or distro choice would boil down to your level of expertise in Linux.

More apps can be found at http://www.infosecisland.com/blogview/12772-Open-Source-Log-Management-Tools-List.html

My preference would be Debian GNU for it's packaging and ease of use for me.

There's a live CD solution that you might want to look into just to get an idea: https://www.wzdftpd.net/redmine/projects/siem-live

mrbayItAuthor Commented: 2014-02-25

i know there were software options, i just recall hearing some Linux distros are optimized for log processing speed wise either via the file system or its structure somehow, but i couldn't remember which one

LVL 21

netcmhCommented: 2014-02-25

Here's a great writeup on how to select your distros: http://www.linuxinsider.com/story/79717.html

And, here's the distros list:http://distrowatch.com/

If you're looking for enterprise grade, I would recommend Fedora. If it's a small to medium enough infrastructure, Debian.

LVL 26

madunix (Fadi SODAH)Commented: 2014-02-25

You can direct logs from all systems in your network to one centralized server in linux RHEL via syslogd daemon. Configuration changes are generally required on both client and server to achieve this.
https://access.redhat.com/site/solutions/2725
http://www.howtoforge.com/syslog-better-logging-tutorial

LVL 28

skullnobrainsCommented: 2014-03-01

the os does not matter much. (i like freebsd so i'm gonna advise freebsd, each and every one of us will advise our favorite os/dist)

the tool will be syslog or one of it's derivatives. most linuxes ship with ksyslogd or rsyslogd, freebsd has it's own version of the original syslogd

i know there were software options, i just recall hearing some Linux distros are optimized for log processing speed wise either via the file system or its structure somehow, but i couldn't remember which one

a log filesystem such as hammer might seem to be a good choice performance-wise

i personally would not go in that direction : what kind of volumes do you expect ? would that really exceed what a regular machine can do ?

if you're looking towards high performance writes, a good idea might be using zfs (on bsd or solaris) in simili-raid10 with as many disks as needed and possibly a tiny SSD for write cache (alias ZIL alias zpool log device)

----

also note that configuring remote logs in syslog does not guarantee the logs will be actually logged. your devices may buffer the logs for a little while but when the machine is down, the logs will be lost. likewise if the disks are not fast enough, syslog will throw the data away.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial