Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 483
  • Last Modified:

Filtering rsyslog messages from vmware hosts

I receive log messages from vmware hosts on to my rsyslog server that forwards the whole thing to elasticsearch on the same server as the rsyslog server. This generates alot of data as everything is shipped over unconditionally. I want to filter the incoming messages so that only messages that are important (level warning and worse, not informational messages) are shipped to elasticsearch. The JSON formatted log messages elasticsearch is receiving look like this:

{"@timestamp":"2014-07-31T12:43:24.024Z","host":"vmware-host-fqdn","severity":"info","facility":"local4","tag":"Vpxa:","message":" [FFC9B6D0 verbose 'VpxaHalCnxHostagent' opID=WFU-169d8d35] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd"}

Here severity is info, this message is unwanted and simply floods my Elasticsearch server with nonsense.

I have the following configuration for sending from rsyslog to Elasticsearch in the file  /etc/rsyslog.d/30-elasticsearch.conf:

$ModLoad /usr/local/lib/rsyslog/omelasticsearch.so
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="vmware-index"
  type="list") {
    constant(value="vmware-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

You can see I have tried some filtering, but it does not seem to be working. How can I filter away those severity: info messages so that they never arrive at the Elasticsearch server?
0
itnifl
Asked:
itnifl
  • 8
  • 5
  • 2
4 Solutions
 
gheistCommented:
i keep just "vmkernel" and "Hostd" programs, and leave rest with vcenter.
0
 
itniflAuthor Commented:
What is the synthax in rsyslog configuration files for doing that?
0
 
gheistCommented:
No idea, i know syslog-ng
0
NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

 
itniflAuthor Commented:
Umm.. I am in a blind end again, how would you do it in syslog-ng?
0
 
Aaron TomoskyTechnology ConsultantCommented:
for rsyslog you put the filter right before the action (then a tab) like this:
*.warn;mail.none;authpriv.none;cron.none	action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

Open in new window

0
 
itniflAuthor Commented:
Hello Aaron! Thank you for keeping this thread alive. My filter as shown in the question is like the one below. It doesn't work as intended. I will try yours out when I get back to work from vacation in a weeks time.

*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

//ans so on...

Open in new window

0
 
Aaron TomoskyTechnology ConsultantCommented:
check if it's .warn or .warning in the current logs going to ES
rsyslog error levels are inclusive so .warn includes anything higher like err, crit, etc...
0
 
itniflAuthor Commented:
I will be taking a look at this right after the weekend
0
 
itniflAuthor Commented:
As of before I do the following changes, it looks like I am mostly receiving info messages in Elasticsearch:
Info messages

I am not sure how else you want me to check if it's .warn or .warning in the current logs going to ES?

When I look into the /var/log/messages log file, I see the warning messages being referred to as 'warning'.
They look like this:
Sep  2 04:19:20 hostname.vmware.local Hostd: [5DF8BB90 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 577536 for scsi0:3 is out of range -- 577536,prevBytes = 174208512 curBytes = 176518656 prevCommands = 3875curCommands = 3879

As you can see from the picture above, these warnings are not showing in ES from what I can see.
But then I see there are not being created any more indexes, I guess because the root file system is full:

curl http://localhost:9200/_aliases?pre                                                                                                                                                             tty=1
{
  "vmware-2014.08.11" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },
  "vmware-2014.08.10" : {
    "aliases" : { }
  },
  "system" : {
    "aliases" : { }
  },
  "vmware-2014.08.03" : {
    "aliases" : { }
  },
  "vmware-2014.08.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.08" : {
    "aliases" : { }
  },
  "vmware-2014.08.09" : {
    "aliases" : { }
  },
  "vmware-2014.08.06" : {
    "aliases" : { }
  },
  "vmware-2014.08.07" : {
    "aliases" : { }
  },
  "vmware-2014.08.12" : {
    "aliases" : { }
  },
  "vmware-2014.08.01" : {
    "aliases" : { }
  },
  "vmware-2014.07.31" : {
    "aliases" : { }
  },
  "vmware-2014.08.04" : {
    "aliases" : { }
  },
  "vmware-2014.08.05" : {
    "aliases" : { }
  }
}

I have a script under /etc/cron.daily that should clean up old vmware indexes like this - I guess ut is not working even if it seemed to do so while testing:
#!/bin/bash
##
## Cleans up indexes for vmware logging in Elasticsearch
## atle@team-holm.net - 01.08.2014
##

DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%y`

DAYn=$(echo $DAY | sed 's/^0*//')
MONTHn=$(echo $MONTH | sed 's/^0*//')
YEARn=$(echo $YEAR | sed 's/^0*//')

if [ `expr $DAYn - 8` -lt 1 ] && [ $MONTHn -eq 1 ]; then
        YEARn=`expr YEARn - 1`
        YEAR="0$YEARn"
fi

if [ `expr $DAYn - 8` -lt 1 ]; then
        nMONTHn=`expr $MONTHn - 1`
        if [ $nMONTHn -lt 10 ]; then
                nMONTHn="0$nMONTHn"
        fi
        nDAYn1=`expr 31 - $DAYn - 8`
        nDAYn2=`expr 30 - $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi
        if [ $nDAYn2 -lt 10 ]; then
                nDAYn2="0$nDAYn2"
        fi
        echo -e "\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2"
        echo -e "\n"
else
        nDAYn1=`expr $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi

        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m*Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1"
        echo -e "\n"
fi

Open in new window

I will clean up the indexes and see where that goes.

In /etc/rsyslog.conf I changed from:
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

to:

*.warn;mail.none;authpriv.none;cron.none                /var/log/messages

I guess it should have no effect, but did it anyway.

In /etc/rsyslog.d/30-elasticsearch.conf I changed the line from:
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

to:

*.warn;mail.none;authpriv.none;cron.none

Now we just have to wait and see.
0
 
Aaron TomoskyTechnology ConsultantCommented:
go get this (I use git to "install" it):
git clone https://github.com/logstash/expire-logs.git
cd expire-logs
python logstash_index_cleaner.py --help
#if you see other missing stuff just pip install nameofmissing

I use it like this:
python ~/expire-logs/logstash_index_cleaner.py -d 7 #will delete anything older than 7 days
0
 
itniflAuthor Commented:
I get this:

:/usr/src/expire-logs/expire-logs# ls
CHANGELOG  CONTRIBUTORS  curator  LICENSE.txt  MANIFEST.in  README.md  setup.cfg  setup.py  test_curator  VERSION
:/usr/src/expire-logs/expire-logs# python setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
0
 
Aaron TomoskyTechnology ConsultantCommented:
Ahh it's now been renamed curator as the 3rd iteration of its existence. So my command isn't accurate any more but it looks to be more powerful than before

https://github.com/elasticsearch/curator/blob/master/README.md
0
 
itniflAuthor Commented:
Looks like a nice utility:

 #curator delete --older-than 7
2014-09-04 15:02:53,559 INFO      Job starting...
2014-09-04 15:02:53,566 INFO      Beginning DELETE operations...
2014-09-04 15:02:53,569 INFO      DELETE index operations completed.
2014-09-04 15:02:53,569 INFO      Done in 0:00:00.036160.

Open in new window


Still shows a bunch of indexes:

~# curl http://localhost:9200/_aliases?pretty=1
{
  "vmware-2014.09.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.23" : {
    "aliases" : { }
  },
  "vmware-2014.09.01" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },

Open in new window

0
 
itniflAuthor Commented:
I actually never got around to follow this up, and it seems that the solution that was built has been taken down for the use of another one.
I might have another try at this system some other day.
Giving points for all the good attempts and advices here.
0
 
Aaron TomoskyTechnology ConsultantCommented:
Check out sexilog, they put together a nice ready to run setup.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

  • 8
  • 5
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now