Solved

Filtering rsyslog messages from vmware hosts

Posted on 2014-07-31
15
409 Views
Last Modified: 2015-04-29
I receive log messages from vmware hosts on to my rsyslog server that forwards the whole thing to elasticsearch on the same server as the rsyslog server. This generates alot of data as everything is shipped over unconditionally. I want to filter the incoming messages so that only messages that are important (level warning and worse, not informational messages) are shipped to elasticsearch. The JSON formatted log messages elasticsearch is receiving look like this:

{"@timestamp":"2014-07-31T12:43:24.024Z","host":"vmware-host-fqdn","severity":"info","facility":"local4","tag":"Vpxa:","message":" [FFC9B6D0 verbose 'VpxaHalCnxHostagent' opID=WFU-169d8d35] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd"}

Here severity is info, this message is unwanted and simply floods my Elasticsearch server with nonsense.

I have the following configuration for sending from rsyslog to Elasticsearch in the file  /etc/rsyslog.d/30-elasticsearch.conf:

$ModLoad /usr/local/lib/rsyslog/omelasticsearch.so
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="vmware-index"
  type="list") {
    constant(value="vmware-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

You can see I have tried some filtering, but it does not seem to be working. How can I filter away those severity: info messages so that they never arrive at the Elasticsearch server?
0
Comment
Question by:itnifl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 5
  • 2
15 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 40231788
i keep just "vmkernel" and "Hostd" programs, and leave rest with vcenter.
0
 
LVL 2

Author Comment

by:itnifl
ID: 40232513
What is the synthax in rsyslog configuration files for doing that?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40232586
No idea, i know syslog-ng
0
NEW Veeam Agent for Microsoft Windows

Backup and recover physical and cloud-based servers and workstations, as well as endpoint devices that belong to remote users. Avoid downtime and data loss quickly and easily for Windows-based physical or public cloud-based workloads!

 
LVL 2

Author Comment

by:itnifl
ID: 40232649
Umm.. I am in a blind end again, how would you do it in syslog-ng?
0
 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 500 total points
ID: 40254293
for rsyslog you put the filter right before the action (then a tab) like this:
*.warn;mail.none;authpriv.none;cron.none	action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

Open in new window

0
 
LVL 2

Author Comment

by:itnifl
ID: 40262409
Hello Aaron! Thank you for keeping this thread alive. My filter as shown in the question is like the one below. It doesn't work as intended. I will try yours out when I get back to work from vacation in a weeks time.

*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

//ans so on...

Open in new window

0
 
LVL 39

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
ID: 40262419
check if it's .warn or .warning in the current logs going to ES
rsyslog error levels are inclusive so .warn includes anything higher like err, crit, etc...
0
 
LVL 2

Author Comment

by:itnifl
ID: 40294473
I will be taking a look at this right after the weekend
0
 
LVL 2

Author Comment

by:itnifl
ID: 40297891
As of before I do the following changes, it looks like I am mostly receiving info messages in Elasticsearch:
Info messages

I am not sure how else you want me to check if it's .warn or .warning in the current logs going to ES?

When I look into the /var/log/messages log file, I see the warning messages being referred to as 'warning'.
They look like this:
Sep  2 04:19:20 hostname.vmware.local Hostd: [5DF8BB90 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 577536 for scsi0:3 is out of range -- 577536,prevBytes = 174208512 curBytes = 176518656 prevCommands = 3875curCommands = 3879

As you can see from the picture above, these warnings are not showing in ES from what I can see.
But then I see there are not being created any more indexes, I guess because the root file system is full:

curl http://localhost:9200/_aliases?pre                                                                                                                                                             tty=1
{
  "vmware-2014.08.11" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },
  "vmware-2014.08.10" : {
    "aliases" : { }
  },
  "system" : {
    "aliases" : { }
  },
  "vmware-2014.08.03" : {
    "aliases" : { }
  },
  "vmware-2014.08.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.08" : {
    "aliases" : { }
  },
  "vmware-2014.08.09" : {
    "aliases" : { }
  },
  "vmware-2014.08.06" : {
    "aliases" : { }
  },
  "vmware-2014.08.07" : {
    "aliases" : { }
  },
  "vmware-2014.08.12" : {
    "aliases" : { }
  },
  "vmware-2014.08.01" : {
    "aliases" : { }
  },
  "vmware-2014.07.31" : {
    "aliases" : { }
  },
  "vmware-2014.08.04" : {
    "aliases" : { }
  },
  "vmware-2014.08.05" : {
    "aliases" : { }
  }
}

I have a script under /etc/cron.daily that should clean up old vmware indexes like this - I guess ut is not working even if it seemed to do so while testing:
#!/bin/bash
##
## Cleans up indexes for vmware logging in Elasticsearch
## atle@team-holm.net - 01.08.2014
##

DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%y`

DAYn=$(echo $DAY | sed 's/^0*//')
MONTHn=$(echo $MONTH | sed 's/^0*//')
YEARn=$(echo $YEAR | sed 's/^0*//')

if [ `expr $DAYn - 8` -lt 1 ] && [ $MONTHn -eq 1 ]; then
        YEARn=`expr YEARn - 1`
        YEAR="0$YEARn"
fi

if [ `expr $DAYn - 8` -lt 1 ]; then
        nMONTHn=`expr $MONTHn - 1`
        if [ $nMONTHn -lt 10 ]; then
                nMONTHn="0$nMONTHn"
        fi
        nDAYn1=`expr 31 - $DAYn - 8`
        nDAYn2=`expr 30 - $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi
        if [ $nDAYn2 -lt 10 ]; then
                nDAYn2="0$nDAYn2"
        fi
        echo -e "\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2"
        echo -e "\n"
else
        nDAYn1=`expr $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi

        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m*Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1"
        echo -e "\n"
fi

Open in new window

I will clean up the indexes and see where that goes.

In /etc/rsyslog.conf I changed from:
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

to:

*.warn;mail.none;authpriv.none;cron.none                /var/log/messages

I guess it should have no effect, but did it anyway.

In /etc/rsyslog.d/30-elasticsearch.conf I changed the line from:
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

to:

*.warn;mail.none;authpriv.none;cron.none

Now we just have to wait and see.
0
 
LVL 39

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
ID: 40298881
go get this (I use git to "install" it):
git clone https://github.com/logstash/expire-logs.git
cd expire-logs
python logstash_index_cleaner.py --help
#if you see other missing stuff just pip install nameofmissing

I use it like this:
python ~/expire-logs/logstash_index_cleaner.py -d 7 #will delete anything older than 7 days
0
 
LVL 2

Author Comment

by:itnifl
ID: 40301040
I get this:

:/usr/src/expire-logs/expire-logs# ls
CHANGELOG  CONTRIBUTORS  curator  LICENSE.txt  MANIFEST.in  README.md  setup.cfg  setup.py  test_curator  VERSION
:/usr/src/expire-logs/expire-logs# python setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
0
 
LVL 39

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
ID: 40301093
Ahh it's now been renamed curator as the 3rd iteration of its existence. So my command isn't accurate any more but it looks to be more powerful than before

https://github.com/elasticsearch/curator/blob/master/README.md
0
 
LVL 2

Author Comment

by:itnifl
ID: 40303388
Looks like a nice utility:

 #curator delete --older-than 7
2014-09-04 15:02:53,559 INFO      Job starting...
2014-09-04 15:02:53,566 INFO      Beginning DELETE operations...
2014-09-04 15:02:53,569 INFO      DELETE index operations completed.
2014-09-04 15:02:53,569 INFO      Done in 0:00:00.036160.

Open in new window


Still shows a bunch of indexes:

~# curl http://localhost:9200/_aliases?pretty=1
{
  "vmware-2014.09.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.23" : {
    "aliases" : { }
  },
  "vmware-2014.09.01" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },

Open in new window

0
 
LVL 2

Author Closing Comment

by:itnifl
ID: 40752042
I actually never got around to follow this up, and it seems that the solution that was built has been taken down for the use of another one.
I might have another try at this system some other day.
Giving points for all the good attempts and advices here.
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 40752045
Check out sexilog, they put together a nice ready to run setup.
0

Featured Post

RHCE - Red Hat OpenStack Prep Course

This course will provide in-depth training so that students who currently hold the EX200 & EX210 certifications can sit for the EX310 exam. Students will learn how to deploy & manage a full Red Hat environment with Ceph block storage, & integrate Ceph into other OpenStack service

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, I will show you HOW TO: Install VMware Tools for Windows on a VMware Windows virtual machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, using the VMware Host Client. The virtual machine has Windows Server 2016 instal…
Ransomware is a malware that is again in the list of security  concerns. Not only for companies, but also for Government security and  even at personal use. IT departments should be aware and have the right  knowledge to how to fight it.
Teach the user how to use create log bundles for vCenter Server or ESXi hosts Open vSphere Web Client: Generate vCenter Server and ESXi host log bundle:  Open vCenter Server Appliance Web Management interface and generate log bundle: Open vCenter Se…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

623 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question