Solved

Filtering rsyslog messages from vmware hosts

Posted on 2014-07-31
15
328 Views
Last Modified: 2015-04-29
I receive log messages from vmware hosts on to my rsyslog server that forwards the whole thing to elasticsearch on the same server as the rsyslog server. This generates alot of data as everything is shipped over unconditionally. I want to filter the incoming messages so that only messages that are important (level warning and worse, not informational messages) are shipped to elasticsearch. The JSON formatted log messages elasticsearch is receiving look like this:

{"@timestamp":"2014-07-31T12:43:24.024Z","host":"vmware-host-fqdn","severity":"info","facility":"local4","tag":"Vpxa:","message":" [FFC9B6D0 verbose 'VpxaHalCnxHostagent' opID=WFU-169d8d35] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd"}

Here severity is info, this message is unwanted and simply floods my Elasticsearch server with nonsense.

I have the following configuration for sending from rsyslog to Elasticsearch in the file  /etc/rsyslog.d/30-elasticsearch.conf:

$ModLoad /usr/local/lib/rsyslog/omelasticsearch.so
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="vmware-index"
  type="list") {
    constant(value="vmware-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

You can see I have tried some filtering, but it does not seem to be working. How can I filter away those severity: info messages so that they never arrive at the Elasticsearch server?
0
Comment
Question by:itnifl
  • 8
  • 5
  • 2
15 Comments
 
LVL 61

Expert Comment

by:gheist
Comment Utility
i keep just "vmkernel" and "Hostd" programs, and leave rest with vcenter.
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
What is the synthax in rsyslog configuration files for doing that?
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
No idea, i know syslog-ng
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
Umm.. I am in a blind end again, how would you do it in syslog-ng?
0
 
LVL 38

Accepted Solution

by:
Aaron Tomosky earned 500 total points
Comment Utility
for rsyslog you put the filter right before the action (then a tab) like this:
*.warn;mail.none;authpriv.none;cron.none	action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

Open in new window

0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
Hello Aaron! Thank you for keeping this thread alive. My filter as shown in the question is like the one below. It doesn't work as intended. I will try yours out when I get back to work from vacation in a weeks time.

*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

//ans so on...

Open in new window

0
 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
Comment Utility
check if it's .warn or .warning in the current logs going to ES
rsyslog error levels are inclusive so .warn includes anything higher like err, crit, etc...
0
Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

 
LVL 2

Author Comment

by:itnifl
Comment Utility
I will be taking a look at this right after the weekend
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
As of before I do the following changes, it looks like I am mostly receiving info messages in Elasticsearch:
Info messages

I am not sure how else you want me to check if it's .warn or .warning in the current logs going to ES?

When I look into the /var/log/messages log file, I see the warning messages being referred to as 'warning'.
They look like this:
Sep  2 04:19:20 hostname.vmware.local Hostd: [5DF8BB90 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 577536 for scsi0:3 is out of range -- 577536,prevBytes = 174208512 curBytes = 176518656 prevCommands = 3875curCommands = 3879

As you can see from the picture above, these warnings are not showing in ES from what I can see.
But then I see there are not being created any more indexes, I guess because the root file system is full:

curl http://localhost:9200/_aliases?pre                                                                                                                                                             tty=1
{
  "vmware-2014.08.11" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },
  "vmware-2014.08.10" : {
    "aliases" : { }
  },
  "system" : {
    "aliases" : { }
  },
  "vmware-2014.08.03" : {
    "aliases" : { }
  },
  "vmware-2014.08.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.08" : {
    "aliases" : { }
  },
  "vmware-2014.08.09" : {
    "aliases" : { }
  },
  "vmware-2014.08.06" : {
    "aliases" : { }
  },
  "vmware-2014.08.07" : {
    "aliases" : { }
  },
  "vmware-2014.08.12" : {
    "aliases" : { }
  },
  "vmware-2014.08.01" : {
    "aliases" : { }
  },
  "vmware-2014.07.31" : {
    "aliases" : { }
  },
  "vmware-2014.08.04" : {
    "aliases" : { }
  },
  "vmware-2014.08.05" : {
    "aliases" : { }
  }
}

I have a script under /etc/cron.daily that should clean up old vmware indexes like this - I guess ut is not working even if it seemed to do so while testing:
#!/bin/bash
##
## Cleans up indexes for vmware logging in Elasticsearch
## atle@team-holm.net - 01.08.2014
##

DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%y`

DAYn=$(echo $DAY | sed 's/^0*//')
MONTHn=$(echo $MONTH | sed 's/^0*//')
YEARn=$(echo $YEAR | sed 's/^0*//')

if [ `expr $DAYn - 8` -lt 1 ] && [ $MONTHn -eq 1 ]; then
        YEARn=`expr YEARn - 1`
        YEAR="0$YEARn"
fi

if [ `expr $DAYn - 8` -lt 1 ]; then
        nMONTHn=`expr $MONTHn - 1`
        if [ $nMONTHn -lt 10 ]; then
                nMONTHn="0$nMONTHn"
        fi
        nDAYn1=`expr 31 - $DAYn - 8`
        nDAYn2=`expr 30 - $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi
        if [ $nDAYn2 -lt 10 ]; then
                nDAYn2="0$nDAYn2"
        fi
        echo -e "\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2"
        echo -e "\n"
else
        nDAYn1=`expr $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi

        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m*Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1"
        echo -e "\n"
fi

Open in new window

I will clean up the indexes and see where that goes.

In /etc/rsyslog.conf I changed from:
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

to:

*.warn;mail.none;authpriv.none;cron.none                /var/log/messages

I guess it should have no effect, but did it anyway.

In /etc/rsyslog.d/30-elasticsearch.conf I changed the line from:
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

to:

*.warn;mail.none;authpriv.none;cron.none

Now we just have to wait and see.
0
 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
Comment Utility
go get this (I use git to "install" it):
git clone https://github.com/logstash/expire-logs.git
cd expire-logs
python logstash_index_cleaner.py --help
#if you see other missing stuff just pip install nameofmissing

I use it like this:
python ~/expire-logs/logstash_index_cleaner.py -d 7 #will delete anything older than 7 days
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
I get this:

:/usr/src/expire-logs/expire-logs# ls
CHANGELOG  CONTRIBUTORS  curator  LICENSE.txt  MANIFEST.in  README.md  setup.cfg  setup.py  test_curator  VERSION
:/usr/src/expire-logs/expire-logs# python setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
0
 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
Comment Utility
Ahh it's now been renamed curator as the 3rd iteration of its existence. So my command isn't accurate any more but it looks to be more powerful than before

https://github.com/elasticsearch/curator/blob/master/README.md
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
Looks like a nice utility:

 #curator delete --older-than 7
2014-09-04 15:02:53,559 INFO      Job starting...
2014-09-04 15:02:53,566 INFO      Beginning DELETE operations...
2014-09-04 15:02:53,569 INFO      DELETE index operations completed.
2014-09-04 15:02:53,569 INFO      Done in 0:00:00.036160.

Open in new window


Still shows a bunch of indexes:

~# curl http://localhost:9200/_aliases?pretty=1
{
  "vmware-2014.09.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.23" : {
    "aliases" : { }
  },
  "vmware-2014.09.01" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },

Open in new window

0
 
LVL 2

Author Closing Comment

by:itnifl
Comment Utility
I actually never got around to follow this up, and it seems that the solution that was built has been taken down for the use of another one.
I might have another try at this system some other day.
Giving points for all the good attempts and advices here.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
Check out sexilog, they put together a nice ready to run setup.
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Join & Write a Comment

Suggested Solutions

HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
In this step by step tutorial with screenshots, we will show you HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 6.5 (ESXi 6.5). This is important if you need to enable SSH remote access for additional troubleshooting of the ESXi hos…
Teach the user how to install and configure the vCenter Orchestrator virtual appliance Open vSphere Web Client: Deploy vCenter Orchestrator virtual appliance OVA file: Verify vCenter Orchestrator virtual appliance boots successfully: Connect to the …
Teach the user how to use create log bundles for vCenter Server or ESXi hosts Open vSphere Web Client: Generate vCenter Server and ESXi host log bundle:  Open vCenter Server Appliance Web Management interface and generate log bundle: Open vCenter Se…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now