Solved

Filtering rsyslog messages from vmware hosts

Posted on 2014-07-31
15
341 Views
Last Modified: 2015-04-29
I receive log messages from vmware hosts on to my rsyslog server that forwards the whole thing to elasticsearch on the same server as the rsyslog server. This generates alot of data as everything is shipped over unconditionally. I want to filter the incoming messages so that only messages that are important (level warning and worse, not informational messages) are shipped to elasticsearch. The JSON formatted log messages elasticsearch is receiving look like this:

{"@timestamp":"2014-07-31T12:43:24.024Z","host":"vmware-host-fqdn","severity":"info","facility":"local4","tag":"Vpxa:","message":" [FFC9B6D0 verbose 'VpxaHalCnxHostagent' opID=WFU-169d8d35] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd"}

Here severity is info, this message is unwanted and simply floods my Elasticsearch server with nonsense.

I have the following configuration for sending from rsyslog to Elasticsearch in the file  /etc/rsyslog.d/30-elasticsearch.conf:

$ModLoad /usr/local/lib/rsyslog/omelasticsearch.so
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="vmware-index"
  type="list") {
    constant(value="vmware-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

You can see I have tried some filtering, but it does not seem to be working. How can I filter away those severity: info messages so that they never arrive at the Elasticsearch server?
0
Comment
Question by:itnifl
  • 8
  • 5
  • 2
15 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 40231788
i keep just "vmkernel" and "Hostd" programs, and leave rest with vcenter.
0
 
LVL 2

Author Comment

by:itnifl
ID: 40232513
What is the synthax in rsyslog configuration files for doing that?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40232586
No idea, i know syslog-ng
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 
LVL 2

Author Comment

by:itnifl
ID: 40232649
Umm.. I am in a blind end again, how would you do it in syslog-ng?
0
 
LVL 38

Accepted Solution

by:
Aaron Tomosky earned 500 total points
ID: 40254293
for rsyslog you put the filter right before the action (then a tab) like this:
*.warn;mail.none;authpriv.none;cron.none	action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

Open in new window

0
 
LVL 2

Author Comment

by:itnifl
ID: 40262409
Hello Aaron! Thank you for keeping this thread alive. My filter as shown in the question is like the one below. It doesn't work as intended. I will try yours out when I get back to work from vacation in a weeks time.

*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

//ans so on...

Open in new window

0
 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
ID: 40262419
check if it's .warn or .warning in the current logs going to ES
rsyslog error levels are inclusive so .warn includes anything higher like err, crit, etc...
0
 
LVL 2

Author Comment

by:itnifl
ID: 40294473
I will be taking a look at this right after the weekend
0
 
LVL 2

Author Comment

by:itnifl
ID: 40297891
As of before I do the following changes, it looks like I am mostly receiving info messages in Elasticsearch:
Info messages

I am not sure how else you want me to check if it's .warn or .warning in the current logs going to ES?

When I look into the /var/log/messages log file, I see the warning messages being referred to as 'warning'.
They look like this:
Sep  2 04:19:20 hostname.vmware.local Hostd: [5DF8BB90 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 577536 for scsi0:3 is out of range -- 577536,prevBytes = 174208512 curBytes = 176518656 prevCommands = 3875curCommands = 3879

As you can see from the picture above, these warnings are not showing in ES from what I can see.
But then I see there are not being created any more indexes, I guess because the root file system is full:

curl http://localhost:9200/_aliases?pre                                                                                                                                                             tty=1
{
  "vmware-2014.08.11" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },
  "vmware-2014.08.10" : {
    "aliases" : { }
  },
  "system" : {
    "aliases" : { }
  },
  "vmware-2014.08.03" : {
    "aliases" : { }
  },
  "vmware-2014.08.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.08" : {
    "aliases" : { }
  },
  "vmware-2014.08.09" : {
    "aliases" : { }
  },
  "vmware-2014.08.06" : {
    "aliases" : { }
  },
  "vmware-2014.08.07" : {
    "aliases" : { }
  },
  "vmware-2014.08.12" : {
    "aliases" : { }
  },
  "vmware-2014.08.01" : {
    "aliases" : { }
  },
  "vmware-2014.07.31" : {
    "aliases" : { }
  },
  "vmware-2014.08.04" : {
    "aliases" : { }
  },
  "vmware-2014.08.05" : {
    "aliases" : { }
  }
}

I have a script under /etc/cron.daily that should clean up old vmware indexes like this - I guess ut is not working even if it seemed to do so while testing:
#!/bin/bash
##
## Cleans up indexes for vmware logging in Elasticsearch
## atle@team-holm.net - 01.08.2014
##

DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%y`

DAYn=$(echo $DAY | sed 's/^0*//')
MONTHn=$(echo $MONTH | sed 's/^0*//')
YEARn=$(echo $YEAR | sed 's/^0*//')

if [ `expr $DAYn - 8` -lt 1 ] && [ $MONTHn -eq 1 ]; then
        YEARn=`expr YEARn - 1`
        YEAR="0$YEARn"
fi

if [ `expr $DAYn - 8` -lt 1 ]; then
        nMONTHn=`expr $MONTHn - 1`
        if [ $nMONTHn -lt 10 ]; then
                nMONTHn="0$nMONTHn"
        fi
        nDAYn1=`expr 31 - $DAYn - 8`
        nDAYn2=`expr 30 - $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi
        if [ $nDAYn2 -lt 10 ]; then
                nDAYn2="0$nDAYn2"
        fi
        echo -e "\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2"
        echo -e "\n"
else
        nDAYn1=`expr $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi

        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m*Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1"
        echo -e "\n"
fi

Open in new window

I will clean up the indexes and see where that goes.

In /etc/rsyslog.conf I changed from:
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

to:

*.warn;mail.none;authpriv.none;cron.none                /var/log/messages

I guess it should have no effect, but did it anyway.

In /etc/rsyslog.d/30-elasticsearch.conf I changed the line from:
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

to:

*.warn;mail.none;authpriv.none;cron.none

Now we just have to wait and see.
0
 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
ID: 40298881
go get this (I use git to "install" it):
git clone https://github.com/logstash/expire-logs.git
cd expire-logs
python logstash_index_cleaner.py --help
#if you see other missing stuff just pip install nameofmissing

I use it like this:
python ~/expire-logs/logstash_index_cleaner.py -d 7 #will delete anything older than 7 days
0
 
LVL 2

Author Comment

by:itnifl
ID: 40301040
I get this:

:/usr/src/expire-logs/expire-logs# ls
CHANGELOG  CONTRIBUTORS  curator  LICENSE.txt  MANIFEST.in  README.md  setup.cfg  setup.py  test_curator  VERSION
:/usr/src/expire-logs/expire-logs# python setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
0
 
LVL 38

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 500 total points
ID: 40301093
Ahh it's now been renamed curator as the 3rd iteration of its existence. So my command isn't accurate any more but it looks to be more powerful than before

https://github.com/elasticsearch/curator/blob/master/README.md
0
 
LVL 2

Author Comment

by:itnifl
ID: 40303388
Looks like a nice utility:

 #curator delete --older-than 7
2014-09-04 15:02:53,559 INFO      Job starting...
2014-09-04 15:02:53,566 INFO      Beginning DELETE operations...
2014-09-04 15:02:53,569 INFO      DELETE index operations completed.
2014-09-04 15:02:53,569 INFO      Done in 0:00:00.036160.

Open in new window


Still shows a bunch of indexes:

~# curl http://localhost:9200/_aliases?pretty=1
{
  "vmware-2014.09.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.23" : {
    "aliases" : { }
  },
  "vmware-2014.09.01" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },

Open in new window

0
 
LVL 2

Author Closing Comment

by:itnifl
ID: 40752042
I actually never got around to follow this up, and it seems that the solution that was built has been taken down for the use of another one.
I might have another try at this system some other day.
Giving points for all the good attempts and advices here.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
ID: 40752045
Check out sexilog, they put together a nice ready to run setup.
0

Featured Post

Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
linux(debian) mouse poor performance 4 39
Move ESX/VMs  from Vcenter 5.5 to Vcenter 6 6 53
Cant access ESXI 4 46
Do hyper-v and VMware clash 4 49
Last article we focus in how to VMware: How to create and use VMs TAGs – Part 1 so before follow this article and perform the next tasks, you should read the first article how to create the TAG before using them in Veeam Backup Jobs.
This article will show you how to create an ISO CD-ROM/DVD-ROM image (*.iso), and MD5 checksum signature, for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5). It's a good idea to compare checksums, because many installations fail because of a corr…
Teach the user how to edit .vmx files to add advanced configuration options Open vSphere Web Client: Edit Settings for a VM: Choose VM Options -> Advanced: Add Configuration Parameters:
Teach the user how to use configure the vCenter Server storage filters Open vSphere Web Client:  Navigate to vCenter Server Advanced Settings: Add the four vCenter Server storage filters: Review the advanced settings: Modify the values of the four v…

815 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now