Expiring Today—Celebrate National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Filtering rsyslog messages from vmware hosts

Posted on 2014-07-31
15
Medium Priority
?
446 Views
Last Modified: 2015-04-29
I receive log messages from vmware hosts on to my rsyslog server that forwards the whole thing to elasticsearch on the same server as the rsyslog server. This generates alot of data as everything is shipped over unconditionally. I want to filter the incoming messages so that only messages that are important (level warning and worse, not informational messages) are shipped to elasticsearch. The JSON formatted log messages elasticsearch is receiving look like this:

{"@timestamp":"2014-07-31T12:43:24.024Z","host":"vmware-host-fqdn","severity":"info","facility":"local4","tag":"Vpxa:","message":" [FFC9B6D0 verbose 'VpxaHalCnxHostagent' opID=WFU-169d8d35] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd"}

Here severity is info, this message is unwanted and simply floods my Elasticsearch server with nonsense.

I have the following configuration for sending from rsyslog to Elasticsearch in the file  /etc/rsyslog.d/30-elasticsearch.conf:

$ModLoad /usr/local/lib/rsyslog/omelasticsearch.so
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# this is for index names to be like: logstash-YYYY.MM.DD
template(name="vmware-index"
  type="list") {
    constant(value="vmware-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"tag\":\"")   property(name="syslogtag" format="json")
      constant(value="\",\"message\":\"")    property(name="msg" format="json")
    constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

You can see I have tried some filtering, but it does not seem to be working. How can I filter away those severity: info messages so that they never arrive at the Elasticsearch server?
0
Comment
Question by:itnifl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 8
  • 5
  • 2
15 Comments
 
LVL 62

Expert Comment

by:gheist
ID: 40231788
i keep just "vmkernel" and "Hostd" programs, and leave rest with vcenter.
0
 
LVL 2

Author Comment

by:itnifl
ID: 40232513
What is the synthax in rsyslog configuration files for doing that?
0
 
LVL 62

Expert Comment

by:gheist
ID: 40232586
No idea, i know syslog-ng
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
LVL 2

Author Comment

by:itnifl
ID: 40232649
Umm.. I am in a blind end again, how would you do it in syslog-ng?
0
 
LVL 39

Accepted Solution

by:
Aaron Tomosky earned 2000 total points
ID: 40254293
for rsyslog you put the filter right before the action (then a tab) like this:
*.warn;mail.none;authpriv.none;cron.none	action(type="omelasticsearch"
    template="plain-syslog"
    searchIndex="vmware-index"
    dynSearchIndex="on")

Open in new window

0
 
LVL 2

Author Comment

by:itnifl
ID: 40262409
Hello Aaron! Thank you for keeping this thread alive. My filter as shown in the question is like the one below. It doesn't work as intended. I will try yours out when I get back to work from vacation in a weeks time.

*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

//ans so on...

Open in new window

0
 
LVL 39

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 2000 total points
ID: 40262419
check if it's .warn or .warning in the current logs going to ES
rsyslog error levels are inclusive so .warn includes anything higher like err, crit, etc...
0
 
LVL 2

Author Comment

by:itnifl
ID: 40294473
I will be taking a look at this right after the weekend
0
 
LVL 2

Author Comment

by:itnifl
ID: 40297891
As of before I do the following changes, it looks like I am mostly receiving info messages in Elasticsearch:
Info messages

I am not sure how else you want me to check if it's .warn or .warning in the current logs going to ES?

When I look into the /var/log/messages log file, I see the warning messages being referred to as 'warning'.
They look like this:
Sep  2 04:19:20 hostname.vmware.local Hostd: [5DF8BB90 warning 'Statssvc.vim.PerformanceManager'] Calculated write I/O size 577536 for scsi0:3 is out of range -- 577536,prevBytes = 174208512 curBytes = 176518656 prevCommands = 3875curCommands = 3879

As you can see from the picture above, these warnings are not showing in ES from what I can see.
But then I see there are not being created any more indexes, I guess because the root file system is full:

curl http://localhost:9200/_aliases?pre                                                                                                                                                             tty=1
{
  "vmware-2014.08.11" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },
  "vmware-2014.08.10" : {
    "aliases" : { }
  },
  "system" : {
    "aliases" : { }
  },
  "vmware-2014.08.03" : {
    "aliases" : { }
  },
  "vmware-2014.08.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.08" : {
    "aliases" : { }
  },
  "vmware-2014.08.09" : {
    "aliases" : { }
  },
  "vmware-2014.08.06" : {
    "aliases" : { }
  },
  "vmware-2014.08.07" : {
    "aliases" : { }
  },
  "vmware-2014.08.12" : {
    "aliases" : { }
  },
  "vmware-2014.08.01" : {
    "aliases" : { }
  },
  "vmware-2014.07.31" : {
    "aliases" : { }
  },
  "vmware-2014.08.04" : {
    "aliases" : { }
  },
  "vmware-2014.08.05" : {
    "aliases" : { }
  }
}

I have a script under /etc/cron.daily that should clean up old vmware indexes like this - I guess ut is not working even if it seemed to do so while testing:
#!/bin/bash
##
## Cleans up indexes for vmware logging in Elasticsearch
## atle@team-holm.net - 01.08.2014
##

DAY=`date +%d`
MONTH=`date +%m`
YEAR=`date +%y`

DAYn=$(echo $DAY | sed 's/^0*//')
MONTHn=$(echo $MONTH | sed 's/^0*//')
YEARn=$(echo $YEAR | sed 's/^0*//')

if [ `expr $DAYn - 8` -lt 1 ] && [ $MONTHn -eq 1 ]; then
        YEARn=`expr YEARn - 1`
        YEAR="0$YEARn"
fi

if [ `expr $DAYn - 8` -lt 1 ]; then
        nMONTHn=`expr $MONTHn - 1`
        if [ $nMONTHn -lt 10 ]; then
                nMONTHn="0$nMONTHn"
        fi
        nDAYn1=`expr 31 - $DAYn - 8`
        nDAYn2=`expr 30 - $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi
        if [ $nDAYn2 -lt 10 ]; then
                nDAYn2="0$nDAYn2"
        fi
        echo -e "\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2/_query?q=facility:local4"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn1"
        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2'\n\e[0m"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$nMONTHn.$nDAYn2"
        echo -e "\n"
else
        nDAYn1=`expr $DAYn - 8`
        if [ $nDAYn1 -lt 10 ]; then
                nDAYn1="0$nDAYn1"
        fi

        echo -e "\n\n\e[33m *Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1/_query?q=facility:local4"
        echo -e "\n\n\e[33m*Executing: curl -XDELETE 'http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1'\e[0m\n"
        curl -XDELETE "http://localhost:9200/vmware-20$YEAR.$MONTH.$nDAYn1"
        echo -e "\n"
fi

Open in new window

I will clean up the indexes and see where that goes.

In /etc/rsyslog.conf I changed from:
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

to:

*.warn;mail.none;authpriv.none;cron.none                /var/log/messages

I guess it should have no effect, but did it anyway.

In /etc/rsyslog.d/30-elasticsearch.conf I changed the line from:
*.warning;*.error;*.crit;*.alert;*.emerg        action(type="omelasticsearch" server="fqdn")

to:

*.warn;mail.none;authpriv.none;cron.none

Now we just have to wait and see.
0
 
LVL 39

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 2000 total points
ID: 40298881
go get this (I use git to "install" it):
git clone https://github.com/logstash/expire-logs.git
cd expire-logs
python logstash_index_cleaner.py --help
#if you see other missing stuff just pip install nameofmissing

I use it like this:
python ~/expire-logs/logstash_index_cleaner.py -d 7 #will delete anything older than 7 days
0
 
LVL 2

Author Comment

by:itnifl
ID: 40301040
I get this:

:/usr/src/expire-logs/expire-logs# ls
CHANGELOG  CONTRIBUTORS  curator  LICENSE.txt  MANIFEST.in  README.md  setup.cfg  setup.py  test_curator  VERSION
:/usr/src/expire-logs/expire-logs# python setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied
0
 
LVL 39

Assisted Solution

by:Aaron Tomosky
Aaron Tomosky earned 2000 total points
ID: 40301093
Ahh it's now been renamed curator as the 3rd iteration of its existence. So my command isn't accurate any more but it looks to be more powerful than before

https://github.com/elasticsearch/curator/blob/master/README.md
0
 
LVL 2

Author Comment

by:itnifl
ID: 40303388
Looks like a nice utility:

 #curator delete --older-than 7
2014-09-04 15:02:53,559 INFO      Job starting...
2014-09-04 15:02:53,566 INFO      Beginning DELETE operations...
2014-09-04 15:02:53,569 INFO      DELETE index operations completed.
2014-09-04 15:02:53,569 INFO      Done in 0:00:00.036160.

Open in new window


Still shows a bunch of indexes:

~# curl http://localhost:9200/_aliases?pretty=1
{
  "vmware-2014.09.02" : {
    "aliases" : { }
  },
  "vmware-2014.08.23" : {
    "aliases" : { }
  },
  "vmware-2014.09.01" : {
    "aliases" : { }
  },
  "vmware-2014.08.13" : {
    "aliases" : { }
  },

Open in new window

0
 
LVL 2

Author Closing Comment

by:itnifl
ID: 40752042
I actually never got around to follow this up, and it seems that the solution that was built has been taken down for the use of another one.
I might have another try at this system some other day.
Giving points for all the good attempts and advices here.
0
 
LVL 39

Expert Comment

by:Aaron Tomosky
ID: 40752045
Check out sexilog, they put together a nice ready to run setup.
0

Featured Post

Three Reasons Why Backup is Strategic

Backup is strategic to your business because your data is strategic to your business. Without backup, your business will fail. This white paper explains why it is vital for you to design and immediately execute a backup strategy to protect 100 percent of your data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this article, I will show you HOW TO: Perform a Physical to Virtual (P2V) Conversion the easy way from a computer backup (image).
When rebooting a vCenters 6.0 and try to connect using vSphere Client we get this issue "Invalid URL: The hostname could not parsed." When we get this error we need to do some changes in the vCenter advanced settings to fix the issue.
Teach the user how to configure vSphere clusters to support the VMware FT feature Open vSphere Web Client: Verify vSphere HA is enabled: Verify netowrking for vMotion and FT Logging is in place or create it: Turn On FT for a virtual machine: Verify …
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.

718 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question