• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 4092
  • Last Modified:

Setting up rsyslog to send to Elasticsearch and Kibana

I want to send logging from vmware hosts to an rsyslog server that forwards the whole thing to Elasticsearch and Kibana on the same server. I used this guide:

http://www.rsyslog.com/tag/elasticsearch/

I can see log entries arriving at /var/log/messages on the rsyslog server from the vmware hosts, but when accessing http://servername:8000 Kibana shows no log entries in any of the predefined Dashboards.

How do I verify and/or troubleshoot if anything is ending up at the Elasticsearch and Kibana server? Is there any more configuration I need doing before I see anything in Kibana?

Adding the rsyslog.conf file here for information. There is nothing sensitive in it.
rsyslog
0
itnifl
Asked:
itnifl
  • 6
  • 5
1 Solution
 
Aaron TomoskyTechnology ConsultantCommented:
So top level view:
esxi -> rsyslog (working since they land in /var/log/messages)
rsyslog -> elasticsearch
kibana -> elasticsearch

I think your rsyslog.conf is missing some stuff from the action (what to send as well as some options). I put the fqdn of the vm for the server line even though rsyslog and elasticsearch are on the same box for me:

# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
*.info;mail.none;authpriv.none;cron.none      action(type="omelasticsearch"
      server="something.company.com"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")

Then you want to install elasticsearch-head so you can browse elasticsearch directly. Here are my notes for installing it on centos
//There is a front-end that I like called called elasticsearch-head
//which can be used to perform various maintenance tasks against the ElasticSearch server. To install (optional):
/etc/elasticsearch/bin/plugin -install mobz/elasticsearch-head
//Once installed, browse to http://{address_of_elasticsearch_node}:9200/_plugin/head/

Once you see stuff going into elasticsearch, then try in kibana as all that does is look directly at elasticsearch
0
 
itniflAuthor Commented:
Thank you alot!

I have now set up rsyslog.conf the way you suggested. For me the elasticsearch-head install was this way: /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head

I have browsed to  http://{address_of_elasticsearch_node}:9200/_plugin/head/ but I am not sure I am doing this correctly. There is no info under Browser, if I push the search button under Structured Query, it searches without end with no results. Under the Any Request tab I run a {"query":{"match_all":{}}} and the response I get is:

{
  took: 0
  timed_out: false
  _shards: {
  total: 0
  successful: 0
  failed: 0
}
hits: {
  total: 0
  max_score: 0
  hits: [ ]
}
}

Open in new window


Approximately the same thing under "Any Request."

# tail /var/log/messages
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'vpxavpxaInvtVm' opID=WFU-e5099ede] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Completed callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Received callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 171649 to 171650 (at 171649)
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'hostdvm' opID=WFU-e48c02cf] [VpxaHalVmHostagent] 12: GuestInfo changed 'guest.disk'
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'halservices' opID=WFU-e48c02cf] [VpxaHalServices] VmGuestDiskChange Event for vm(13) 12
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'vpxavpxaInvtVm' opID=WFU-e48c02cf] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Completed callback

I checked both the system clock with the date command and the clock on the VMWare hosts, both are correct, while the log entries that seem to arrive are 2 hours back in time.

Last message above says it was sent 06:13:33, but the real time was approx. 08:13.

Adding rsyslog.conf here again.

Port 9200 on tcp and udp are open in the firewall, same goes for port 8000tcp+udp, 512:tcp+udp, 1501:tcp, 1581-1583:tco, 48000-48020:tcp. Anything missing maybe?
rsyslog
0
 
Aaron TomoskyTechnology ConsultantCommented:
I'll assume you changed the line "host.domain.local" to hide your fqdn right?

It looks like your elasticsearch is empty so lets start there.
Is it on the same box as your rsyslog receiver?
Do this from the rsyslog box (if it's different) to test you can manually put something in elasticsearch:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
itniflAuthor Commented:
FQDN is changed to the correct FQDN, yes.
Elasticsearch and Kibana are on the same box as the rsyslog server.

I tried  the following on the elasticsearch server:
# curl -XPUT 'http://myserver.domain.local:9200/twitter/tweet/1' -d '{
     "user" : "kimchy",
     "post_date" : "2009-11-15T14:12:12",
     "message" : "trying out Elasticsearch"
 }'
Response was:
{"_index":"twitter","_type":"tweet","_id":"1","_version":1,"created":true}

I then logged in on to the Elasticsearch Dashboard and then selected Sample Dashboard. I saw the tweet there, so I think Elasticsearch is set up correctly. It is the rsyslog that is not shipping things right.
0
 
Aaron TomoskyTechnology ConsultantCommented:
ok, that ruleset is to send logs from THAT server to elasticsearch. You have to add a different ruleset for whatever your receiver is. For example I use relp (for rsyslog on other boxes to send to my main box) like this:

$InputRELPServerBindRuleset relp
$InputRELPServerRun 2514
#this is for sending stuff received from other rsyslog senders
ruleset(name="relp") {
      action(type="omelasticsearch"
      server="fqdn"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")
}

I haven't done UDP or TCP ruleset binding and I'm not sure which one you're using so I'll just point you to the doc. The trick is to name the listener to a ruleset then reference that ruleset in the action
http://www.rsyslog.com/doc/master/configuration/modules/imudp.html
0
 
itniflAuthor Commented:
I am not sure I understand what you are referring to when you say "that" server, which one to you refer to? All products are on the same box, rsyslog, elasticsearch and kibana. Are we misunderstanding each other?
0
 
Aaron TomoskyTechnology ConsultantCommented:
The *.info; line sends logs generated on localhost to ES.
So you need a rule set name for your input receiver (tcp,udp), and an omelasticsearch action for it.
0
 
itniflAuthor Commented:
It doesn't seem to be working.

I now have this rsyslog.conf:

module(load="imtcp")   # TCP input module
module(load="imudp") # needs to be done just once
module(load="omelasticsearch") # Elasticsearch output module

input(type="imtcp" port="514")  # where to listen for TCP messages
input(type="imudp" port="514")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# template to generate JSON documents for Elasticsearch in Logstash format
template(name="plain-syslog"
         type="list") {
           constant(value="{")
             constant(value="\"@timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
             constant(value="\",\"host\":\"")        property(name="hostname")
             constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
             constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
             constant(value="\",\"syslogtag\":\"")   property(name="syslogtag" format="json")
             constant(value="\",\"message\":\"")    property(name="msg" format="json")
             constant(value="\"}")
         }

action(type="omelasticsearch"
       template="plain-syslog"  # use the template defined earlier
       searchIndex="test-index"
       bulkmode="on"                   # use the Bulk API
       queue.dequeuebatchsize="5000"   # ES bulk size
       queue.size="100000"   # capacity of the action queue
       queue.workerthreads="5"   # 5 workers for the action
       action.resumeretrycount="-1"  # retry indefinitely if ES is unreachable
)

It is very simple, nothing else then needed. It is from http://www.rsyslog.com/tag/elasticsearch/
It doesn't work. I understand Elasticsearch sends to localhost:9200 by default. I have set the vmware hosts to send on udp to the rsyslog server (that is also on the same box as kibana and elasticsearch as mentioned earlier). I tried to configure two of the hosts to send with tcp instead, and offcourse that doesn't seem to make much difference.
0
 
Aaron TomoskyTechnology ConsultantCommented:
Personally I use the bind to rule set like a posted above and then reference the rule set name in the action and it works. I didn't find any simple examples on the site that don't use the rule set method.
0
 
itniflAuthor Commented:
It will take a very long time before I can close this question. No matter what guides I use out there, the setup is not working. I have now ordered a book - Elasticsearch Server Second Edition. I will see what I can get from it and whatever I find on the internet. Meanwhile, this just has to wait.

This is also a non-prioritized task at my work. So I will have to do this in my free time to get this done, if i want to. Might go a bit slower then if i would be able to spend more work time on this.
0
 
itniflAuthor Commented:
It seems that the omelasticsearch wasn't sending anything to elasticsearch from rsyslog. I set up a new server and compiled rsyslog manually, and in the process ran:
./configure --enable-elasticsearch
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 6
  • 5
Tackle projects and never again get stuck behind a technical roadblock.
Join Now