Setting up rsyslog to send to Elasticsearch and Kibana

So top level view:
esxi -> rsyslog (working since they land in /var/log/messages)
rsyslog -> elasticsearch
kibana -> elasticsearch

I think your rsyslog.conf is missing some stuff from the action (what to send as well as some options). I put the fqdn of the vm for the server line even though rsyslog and elasticsearch are on the same box for me:

# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
*.info;mail.none;authpriv.none;cron.none      action(type="omelasticsearch"
      server="something.company.com"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")

Then you want to install elasticsearch-head so you can browse elasticsearch directly. Here are my notes for installing it on centos
//There is a front-end that I like called called elasticsearch-head
//which can be used to perform various maintenance tasks against the ElasticSearch server. To install (optional):
/etc/elasticsearch/bin/plugin -install mobz/elasticsearch-head
//Once installed, browse to http://{address_of_elasticsearch_node}:9200/_plugin/head/

Once you see stuff going into elasticsearch, then try in kibana as all that does is look directly at elasticsearch

Thank you alot!

I have now set up rsyslog.conf the way you suggested. For me the elasticsearch-head install was this way: /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head

I have browsed to  http://{address_of_elasticsearch_node}:9200/_plugin/head/ but I am not sure I am doing this correctly. There is no info under Browser, if I push the search button under Structured Query, it searches without end with no results. Under the Any Request tab I run a {"query":{"match_all":{}}} and the response I get is:

{
  took: 0
  timed_out: false
  _shards: {
  total: 0
  successful: 0
  failed: 0
}
hits: {
  total: 0
  max_score: 0
  hits: [ ]
}
}

Select allOpen in new window

Approximately the same thing under "Any Request."

# tail /var/log/messages
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'vpxavpxaInvtVm' opID=WFU-e5099ede] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Completed callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Received callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 171649 to 171650 (at 171649)
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'hostdvm' opID=WFU-e48c02cf] [VpxaHalVmHostagent] 12: GuestInfo changed 'guest.disk'
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'halservices' opID=WFU-e48c02cf] [VpxaHalServices] VmGuestDiskChange Event for vm(13) 12
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'vpxavpxaInvtVm' opID=WFU-e48c02cf] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Completed callback

I checked both the system clock with the date command and the clock on the VMWare hosts, both are correct, while the log entries that seem to arrive are 2 hours back in time.

Last message above says it was sent 06:13:33, but the real time was approx. 08:13.

Adding rsyslog.conf here again.

Port 9200 on tcp and udp are open in the firewall, same goes for port 8000tcp+udp, 512:tcp+udp, 1501:tcp, 1581-1583:tco, 48000-48020:tcp. Anything missing maybe?
rsyslog

I'll assume you changed the line "host.domain.local" to hide your fqdn right?

It looks like your elasticsearch is empty so lets start there.
Is it on the same box as your rsyslog receiver?
Do this from the rsyslog box (if it's different) to test you can manually put something in elasticsearch:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html

FQDN is changed to the correct FQDN, yes.
Elasticsearch and Kibana are on the same box as the rsyslog server.

I tried  the following on the elasticsearch server:
# curl -XPUT 'http://myserver.domain.local:9200/twitter/tweet/1' -d '{
     "user" : "kimchy",
     "post_date" : "2009-11-15T14:12:12",
     "message" : "trying out Elasticsearch"
 }'
Response was:
{"_index":"twitter","_type":"tweet","_id":"1","_version":1,"created":true}

I then logged in on to the Elasticsearch Dashboard and then selected Sample Dashboard. I saw the tweet there, so I think Elasticsearch is set up correctly. It is the rsyslog that is not shipping things right.

ok, that ruleset is to send logs from THAT server to elasticsearch. You have to add a different ruleset for whatever your receiver is. For example I use relp (for rsyslog on other boxes to send to my main box) like this:

$InputRELPServerBindRuleset relp
$InputRELPServerRun 2514
#this is for sending stuff received from other rsyslog senders
ruleset(name="relp") {
      action(type="omelasticsearch"
      server="fqdn"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")
}

I haven't done UDP or TCP ruleset binding and I'm not sure which one you're using so I'll just point you to the doc. The trick is to name the listener to a ruleset then reference that ruleset in the action
http://www.rsyslog.com/doc/master/configuration/modules/imudp.html

I am not sure I understand what you are referring to when you say "that" server, which one to you refer to? All products are on the same box, rsyslog, elasticsearch and kibana. Are we misunderstanding each other?

The *.info; line sends logs generated on localhost to ES.
So you need a rule set name for your input receiver (tcp,udp), and an omelasticsearch action for it.

It doesn't seem to be working.

I now have this rsyslog.conf:

module(load="imtcp")   # TCP input module
module(load="imudp") # needs to be done just once
module(load="omelasticsearch") # Elasticsearch output module

input(type="imtcp" port="514")  # where to listen for TCP messages
input(type="imudp" port="514")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# template to generate JSON documents for Elasticsearch in Logstash format
template(name="plain-syslog"
         type="list") {
           constant(value="{")
             constant(value="\"@timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
             constant(value="\",\"host\":\"")        property(name="hostname")
             constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
             constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
             constant(value="\",\"syslogtag\":\"")   property(name="syslogtag" format="json")
             constant(value="\",\"message\":\"")    property(name="msg" format="json")
             constant(value="\"}")
         }

action(type="omelasticsearch"
       template="plain-syslog"  # use the template defined earlier
       searchIndex="test-index"
       bulkmode="on"                   # use the Bulk API
       queue.dequeuebatchsize="5000"   # ES bulk size
       queue.size="100000"   # capacity of the action queue
       queue.workerthreads="5"   # 5 workers for the action
       action.resumeretrycount="-1"  # retry indefinitely if ES is unreachable
)

It is very simple, nothing else then needed. It is from http://www.rsyslog.com/tag/elasticsearch/
It doesn't work. I understand Elasticsearch sends to localhost:9200 by default. I have set the vmware hosts to send on udp to the rsyslog server (that is also on the same box as kibana and elasticsearch as mentioned earlier). I tried to configure two of the hosts to send with tcp instead, and offcourse that doesn't seem to make much difference.

It will take a very long time before I can close this question. No matter what guides I use out there, the setup is not working. I have now ordered a book - Elasticsearch Server Second Edition. I will see what I can get from it and whatever I find on the internet. Meanwhile, this just has to wait.

This is also a non-prioritized task at my work. So I will have to do this in my free time to get this done, if i want to. Might go a bit slower then if i would be able to spend more work time on this.

It seems that the omelasticsearch wasn't sending anything to elasticsearch from rsyslog. I set up a new server and compiled rsyslog manually, and in the process ran:
./configure --enable-elasticsearch