Solved

Setting up rsyslog to send to Elasticsearch and Kibana

Posted on 2014-04-09
11
3,137 Views
Last Modified: 2014-07-31
I want to send logging from vmware hosts to an rsyslog server that forwards the whole thing to Elasticsearch and Kibana on the same server. I used this guide:

http://www.rsyslog.com/tag/elasticsearch/

I can see log entries arriving at /var/log/messages on the rsyslog server from the vmware hosts, but when accessing http://servername:8000 Kibana shows no log entries in any of the predefined Dashboards.

How do I verify and/or troubleshoot if anything is ending up at the Elasticsearch and Kibana server? Is there any more configuration I need doing before I see anything in Kibana?

Adding the rsyslog.conf file here for information. There is nothing sensitive in it.
rsyslog
0
Comment
Question by:itnifl
  • 6
  • 5
11 Comments
 
LVL 38

Expert Comment

by:Aaron Tomosky
ID: 39989316
So top level view:
esxi -> rsyslog (working since they land in /var/log/messages)
rsyslog -> elasticsearch
kibana -> elasticsearch

I think your rsyslog.conf is missing some stuff from the action (what to send as well as some options). I put the fqdn of the vm for the server line even though rsyslog and elasticsearch are on the same box for me:

# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
*.info;mail.none;authpriv.none;cron.none      action(type="omelasticsearch"
      server="something.company.com"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")

Then you want to install elasticsearch-head so you can browse elasticsearch directly. Here are my notes for installing it on centos
//There is a front-end that I like called called elasticsearch-head
//which can be used to perform various maintenance tasks against the ElasticSearch server. To install (optional):
/etc/elasticsearch/bin/plugin -install mobz/elasticsearch-head
//Once installed, browse to http://{address_of_elasticsearch_node}:9200/_plugin/head/

Once you see stuff going into elasticsearch, then try in kibana as all that does is look directly at elasticsearch
0
 
LVL 2

Author Comment

by:itnifl
ID: 39990825
Thank you alot!

I have now set up rsyslog.conf the way you suggested. For me the elasticsearch-head install was this way: /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head

I have browsed to  http://{address_of_elasticsearch_node}:9200/_plugin/head/ but I am not sure I am doing this correctly. There is no info under Browser, if I push the search button under Structured Query, it searches without end with no results. Under the Any Request tab I run a {"query":{"match_all":{}}} and the response I get is:

{
  took: 0
  timed_out: false
  _shards: {
  total: 0
  successful: 0
  failed: 0
}
hits: {
  total: 0
  max_score: 0
  hits: [ ]
}
}

Open in new window


Approximately the same thing under "Any Request."

# tail /var/log/messages
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'vpxavpxaInvtVm' opID=WFU-e5099ede] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Completed callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Received callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 171649 to 171650 (at 171649)
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'hostdvm' opID=WFU-e48c02cf] [VpxaHalVmHostagent] 12: GuestInfo changed 'guest.disk'
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'halservices' opID=WFU-e48c02cf] [VpxaHalServices] VmGuestDiskChange Event for vm(13) 12
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'vpxavpxaInvtVm' opID=WFU-e48c02cf] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Completed callback

I checked both the system clock with the date command and the clock on the VMWare hosts, both are correct, while the log entries that seem to arrive are 2 hours back in time.

Last message above says it was sent 06:13:33, but the real time was approx. 08:13.

Adding rsyslog.conf here again.

Port 9200 on tcp and udp are open in the firewall, same goes for port 8000tcp+udp, 512:tcp+udp, 1501:tcp, 1581-1583:tco, 48000-48020:tcp. Anything missing maybe?
rsyslog
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
ID: 39992210
I'll assume you changed the line "host.domain.local" to hide your fqdn right?

It looks like your elasticsearch is empty so lets start there.
Is it on the same box as your rsyslog receiver?
Do this from the rsyslog box (if it's different) to test you can manually put something in elasticsearch:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html
0
 
LVL 2

Author Comment

by:itnifl
ID: 39993968
FQDN is changed to the correct FQDN, yes.
Elasticsearch and Kibana are on the same box as the rsyslog server.

I tried  the following on the elasticsearch server:
# curl -XPUT 'http://myserver.domain.local:9200/twitter/tweet/1' -d '{
     "user" : "kimchy",
     "post_date" : "2009-11-15T14:12:12",
     "message" : "trying out Elasticsearch"
 }'
Response was:
{"_index":"twitter","_type":"tweet","_id":"1","_version":1,"created":true}

I then logged in on to the Elasticsearch Dashboard and then selected Sample Dashboard. I saw the tweet there, so I think Elasticsearch is set up correctly. It is the rsyslog that is not shipping things right.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
ID: 39995376
ok, that ruleset is to send logs from THAT server to elasticsearch. You have to add a different ruleset for whatever your receiver is. For example I use relp (for rsyslog on other boxes to send to my main box) like this:

$InputRELPServerBindRuleset relp
$InputRELPServerRun 2514
#this is for sending stuff received from other rsyslog senders
ruleset(name="relp") {
      action(type="omelasticsearch"
      server="fqdn"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")
}

I haven't done UDP or TCP ruleset binding and I'm not sure which one you're using so I'll just point you to the doc. The trick is to name the listener to a ruleset then reference that ruleset in the action
http://www.rsyslog.com/doc/master/configuration/modules/imudp.html
0
New! My Passport Wireless Pro Wi-Fi Mobile Storage

Portable wireless storage to offload, edit, and stream anywhere.

High-capacity, wireless mobile storage designed to accompany professional photographers and videographers in the field to easily offload, edit and stream captured photos and high-definition videos.

 
LVL 2

Author Comment

by:itnifl
ID: 39998396
I am not sure I understand what you are referring to when you say "that" server, which one to you refer to? All products are on the same box, rsyslog, elasticsearch and kibana. Are we misunderstanding each other?
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
ID: 39999367
The *.info; line sends logs generated on localhost to ES.
So you need a rule set name for your input receiver (tcp,udp), and an omelasticsearch action for it.
0
 
LVL 2

Author Comment

by:itnifl
ID: 40029660
It doesn't seem to be working.

I now have this rsyslog.conf:

module(load="imtcp")   # TCP input module
module(load="imudp") # needs to be done just once
module(load="omelasticsearch") # Elasticsearch output module

input(type="imtcp" port="514")  # where to listen for TCP messages
input(type="imudp" port="514")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# template to generate JSON documents for Elasticsearch in Logstash format
template(name="plain-syslog"
         type="list") {
           constant(value="{")
             constant(value="\"@timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
             constant(value="\",\"host\":\"")        property(name="hostname")
             constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
             constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
             constant(value="\",\"syslogtag\":\"")   property(name="syslogtag" format="json")
             constant(value="\",\"message\":\"")    property(name="msg" format="json")
             constant(value="\"}")
         }

action(type="omelasticsearch"
       template="plain-syslog"  # use the template defined earlier
       searchIndex="test-index"
       bulkmode="on"                   # use the Bulk API
       queue.dequeuebatchsize="5000"   # ES bulk size
       queue.size="100000"   # capacity of the action queue
       queue.workerthreads="5"   # 5 workers for the action
       action.resumeretrycount="-1"  # retry indefinitely if ES is unreachable
)

It is very simple, nothing else then needed. It is from http://www.rsyslog.com/tag/elasticsearch/
It doesn't work. I understand Elasticsearch sends to localhost:9200 by default. I have set the vmware hosts to send on udp to the rsyslog server (that is also on the same box as kibana and elasticsearch as mentioned earlier). I tried to configure two of the hosts to send with tcp instead, and offcourse that doesn't seem to make much difference.
0
 
LVL 38

Accepted Solution

by:
Aaron Tomosky earned 500 total points
ID: 40029912
Personally I use the bind to rule set like a posted above and then reference the rule set name in the action and it works. I didn't find any simple examples on the site that don't use the rule set method.
0
 
LVL 2

Author Comment

by:itnifl
ID: 40087937
It will take a very long time before I can close this question. No matter what guides I use out there, the setup is not working. I have now ordered a book - Elasticsearch Server Second Edition. I will see what I can get from it and whatever I find on the internet. Meanwhile, this just has to wait.

This is also a non-prioritized task at my work. So I will have to do this in my free time to get this done, if i want to. Might go a bit slower then if i would be able to spend more work time on this.
0
 
LVL 2

Author Closing Comment

by:itnifl
ID: 40231538
It seems that the omelasticsearch wasn't sending anything to elasticsearch from rsyslog. I set up a new server and compiled rsyslog manually, and in the process ran:
./configure --enable-elasticsearch
0

Featured Post

Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Network Interface Card (NIC) bonding, also known as link aggregation, NIC teaming and trunking, is an important concept to understand and implement in any environment where high availability is of concern. Using this feature, a server administrator …
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Get a first impression of how PRTG looks and learn how it works.   This video is a short introduction to PRTG, as an initial overview or as a quick start for new PRTG users.

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now