Solved

Setting up rsyslog to send to Elasticsearch and Kibana

Posted on 2014-04-09
11
3,031 Views
Last Modified: 2014-07-31
I want to send logging from vmware hosts to an rsyslog server that forwards the whole thing to Elasticsearch and Kibana on the same server. I used this guide:

http://www.rsyslog.com/tag/elasticsearch/

I can see log entries arriving at /var/log/messages on the rsyslog server from the vmware hosts, but when accessing http://servername:8000 Kibana shows no log entries in any of the predefined Dashboards.

How do I verify and/or troubleshoot if anything is ending up at the Elasticsearch and Kibana server? Is there any more configuration I need doing before I see anything in Kibana?

Adding the rsyslog.conf file here for information. There is nothing sensitive in it.
rsyslog
0
Comment
Question by:itnifl
  • 6
  • 5
11 Comments
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
So top level view:
esxi -> rsyslog (working since they land in /var/log/messages)
rsyslog -> elasticsearch
kibana -> elasticsearch

I think your rsyslog.conf is missing some stuff from the action (what to send as well as some options). I put the fqdn of the vm for the server line even though rsyslog and elasticsearch are on the same box for me:

# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
*.info;mail.none;authpriv.none;cron.none      action(type="omelasticsearch"
      server="something.company.com"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")

Then you want to install elasticsearch-head so you can browse elasticsearch directly. Here are my notes for installing it on centos
//There is a front-end that I like called called elasticsearch-head
//which can be used to perform various maintenance tasks against the ElasticSearch server. To install (optional):
/etc/elasticsearch/bin/plugin -install mobz/elasticsearch-head
//Once installed, browse to http://{address_of_elasticsearch_node}:9200/_plugin/head/

Once you see stuff going into elasticsearch, then try in kibana as all that does is look directly at elasticsearch
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
Thank you alot!

I have now set up rsyslog.conf the way you suggested. For me the elasticsearch-head install was this way: /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head

I have browsed to  http://{address_of_elasticsearch_node}:9200/_plugin/head/ but I am not sure I am doing this correctly. There is no info under Browser, if I push the search button under Structured Query, it searches without end with no results. Under the Any Request tab I run a {"query":{"match_all":{}}} and the response I get is:

{
  took: 0
  timed_out: false
  _shards: {
  total: 0
  successful: 0
  failed: 0
}
hits: {
  total: 0
  max_score: 0
  hits: [ ]
}
}

Open in new window


Approximately the same thing under "Any Request."

# tail /var/log/messages
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'vpxavpxaInvtVm' opID=WFU-e5099ede] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:12:33 hostname.domain.do Vpxa: [FFCC1B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e5099ede] [WaitForUpdatesDone] Completed callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Received callback
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 171649 to 171650 (at 171649)
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'hostdvm' opID=WFU-e48c02cf] [VpxaHalVmHostagent] 12: GuestInfo changed 'guest.disk'
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'halservices' opID=WFU-e48c02cf] [VpxaHalServices] VmGuestDiskChange Event for vm(13) 12
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'vpxavpxaInvtVm' opID=WFU-e48c02cf] [VpxaInvtVmChangeListener] Guest DiskInfo Changed
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd
Apr 10 06:13:33 hostname2.domain.do Vpxa: [41234B90 verbose 'VpxaHalCnxHostagent' opID=WFU-e48c02cf] [WaitForUpdatesDone] Completed callback

I checked both the system clock with the date command and the clock on the VMWare hosts, both are correct, while the log entries that seem to arrive are 2 hours back in time.

Last message above says it was sent 06:13:33, but the real time was approx. 08:13.

Adding rsyslog.conf here again.

Port 9200 on tcp and udp are open in the firewall, same goes for port 8000tcp+udp, 512:tcp+udp, 1501:tcp, 1581-1583:tco, 48000-48020:tcp. Anything missing maybe?
rsyslog
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
I'll assume you changed the line "host.domain.local" to hide your fqdn right?

It looks like your elasticsearch is empty so lets start there.
Is it on the same box as your rsyslog receiver?
Do this from the rsyslog box (if it's different) to test you can manually put something in elasticsearch:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
FQDN is changed to the correct FQDN, yes.
Elasticsearch and Kibana are on the same box as the rsyslog server.

I tried  the following on the elasticsearch server:
# curl -XPUT 'http://myserver.domain.local:9200/twitter/tweet/1' -d '{
     "user" : "kimchy",
     "post_date" : "2009-11-15T14:12:12",
     "message" : "trying out Elasticsearch"
 }'
Response was:
{"_index":"twitter","_type":"tweet","_id":"1","_version":1,"created":true}

I then logged in on to the Elasticsearch Dashboard and then selected Sample Dashboard. I saw the tweet there, so I think Elasticsearch is set up correctly. It is the rsyslog that is not shipping things right.
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
ok, that ruleset is to send logs from THAT server to elasticsearch. You have to add a different ruleset for whatever your receiver is. For example I use relp (for rsyslog on other boxes to send to my main box) like this:

$InputRELPServerBindRuleset relp
$InputRELPServerRun 2514
#this is for sending stuff received from other rsyslog senders
ruleset(name="relp") {
      action(type="omelasticsearch"
      server="fqdn"
      serverport="9200"
      template="plain-syslog"
      searchIndex="logstash-index"
      bulkmode="on"
      queue.type="linkedlist"
      queue.size="5000"
      queue.dequeuebatchsize="300"
      queue.saveonshutdown="on"
      action.resumeretrycount="-1"
      dynSearchIndex="on")
}

I haven't done UDP or TCP ruleset binding and I'm not sure which one you're using so I'll just point you to the doc. The trick is to name the listener to a ruleset then reference that ruleset in the action
http://www.rsyslog.com/doc/master/configuration/modules/imudp.html
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 2

Author Comment

by:itnifl
Comment Utility
I am not sure I understand what you are referring to when you say "that" server, which one to you refer to? All products are on the same box, rsyslog, elasticsearch and kibana. Are we misunderstanding each other?
0
 
LVL 38

Expert Comment

by:Aaron Tomosky
Comment Utility
The *.info; line sends logs generated on localhost to ES.
So you need a rule set name for your input receiver (tcp,udp), and an omelasticsearch action for it.
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
It doesn't seem to be working.

I now have this rsyslog.conf:

module(load="imtcp")   # TCP input module
module(load="imudp") # needs to be done just once
module(load="omelasticsearch") # Elasticsearch output module

input(type="imtcp" port="514")  # where to listen for TCP messages
input(type="imudp" port="514")

main_queue(
  queue.size="1000000"   # capacity of the main queue
  queue.debatchsize="1000"  # process messages in batches of 1000 and move them to the action queues
  queue.workerthreads="2"  # 2 threads for the main queue
)

# template to generate JSON documents for Elasticsearch in Logstash format
template(name="plain-syslog"
         type="list") {
           constant(value="{")
             constant(value="\"@timestamp\":\"")      property(name="timereported" dateFormat="rfc3339")
             constant(value="\",\"host\":\"")        property(name="hostname")
             constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
             constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
             constant(value="\",\"syslogtag\":\"")   property(name="syslogtag" format="json")
             constant(value="\",\"message\":\"")    property(name="msg" format="json")
             constant(value="\"}")
         }

action(type="omelasticsearch"
       template="plain-syslog"  # use the template defined earlier
       searchIndex="test-index"
       bulkmode="on"                   # use the Bulk API
       queue.dequeuebatchsize="5000"   # ES bulk size
       queue.size="100000"   # capacity of the action queue
       queue.workerthreads="5"   # 5 workers for the action
       action.resumeretrycount="-1"  # retry indefinitely if ES is unreachable
)

It is very simple, nothing else then needed. It is from http://www.rsyslog.com/tag/elasticsearch/
It doesn't work. I understand Elasticsearch sends to localhost:9200 by default. I have set the vmware hosts to send on udp to the rsyslog server (that is also on the same box as kibana and elasticsearch as mentioned earlier). I tried to configure two of the hosts to send with tcp instead, and offcourse that doesn't seem to make much difference.
0
 
LVL 38

Accepted Solution

by:
Aaron Tomosky earned 500 total points
Comment Utility
Personally I use the bind to rule set like a posted above and then reference the rule set name in the action and it works. I didn't find any simple examples on the site that don't use the rule set method.
0
 
LVL 2

Author Comment

by:itnifl
Comment Utility
It will take a very long time before I can close this question. No matter what guides I use out there, the setup is not working. I have now ordered a book - Elasticsearch Server Second Edition. I will see what I can get from it and whatever I find on the internet. Meanwhile, this just has to wait.

This is also a non-prioritized task at my work. So I will have to do this in my free time to get this done, if i want to. Might go a bit slower then if i would be able to spend more work time on this.
0
 
LVL 2

Author Closing Comment

by:itnifl
Comment Utility
It seems that the omelasticsearch wasn't sending anything to elasticsearch from rsyslog. I set up a new server and compiled rsyslog manually, and in the process ran:
./configure --enable-elasticsearch
0

Featured Post

Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

Join & Write a Comment

Daily system administration tasks often require administrators to connect remote systems. But allowing these remote systems to accept passwords makes these systems vulnerable to the risk of brute-force password guessing attacks. Furthermore there ar…
SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now