jdenver247
asked on
Host Entry Error (Cacti Syslog Plugin)
I am using Syslog-ng Server and forwarding all the syslog messages to this server. I have around 98 hosts reporting. Here are syslog messages for some of them
-------------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- --------
Nov 13 12:49:39 rr9513-3.xyzcompany.com syslog:crit : 2008 Nov 13 13:09:04 EXT: %LICMGR-2-LOG_LIC_GRACE_EX PIRED: Grace period expired for feature FM_SERVER_PKG.
Nov 13 12:59:36 rr9513-2.xyzcompany.com syslog:crit : 2008 Nov 13 12:59:36 EST: %LICMGR-2-LOG_LIC_GRA .
Nov 13 13:19:01 rr9513-1.xyzcompany.com syslog:crit : 2008 Nov 13 13:19:01 EST: %LICMGR-2-LOG_LIC_GRA
-------------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- --------
Now in the above case rr9513-1.xyzcompany.com is the hostname but when I forward the messages to the syslog-ng server. Here is how it looks like....
-------------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------
Host Date Time Message Facility Level Options
2008 2008-11-13 13:36:11 Nov 13 13:29:34 EST: %LICMGR-2-LOG_LIC_GRACE_EX PIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit
2008 2008-11-13 13:34:59 Nov 13 13:31:05 EST: %LICMGR-2-LOG_LIC_GRACE_EX PIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit
-------------------------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- --------
It shows 2008 instead of rr9513-1.xyzcompany.com . Attached below is the syslog-conf script also I want to mention that this is happening only for this particular host
--------------------------
Nov 13 12:49:39 rr9513-3.xyzcompany.com syslog:crit : 2008 Nov 13 13:09:04 EXT: %LICMGR-2-LOG_LIC_GRACE_EX
Nov 13 12:59:36 rr9513-2.xyzcompany.com syslog:crit : 2008 Nov 13 12:59:36 EST: %LICMGR-2-LOG_LIC_GRA .
Nov 13 13:19:01 rr9513-1.xyzcompany.com syslog:crit : 2008 Nov 13 13:19:01 EST: %LICMGR-2-LOG_LIC_GRA
--------------------------
Now in the above case rr9513-1.xyzcompany.com is the hostname but when I forward the messages to the syslog-ng server. Here is how it looks like....
--------------------------
Host Date Time Message Facility Level Options
2008 2008-11-13 13:36:11 Nov 13 13:29:34 EST: %LICMGR-2-LOG_LIC_GRACE_EX
2008 2008-11-13 13:34:59 Nov 13 13:31:05 EST: %LICMGR-2-LOG_LIC_GRACE_EX
--------------------------
It shows 2008 instead of rr9513-1.xyzcompany.com . Attached below is the syslog-conf script also I want to mention that this is happening only for this particular host
#----------------------------------------------------------------------
# Program: syslog-ng.conf
# Notes: Embedded most of the manual notes within the configuration
# file. The original manual can be found at:
#
#----------------------------------------------------------------------
# 05-Nov-08 - REP - Added some extra definitions to the file.
# 12-Nov-08 - REP - Added back the comments on filtering.
# 13-Nov-08 - REP - Further modified for local environment,Initial creation for testing.
#----------------------------------------------------------------------
# Options
#----------------------------------------------------------------------
#
# Name Values Description
# ------------------------- ------- ------------------------------------
# bad_hostname reg exp A regexp which matches hostnames
# which should not be taken as such.
# chain_hostnames y/n Enable or disable the chained
# hostname format.
# create_dirs y/n Enable or disable directory creation
# for destination files.
# dir_group groupid
# dir_owner userid
# dir_perm perm
# dns_cache y/n Enable or disable DNS cache usage.
# dns_cache_expire num Number of seconds while a successful
# lookup is cached.
# dns_cache_expire_failed num Number of seconds while a failed
# lookup is cached.
# dns_cache_size num Number of hostnames in the DNS cache.
# gc_busy_threshold num Sets the threshold value for the
# garbage collector, when syslog-ng is
# busy. GC phase starts when the number
# of allocated objects reach this
# number. Default: 3000.
# gc_idle_threshold num Sets the threshold value for the
# garbage collector, when syslog-ng is
# idle. GC phase starts when the number
# of allocated objects reach this
# number. Default: 100.
# group groupid
# keep_hostname y/n Enable or disable hostname rewriting.
# This means that if the log entry had
# been passed through at least one other
# logging system, the ORIGINAL hostname
# will be kept attached to the log.
# Otherwise the last logger will be
# considered the log entry owner and
# the log entry will appear to have
# come from that host.
# log_fifo_size num The number of lines fitting to the
# output queue
# log_msg_size num Maximum length of message in bytes.
# long_hostnames on/off This options appears to only really
# have an affect on the local system.
# which removes the source of the log.
# As an example, normally the local
# logs will state src@hostname, but
# with this feature off, the source
# is not reported.
# mark num The number of seconds between two
# MARK lines. NOTE: not implemented
# yet.
# owner userid
# perm perm
# stats num The number of seconds between two
# STATS.
# sync num The number of lines buffered before
# written to file
# time_reap num The time to wait before an idle
# destination file is closed.
# time_reopen num The time to wait before a died
# connection is reestablished
# use_dns y/n Enable or disable DNS usage.
# syslog-ng blocks on DNS queries,
# so enabling DNS may lead to a
# Denial of Service attack. To
# prevent DoS, protect your
# syslog-ng network endpoint with
# firewall rules, and make sure that
# all hosts, which may get to
# syslog-ng is resolvable.
# use_fqdn y/n Add Fully Qualified Domain Name
# instead of short hostname.
# use_time_recvd y/n Use the time a message is
# received instead of the one
# specified in the message.
#----------------------------------------------------------------------
# 12-Nov-08 - REP - Some of the Configuration option are not fuctioning
#
# 12-Mar-03 - REP - I have increased the log_msg_size and log_fifo_size
# to increase the amount of buffering that we do.
# While for most systems this may not have a noticeable
# affect, it will for systems that are at the end of a
# lot of logging systems.
#----------------------------------------------------------------------
options
{
chain_hostnames(no);
create_dirs (yes);
dir_perm(0755);
dns_cache(yes);
keep_hostname(yes);
log_fifo_size(2048);
log_msg_size(8192);
long_hostnames(off);
perm(0644);
stats(3600);
sync(0);
time_reopen (10);
use_dns(yes);
use_fqdn(no);
};
#----------------------------------------------------------------------
# Sources
#----------------------------------------------------------------------
source s_dgram
{ unix-stream("/dev/log"); };
source s_internal
{ internal(); };
source s_kernel
{ pipe("/proc/kmsg" log_prefix("kernel: ")); };
source s_tcp
{ udp(); };
#----------------------------------------------------------------------
# Standard Log file locations
#----------------------------------------------------------------------
destination authlog { file("/var/log/auth.log"); };
destination bootlog { file("/var/log/boot.log"); };
destination debug { file("/var/log/debug"); };
destination explan { file("/var/log/explanations"); };
destination messages { file("/var/log/messages"); };
destination routers { file("/var/log/routers.log"); };
destination secure { file("/var/log/secure"); };
destination spooler { file("/var/log/spooler"); };
destination syslog { file("/var/log/syslog"); };
destination user { file("/var/log/user.log"); };
#----------------------------------------------------------------------
# Special catch all destination sorting by host
#----------------------------------------------------------------------
destination hosts { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY_$HOST_$YEAR_$MONTH_$DAY"
owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); };
#----------------------------------------------------------------------
# Forward to a loghost server
#----------------------------------------------------------------------
#destination loghost { udp("10.1.1.254" port(514)); };
#----------------------------------------------------------------------
# Mail subsystem logs
#----------------------------------------------------------------------
destination mail { file("/var/log/mail.log"); };
destination mailerr { file("/var/log/mail/errors"); };
destination mailinfo { file("/var/log/mail/info"); };
destination mailwarn { file("/var/log/mail/warnings"); };
#----------------------------------------------------------------------
# INN news subsystem
#----------------------------------------------------------------------
destination newscrit { file("/var/log/news/critical"); };
destination newserr { file("/var/log/news/errors"); };
destination newsnotice { file("/var/log/news/notice"); };
destination newswarn { file("/var/log/news/warnings"); };
#----------------------------------------------------------------------
# Cron subsystem
#----------------------------------------------------------------------
destination cron { file("/var/log/cron.log"); };
destination crondebug { file("/var/log/cron/debug"); };
destination cronerr { file("/var/log/cron/errors"); };
destination croninfo { file("/var/log/cron/info"); };
destination cronwarn { file("/var/log/cron/warnings"); };
#----------------------------------------------------------------------
# LPR subsystem
#----------------------------------------------------------------------
destination lpr { file("/var/log/lpr.log"); };
destination lprerr { file("/var/log/lpr/errors"); };
destination lprinfo { file("/var/log/lpr/info"); };
destination lprwarn { file("/var/log/lpr/warnings"); };
#----------------------------------------------------------------------
# Kernel messages
#----------------------------------------------------------------------
destination kern { file("/var/log/kern.log"); };
destination kernerr { file("/var/log/kernel/errors"); };
destination kerninfo { file("/var/log/kernel/info"); };
destination kernwarn { file("/var/log/kernel/warnings"); };
#----------------------------------------------------------------------
# Daemon messages
#----------------------------------------------------------------------
destination daemon { file("/var/log/daemon.log"); };
destination daemonerr { file("/var/log/daemons/errors"); };
destination daemoninfo { file("/var/log/daemons/info"); };
destination daemonwarn { file("/var/log/daemons/warnings"); };
#----------------------------------------------------------------------
# Console warnings
#----------------------------------------------------------------------
destination console { file("/dev/tty12"); };
#----------------------------------------------------------------------
# All users
#----------------------------------------------------------------------
destination users { usertty("*"); };
#----------------------------------------------------------------------
# Database notes: Piping method
#----------------------------------------------------------------------
destination d_mysql {
pipe("/var/log/mysql.pipe"
template("INSERT INTO syslog_incoming (facility, priority, date, time, host, message, seq, status)
VALUES ( '$FACILITY', '$PRIORITY', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$HOST', '$MSG', '$SEQ', '$STATUS' );\n")
template-escape(yes));
};
#----------------------------------------------------------------------
# Standard filters for the standard destinations.
#----------------------------------------------------------------------
filter f_auth { facility(auth, authpriv); };
filter f_authpriv { facility(authpriv); };
filter f_cron { facility(cron); };
filter f_daemon { facility(daemon); };
filter f_kern { facility(kern); };
filter f_local1 { facility(local1); };
filter f_local2 { facility(local2); };
filter f_local3 { facility(local3); };
filter f_local4 { facility(local4); };
filter f_local5 { facility(local5); };
filter f_local6 { facility(local6); };
filter f_local7 { facility(local7); };
filter f_lpr { facility(lpr); };
filter f_mail { facility(mail); };
filter f_messages { facility(daemon, kern, user); };
filter f_news { facility(news); };
filter f_spooler { facility(uucp,news) and level(crit); };
filter f_syslog { not facility(auth, authpriv) and not facility(mail); };
filter f_user { facility(user); };
#----------------------------------------------------------------------
# Other catch-all filters
#----------------------------------------------------------------------
filter f_crit { level(crit); };
#filter f_debug { not facility(auth, authpriv, news, mail); };
filter f_debug { level(debug); };
filter f_emergency { level(emerg); };
filter f_err { level(err); };
filter f_info { level(info); };
filter f_notice { level(notice); };
filter f_warn { level(warn); };
#----------------------------------------------------------------------
# Logging
#----------------------------------------------------------------------
#
# Notes: When applying filters, remember that each subsequent filter
# acts as a filter on the previous data flow. This means that
# if the first filter limits the flow to only data from the
# auth system, a subsequent filter for authpriv will cause
# no data to be written. An example of this would be:
#
# log { source(s_dgram);
# source(s_internal);
# source(s_kernel);
# source(s_tcp);
# source(s_udp); filter(f_auth);
# filter(f_authpriv); destination(authlog); };
#
# So, one can cancel out the other.
#
# There are also certain flags that can be attached to each of the log
# statements:
#
# Flag Description
# -------- ----------------------------------------------------------
# catchall This flag means that the source of the message is ignored,
# only the filters are taken into account when matching
# messages.
# fallback This flag makes a log statement 'fallback'. Being a
# fallback statement means that only messages not matching
# any 'non-fallback' log statements will be dispatched.
# final This flag means that the processing of log statements ends
# here. Note that this doesn't necessarily mean that
# matching messages will be stored once, as they can be
# matching log statements processed prior the current one.
#----------------------------------------------------------------------
#----------------------------------------------------------------------
# Standard logging
#----------------------------------------------------------------------
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_auth); destination(authlog); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_local7); destination(bootlog); };
#log{ source(s_dgram);
# source(s_internal);
# source(s_kernel);
# source(s_tcp);
# source(s_udp); filter(f_debug); destination(debug); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_local1); destination(explan); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_local5); destination(routers); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_messages); destination(messages); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_authpriv); destination(secure); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_spooler); destination(spooler); };
log { source(s_dgram);
source(s_internal);
source(s_kernel);
source(s_tcp); filter(f_syslog); destination(syslog); };
#log { source(s_dgram);
# source(s_internal);
# source(s_kernel);
# source(s_tcp);
# source(s_udp); destination(syslog); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_user); destination(user); };
#----------------------------------------------------------------------
# Special catch all destination sorting by host
#----------------------------------------------------------------------
log { source(s_dgram);
source(s_internal);
source(s_kernel);
source(s_tcp); destination(hosts); };
#----------------------------------------------------------------------
# Mail subsystem logging
#----------------------------------------------------------------------
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_mail);
filter(f_err); destination(mailerr); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_mail);
filter(f_info); destination(mailinfo); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_mail);
filter(f_notice); destination(mailinfo); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_mail);
filter(f_warn); destination(mailwarn); };
#----------------------------------------------------------------------
# INN subsystem logging
#----------------------------------------------------------------------
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_news);
filter(f_crit); destination(newscrit); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_news);
filter(f_err); destination(newserr); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_news);
filter(f_notice); destination(newsnotice); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_news);
filter(f_warn); destination(newswarn); };
#----------------------------------------------------------------------
# Cron subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
# source(s_internal);
# source(s_tcp);
# source(s_udp); filter(f_cron); destination(crondebug); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_cron);
filter(f_err); destination(cronerr); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_cron);
filter(f_info); destination(croninfo); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_cron);
filter(f_warn); destination(cronwarn); };
#----------------------------------------------------------------------
# LPR subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
# source(s_internal);
# source(s_tcp);
# source(s_udp); filter(f_lpr); destination(lpr); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_lpr);
filter(f_err); destination(lprerr); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_lpr);
filter(f_info); destination(lprinfo); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_lpr);
filter(f_warn); destination(lprwarn); };
#----------------------------------------------------------------------
# Kernel subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
# source(s_internal);
# source(s_kernel);
# source(s_tcp);
# source(s_udp); filter(f_kern); destination(kern); };
log { source(s_dgram);
source(s_internal);
source(s_kernel);
source(s_tcp); filter(f_kern);
filter(f_err); destination(kernerr); };
log { source(s_dgram);
source(s_internal);
source(s_kernel);
source(s_tcp); filter(f_kern);
filter(f_info); destination(kerninfo); };
log { source(s_dgram);
source(s_internal);
source(s_kernel);
source(s_tcp); filter(f_kern);
filter(f_warn); destination(kernwarn); };
#----------------------------------------------------------------------
# Daemon subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
# source(s_internal);
# source(s_tcp);
# source(s_udp); filter(f_daemon); destination(daemon); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_daemon);
filter(f_err); destination(daemonerr); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_daemon);
filter(f_info); destination(daemoninfo); };
log { source(s_dgram);
source(s_internal);
source(s_tcp); filter(f_daemon);
filter(f_warn); destination(daemonwarn); };
#----------------------------------------------------------------------
# Console logging
#----------------------------------------------------------------------
# 16-Mar-03 - REP - Removed logging to the console for performance
# reasons. Since we are not really going to be
# looking at the console all the time, why log there
# anyway.
#----------------------------------------------------------------------
#log { source(s_dgram);
# source(s_internal);
# source(s_kernel);
# source(s_tcp); filter(f_syslog); destination(console); };
#----------------------------------------------------------------------
# Logging to a database
#----------------------------------------------------------------------
log { source(s_dgram);
source(s_internal);
source(s_kernel);
source(s_tcp);
destination(d_mysql); };
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Yes I checked the nslookup "IP" and it resolves it into the hostname and other host names reflect as aixser1,aixser2 and sunser1.xyzcompany.com . This is only happening for this particular host
The server is not multi-homed. Can you suggest the right options for this as you commented on my long_hostnames , use_fqdn(no) , use_dns , keep_hostname
The server is not multi-homed. Can you suggest the right options for this as you commented on my long_hostnames , use_fqdn(no) , use_dns , keep_hostname
Try enabling the long_hostname. Are all the hosts run the same OS/version/updates?
Capture the syslog packet from this host and comparing it to one of the other host's packets, would shed a light on the issue.
It looks as though the packet from this host is parsed and the year from the packet is seen in the order where the host is expected.
Try the suggestion of woolmilkporc, add the IP into the syslog server's /etc/hosts file.
Capture the syslog packet from this host and comparing it to one of the other host's packets, would shed a light on the issue.
It looks as though the packet from this host is parsed and the year from the packet is seen in the order where the host is expected.
Try the suggestion of woolmilkporc, add the IP into the syslog server's /etc/hosts file.
ASKER
Ok As suggested woolmilkporc I have added them to /etc/hosts, Let me see if this does the trick
ASKER
Adding the server names in /etc/hosts didn't solve the issue...Its still showing 2008 as the hostname
Host Date Time Message Facility Level Options
2008 2008-11-13 15:49:39 Nov 13 16:09:04 EXT: %LICMGR-2-LOG_LIC_GRACE_EX PIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit
2008 2008-11-13 15:40:55 Nov 13 15:28:23 EST: %LICMGR-2-LOG_LIC_GRACE_EX PIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit
2008 2008-11-13 15:36:11 Nov 13 15:29:34 EST: %LICMGR-2-LOG_LIC_GRACE_EX PIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit
Host Date Time Message Facility Level Options
2008 2008-11-13 15:49:39 Nov 13 16:09:04 EXT: %LICMGR-2-LOG_LIC_GRACE_EX
2008 2008-11-13 15:40:55 Nov 13 15:28:23 EST: %LICMGR-2-LOG_LIC_GRACE_EX
2008 2008-11-13 15:36:11 Nov 13 15:29:34 EST: %LICMGR-2-LOG_LIC_GRACE_EX
ASKER
Any Ideas....
You can test the setup by using logger to generate a syslog message.
logger -p syslog.crit "2009 test 13 13:09:04 EXT: test error"
You have an issue with the parsing of this particular datagram. I suspect that the host will now reflect 2009 versus 2008 for the event.
Note that the date included in the recorded event on the rr9513-3.xyzcompany.com system is replicated in different sections in the syslog server records.
You may need to create a specific conversion for datagrams coming from this server.
You have not answered whether the OS/kernel, and updates are the same on rr9513-3.xyzcompany.com to the other systems.
logger -p syslog.crit "2009 test 13 13:09:04 EXT: test error"
You have an issue with the parsing of this particular datagram. I suspect that the host will now reflect 2009 versus 2008 for the event.
Note that the date included in the recorded event on the rr9513-3.xyzcompany.com system is replicated in different sections in the syslog server records.
You may need to create a specific conversion for datagrams coming from this server.
You have not answered whether the OS/kernel, and updates are the same on rr9513-3.xyzcompany.com to the other systems.
ASKER
Ohh I am sorry, the one where I am having the issue is a cisco switch. Its only for these 4 cisco switched I am having this problem. As a matter of fact I have tested this just now...
Before I had
ALL SERVERS -> SYSLOG-NG server
Now I have
ALL SERVERS -> gateway server -> SYSLOG-NG server
So i can see that on the gateway server I can see the right hostname....but in either cases syslog-ng server is not showing the right hostname insted..2008
Before I had
ALL SERVERS -> SYSLOG-NG server
Now I have
ALL SERVERS -> gateway server -> SYSLOG-NG server
So i can see that on the gateway server I can see the right hostname....but in either cases syslog-ng server is not showing the right hostname insted..2008
The gateway server is not properly generating the datagram.
ASKER
how can I fix this.... Since its s switch does a separate source be defined....?///
The issue is not with the device that generates the original datagram, the issue is that the gateway server seems to reformat or the datagram or the receiving syslog server handles the packet differently.
What is the purpose of your network layout change? Compare the recorded entry on the gateway server versus the received data on the syslog server.
Without knowing how the data packets differ, there is no way to suggest a mechanism to correct the situation.
What is the purpose of your network layout change? Compare the recorded entry on the gateway server versus the received data on the syslog server.
Without knowing how the data packets differ, there is no way to suggest a mechanism to correct the situation.
ASKER
Hello Arnold,
Messages on Gateway server look fine and it actually shows the hostname but on the syslog-ng server.they seems to be different.
Messages on Gateway server look fine and it actually shows the hostname but on the syslog-ng server.they seems to be different.
ASKER
Also my gateway is an AIX server and syslog-ng server is redhat
Try capturing the packet the gateway server receives and the packet the redhat system receives. compare the data in the packets.
How are the servers that send these syslog messages differ if at all?
How are the servers that send these syslog messages differ if at all?
your new /etc/hosts entry (if any) for this particular host is OK?
If yes, I fear I can't help in this case.
Good Luck!
wmp