Link to home
Start Free TrialLog in
Avatar of jdenver247
jdenver247

asked on

Host Entry Error (Cacti Syslog Plugin)

I am using Syslog-ng Server and forwarding all the syslog messages to this server. I have around 98 hosts reporting. Here are syslog messages for some of them
----------------------------------------------------------------------------------------------------------------------------
Nov 13 12:49:39 rr9513-3.xyzcompany.com syslog:crit : 2008 Nov 13 13:09:04 EXT: %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG.
Nov 13 12:59:36 rr9513-2.xyzcompany.com syslog:crit : 2008 Nov 13 12:59:36 EST: %LICMGR-2-LOG_LIC_GRA                                                          .
Nov 13 13:19:01 rr9513-1.xyzcompany.com syslog:crit : 2008 Nov 13 13:19:01 EST: %LICMGR-2-LOG_LIC_GRA
----------------------------------------------------------------------------------------------------------------------------

Now in the above case rr9513-1.xyzcompany.com  is the hostname but when I forward the messages to the syslog-ng server. Here is how it looks like....
-----------------------------------------------------------------------------------------------------------------------------
Host   Date Time   Message Facility Level Options
2008  2008-11-13 13:36:11  Nov 13 13:29:34 EST: %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit  
2008  2008-11-13 13:34:59  Nov 13 13:31:05 EST: %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit  
----------------------------------------------------------------------------------------------------------------------------

It shows 2008 instead of rr9513-1.xyzcompany.com  . Attached below is the syslog-conf script also I want to mention that this is happening only for this particular host

#----------------------------------------------------------------------
#  Program:  syslog-ng.conf
#  Notes:    Embedded most of the manual notes within the configuration
#            file.  The original manual can be found at:
#
#----------------------------------------------------------------------
#  05-Nov-08 - REP - Added some extra definitions to the file.
#  12-Nov-08 - REP - Added back the comments on filtering.
#  13-Nov-08 - REP - Further modified for local environment,Initial creation for testing.
 
 
#----------------------------------------------------------------------
#  Options
#----------------------------------------------------------------------
#
#  Name                       Values   Description
#  -------------------------  -------  ------------------------------------
#  bad_hostname               reg exp  A regexp which matches hostnames
#                                      which should not be taken as such.
#  chain_hostnames            y/n      Enable or disable the chained
#                                      hostname format.
#  create_dirs                y/n      Enable or disable directory creation
#                                      for destination files.
#  dir_group                  groupid
#  dir_owner                  userid
#  dir_perm                   perm
#  dns_cache                  y/n      Enable or disable DNS cache usage.
#  dns_cache_expire           num      Number of seconds while a successful
#                                      lookup is cached.
#  dns_cache_expire_failed    num      Number of seconds while a failed
#                                      lookup is cached.
#  dns_cache_size             num      Number of hostnames in the DNS cache.
#  gc_busy_threshold          num      Sets the threshold value for the
#                                      garbage collector, when syslog-ng is
#                                      busy. GC phase starts when the number
#                                      of allocated objects reach this
#                                      number. Default: 3000.
#  gc_idle_threshold          num      Sets the threshold value for the
#                                      garbage collector, when syslog-ng is
#                                      idle. GC phase starts when the number
#                                      of allocated objects reach this
#                                      number. Default: 100.
#  group                      groupid
#  keep_hostname              y/n      Enable or disable hostname rewriting.
#                                      This means that if the log entry had
#                                      been passed through at least one other
#                                      logging system, the ORIGINAL hostname
#                                      will be kept attached to the log.
#                                      Otherwise the last logger will be
#                                      considered the log entry owner and
#                                      the log entry will appear to have
#                                      come from that host.
#  log_fifo_size              num      The number of lines fitting to the
#                                      output queue
#  log_msg_size               num      Maximum length of message in bytes.
#  long_hostnames             on/off   This options appears to only really
#                                      have an affect on the local system.
#                                      which removes the source of the log.
#                                      As an example, normally the local
#                                      logs will state src@hostname, but
#                                      with this feature off, the source
#                                      is not reported.
#  mark                       num      The number of seconds between two
#                                      MARK lines. NOTE: not implemented
#                                      yet.
#  owner                      userid
#  perm                       perm
#  stats                      num      The number of seconds between two
#                                      STATS.
#  sync                       num      The number of lines buffered before
#                                      written to file
#  time_reap                  num      The time to wait before an idle
#                                      destination file is closed.
#  time_reopen                num      The time to wait before a died
#                                      connection is reestablished
#  use_dns                    y/n      Enable or disable DNS usage.
#                                      syslog-ng blocks on DNS queries,
#                                      so enabling DNS may lead to a
#                                      Denial of Service attack. To
#                                      prevent DoS, protect your
#                                      syslog-ng network endpoint with
#                                      firewall rules, and make sure that
#                                      all hosts, which may get to
#                                      syslog-ng is resolvable.
#  use_fqdn                   y/n      Add Fully Qualified Domain Name
#                                      instead of short hostname.
#  use_time_recvd             y/n      Use the time a message is
#                                      received instead of the one
#                                      specified in the message.
#----------------------------------------------------------------------
#  12-Nov-08 - REP - Some of the Configuration option are not fuctioning
#
#  12-Mar-03 - REP - I have increased the log_msg_size and log_fifo_size
#                    to increase the amount of buffering that we do.
#                    While for most systems this may not have a noticeable
#                    affect, it will for systems that are at the end of a
#                    lot of logging systems.
#----------------------------------------------------------------------
options
  {
    chain_hostnames(no);
    create_dirs (yes);
    dir_perm(0755);
    dns_cache(yes);
    keep_hostname(yes);
    log_fifo_size(2048);
    log_msg_size(8192);
    long_hostnames(off);
    perm(0644);
    stats(3600);
    sync(0);
    time_reopen (10);
    use_dns(yes);
    use_fqdn(no);
  };
 
#----------------------------------------------------------------------
#  Sources
#----------------------------------------------------------------------
 
source s_dgram
  { unix-stream("/dev/log"); };
 
source s_internal
  { internal(); };
 
source s_kernel
  { pipe("/proc/kmsg" log_prefix("kernel: ")); };
 
source s_tcp
  { udp(); };
 
#----------------------------------------------------------------------
#  Standard Log file locations
#----------------------------------------------------------------------
destination authlog        { file("/var/log/auth.log"); };
destination bootlog        { file("/var/log/boot.log"); };
destination debug          { file("/var/log/debug"); };
destination explan         { file("/var/log/explanations"); };
destination messages       { file("/var/log/messages"); };
destination routers        { file("/var/log/routers.log"); };
destination secure         { file("/var/log/secure"); };
destination spooler        { file("/var/log/spooler"); };
destination syslog         { file("/var/log/syslog"); };
destination user           { file("/var/log/user.log"); };
 
#----------------------------------------------------------------------
#  Special catch all destination sorting by host
#----------------------------------------------------------------------
destination hosts          { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY_$HOST_$YEAR_$MONTH_$DAY"
                             owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); };
 
#----------------------------------------------------------------------
#  Forward to a loghost server
#----------------------------------------------------------------------
#destination loghost       { udp("10.1.1.254" port(514)); };
 
#----------------------------------------------------------------------
#  Mail subsystem logs
#----------------------------------------------------------------------
destination mail           { file("/var/log/mail.log"); };
destination mailerr        { file("/var/log/mail/errors"); };
destination mailinfo       { file("/var/log/mail/info"); };
destination mailwarn       { file("/var/log/mail/warnings"); };
 
#----------------------------------------------------------------------
#  INN news subsystem
#----------------------------------------------------------------------
destination newscrit       { file("/var/log/news/critical"); };
destination newserr        { file("/var/log/news/errors"); };
destination newsnotice     { file("/var/log/news/notice"); };
destination newswarn       { file("/var/log/news/warnings"); };
 
#----------------------------------------------------------------------
#  Cron subsystem
#----------------------------------------------------------------------
destination cron           { file("/var/log/cron.log"); };
destination crondebug      { file("/var/log/cron/debug"); };
destination cronerr        { file("/var/log/cron/errors"); };
destination croninfo       { file("/var/log/cron/info"); };
destination cronwarn       { file("/var/log/cron/warnings"); };
 
#----------------------------------------------------------------------
#  LPR subsystem
#----------------------------------------------------------------------
destination lpr            { file("/var/log/lpr.log"); };
destination lprerr         { file("/var/log/lpr/errors"); };
destination lprinfo        { file("/var/log/lpr/info"); };
destination lprwarn        { file("/var/log/lpr/warnings"); };
 
#----------------------------------------------------------------------
#  Kernel messages
#----------------------------------------------------------------------
destination kern           { file("/var/log/kern.log"); };
destination kernerr        { file("/var/log/kernel/errors"); };
destination kerninfo       { file("/var/log/kernel/info"); };
destination kernwarn       { file("/var/log/kernel/warnings"); };
 
#----------------------------------------------------------------------
#  Daemon messages
#----------------------------------------------------------------------
destination daemon         { file("/var/log/daemon.log"); };
destination daemonerr      { file("/var/log/daemons/errors"); };
destination daemoninfo     { file("/var/log/daemons/info"); };
destination daemonwarn     { file("/var/log/daemons/warnings"); };
 
#----------------------------------------------------------------------
#  Console warnings
#----------------------------------------------------------------------
destination console        { file("/dev/tty12"); };
 
#----------------------------------------------------------------------
#  All users
#----------------------------------------------------------------------
destination users          { usertty("*"); };
 
#----------------------------------------------------------------------
#  Database notes: Piping method
#----------------------------------------------------------------------
 
destination d_mysql {
                    pipe("/var/log/mysql.pipe"
                    template("INSERT INTO syslog_incoming (facility, priority, date, time, host, message, seq, status)
                    VALUES ( '$FACILITY', '$PRIORITY', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$HOST', '$MSG', '$SEQ', '$STATUS' );\n")
                    template-escape(yes));
};
 
 
#----------------------------------------------------------------------
#  Standard filters for the standard destinations.
#----------------------------------------------------------------------
filter      f_auth         { facility(auth, authpriv); };
filter      f_authpriv     { facility(authpriv); };
filter      f_cron         { facility(cron); };
filter      f_daemon       { facility(daemon); };
filter      f_kern         { facility(kern); };
filter      f_local1       { facility(local1); };
filter      f_local2       { facility(local2); };
filter      f_local3       { facility(local3); };
filter      f_local4       { facility(local4); };
filter      f_local5       { facility(local5); };
filter      f_local6       { facility(local6); };
filter      f_local7       { facility(local7); };
filter      f_lpr          { facility(lpr); };
filter      f_mail         { facility(mail); };
filter      f_messages     { facility(daemon, kern, user); };
filter      f_news         { facility(news); };
filter      f_spooler      { facility(uucp,news) and level(crit); };
filter      f_syslog       { not facility(auth, authpriv) and not facility(mail); };
filter      f_user         { facility(user); };
 
#----------------------------------------------------------------------
#  Other catch-all filters
#----------------------------------------------------------------------
filter      f_crit         { level(crit); };
#filter     f_debug        { not facility(auth, authpriv, news, mail); };
filter      f_debug        { level(debug); };
filter      f_emergency    { level(emerg); };
filter      f_err          { level(err); };
filter      f_info         { level(info); };
filter      f_notice       { level(notice); };
filter      f_warn         { level(warn); };
 
#----------------------------------------------------------------------
#  Logging
#----------------------------------------------------------------------
#
#  Notes:  When applying filters, remember that each subsequent filter
#          acts as a filter on the previous data flow.  This means that
#          if the first filter limits the flow to only data from the
#          auth system, a subsequent filter for authpriv will cause
#          no data to be written.  An example of this would be:
#
# log { source(s_dgram);
#       source(s_internal);
#       source(s_kernel);
#       source(s_tcp);
#       source(s_udp);      filter(f_auth);
#                           filter(f_authpriv);  destination(authlog); };
#
#          So, one can cancel out the other.
#
#  There are also certain flags that can be attached to each of the log
#  statements:
#
#  Flag      Description
#  --------  ----------------------------------------------------------
#  catchall  This flag means that the source of the message is ignored,
#            only the filters are taken into account when matching
#            messages.
#  fallback  This flag makes a log statement 'fallback'. Being a
#            fallback statement means that only messages not matching
#            any 'non-fallback' log statements will be dispatched.
#  final     This flag means that the processing of log statements ends
#            here. Note that this doesn't necessarily mean that
#            matching messages will be stored once, as they can be
#            matching log statements processed prior the current one.
#----------------------------------------------------------------------
 
#----------------------------------------------------------------------
#  Standard logging
#----------------------------------------------------------------------
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_auth);      destination(authlog); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_local7);    destination(bootlog); };
#log{ source(s_dgram);
#      source(s_internal);
#      source(s_kernel);
#      source(s_tcp);
#      source(s_udp);      filter(f_debug);     destination(debug); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_local1);    destination(explan); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_local5);    destination(routers); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_messages);  destination(messages); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_authpriv);  destination(secure); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_spooler);   destination(spooler); };
log { source(s_dgram);
      source(s_internal);
      source(s_kernel);
      source(s_tcp);      filter(f_syslog);    destination(syslog); };
#log { source(s_dgram);
#      source(s_internal);
#      source(s_kernel);
#      source(s_tcp);
#      source(s_udp);                       destination(syslog); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_user);      destination(user); };
 
#----------------------------------------------------------------------
#  Special catch all destination sorting by host
#----------------------------------------------------------------------
log { source(s_dgram);
      source(s_internal);
      source(s_kernel);
      source(s_tcp);                           destination(hosts); };
 
 
#----------------------------------------------------------------------
#  Mail subsystem logging
#----------------------------------------------------------------------
 
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_mail);
                          filter(f_err);       destination(mailerr); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_mail);
                          filter(f_info);      destination(mailinfo); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_mail);
                          filter(f_notice);    destination(mailinfo); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_mail);
                          filter(f_warn);      destination(mailwarn); };
 
#----------------------------------------------------------------------
#  INN subsystem logging
#----------------------------------------------------------------------
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_news);
                          filter(f_crit);      destination(newscrit); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_news);
                          filter(f_err);       destination(newserr); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_news);
                          filter(f_notice);    destination(newsnotice); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_news);
                          filter(f_warn);      destination(newswarn); };
 
#----------------------------------------------------------------------
#  Cron subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
#      source(s_internal);
#      source(s_tcp);
#      source(s_udp);     filter(f_cron);      destination(crondebug); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_cron);
                          filter(f_err);       destination(cronerr); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_cron);
                          filter(f_info);      destination(croninfo); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_cron);
                          filter(f_warn);      destination(cronwarn); };
 
#----------------------------------------------------------------------
#  LPR subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
#      source(s_internal);
#      source(s_tcp);
#      source(s_udp);     filter(f_lpr);       destination(lpr); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_lpr);
                          filter(f_err);       destination(lprerr); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_lpr);
                          filter(f_info);      destination(lprinfo); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_lpr);
                          filter(f_warn);      destination(lprwarn); };
 
#----------------------------------------------------------------------
#  Kernel subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
#      source(s_internal);
#      source(s_kernel);
#      source(s_tcp);
#      source(s_udp);     filter(f_kern);      destination(kern); };
log { source(s_dgram);
      source(s_internal);
      source(s_kernel);
      source(s_tcp);      filter(f_kern);
                          filter(f_err);       destination(kernerr); };
log { source(s_dgram);
      source(s_internal);
      source(s_kernel);
      source(s_tcp);      filter(f_kern);
                          filter(f_info);      destination(kerninfo); };
log { source(s_dgram);
      source(s_internal);
      source(s_kernel);
      source(s_tcp);      filter(f_kern);
                          filter(f_warn);      destination(kernwarn); };
 
#----------------------------------------------------------------------
#  Daemon subsystem logging
#----------------------------------------------------------------------
#log { source(s_dgram);
#      source(s_internal);
#      source(s_tcp);
#      source(s_udp);     filter(f_daemon);    destination(daemon); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_daemon);
                          filter(f_err);       destination(daemonerr); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_daemon);
                          filter(f_info);      destination(daemoninfo); };
log { source(s_dgram);
      source(s_internal);
      source(s_tcp);      filter(f_daemon);
                          filter(f_warn);      destination(daemonwarn); };
 
#----------------------------------------------------------------------
#  Console logging
#----------------------------------------------------------------------
#  16-Mar-03 - REP - Removed logging to the console for performance
#                    reasons.  Since we are not really going to be
#                    looking at the console all the time, why log there
#                    anyway.
#----------------------------------------------------------------------
#log { source(s_dgram);
#      source(s_internal);
#      source(s_kernel);
#      source(s_tcp);      filter(f_syslog);    destination(console); };
 
#----------------------------------------------------------------------
#  Logging to a database
#----------------------------------------------------------------------
log { source(s_dgram);
      source(s_internal);
      source(s_kernel);
      source(s_tcp);
      destination(d_mysql); };

Open in new window

Avatar of woolmilkporc
woolmilkporc
Flag of Germany image

Just to be sure -
your new /etc/hosts entry (if any)  for this particular host is OK?
If yes, I fear I can't help in this case.

Good Luck!

wmp
ASKER CERTIFIED SOLUTION
Avatar of arnold
arnold
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of jdenver247
jdenver247

ASKER

Yes I checked the nslookup "IP" and it resolves it into the hostname and other host names reflect as aixser1,aixser2 and sunser1.xyzcompany.com . This is only happening for this particular host

The server is not multi-homed. Can you suggest the right options for this as you commented on my long_hostnames , use_fqdn(no) , use_dns , keep_hostname
Try enabling the long_hostname.  Are all the hosts run the same OS/version/updates?

Capture the syslog packet from this host and comparing it to one of the other host's packets, would shed a light on the issue.

It looks as though the packet from this host is parsed and the year from the packet is seen in the order where the host is expected.
Try the suggestion of woolmilkporc, add the IP into the syslog server's /etc/hosts file.
Ok As suggested woolmilkporc I have added them to /etc/hosts, Let me see if this does the trick
Adding the server names in /etc/hosts didn't solve the issue...Its still showing 2008 as the hostname

Host   Date Time   Message Facility Level Options
2008  2008-11-13 15:49:39  Nov 13 16:09:04 EXT: %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit  
2008  2008-11-13 15:40:55  Nov 13 15:28:23 EST: %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit  
2008  2008-11-13 15:36:11  Nov 13 15:29:34 EST: %LICMGR-2-LOG_LIC_GRACE_EXPIRED: Grace period expired for feature FM_SERVER_PKG. syslog crit  
Any Ideas....
You can test the setup by using logger to generate a syslog message.
logger -p syslog.crit "2009 test 13 13:09:04 EXT: test error"

You have an issue with the parsing of this particular datagram.  I suspect that the host will now reflect 2009 versus 2008 for the event.
Note that the date included in the recorded event on the rr9513-3.xyzcompany.com system is replicated in different sections in the syslog server records.

You may need to create a specific conversion for datagrams coming from this server.

You have not answered whether the OS/kernel, and updates are the same on rr9513-3.xyzcompany.com to the other systems.
Ohh I am sorry, the one where I am having the issue is a cisco switch. Its only for these 4 cisco switched I am having this problem. As a matter of fact I have tested this just now...

Before I had
ALL SERVERS -> SYSLOG-NG server

Now I have
ALL SERVERS -> gateway server -> SYSLOG-NG server

So i can see that on the gateway server I can see the right hostname....but in either cases syslog-ng server is not showing the right hostname insted..2008

The gateway server is not properly generating the datagram.
how can I fix this.... Since its s switch does a separate source be defined....?///
The issue is not with the device that generates the original datagram, the issue is that the gateway server seems to reformat or the datagram or the receiving syslog server handles the packet differently.
What is the purpose of your network layout change? Compare the recorded entry on the gateway server versus the received data on the syslog server.
Without knowing how the data packets differ, there is no way to suggest a mechanism to correct the situation.
Hello Arnold,

Messages on Gateway server look fine and it actually shows the hostname but on the syslog-ng server.they seems to be different.
Also my gateway is an AIX server and syslog-ng server is redhat
Try capturing the packet the gateway server receives and the packet the redhat system receives.  compare the data in the packets.  
How are the servers that send these syslog messages differ if at all?