Link to home
Start Free TrialLog in
Avatar of amigan_99
amigan_99Flag for United States of America

asked on

Syslog is broken

Someone booted the host and now syslog's not working.  Would appreciate trouble-shooting tips.

ps -ef | grep syslog
root      3954     1  0 15:50 ?        00:00:00 syslogd -m 0

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

tcpdump output:
15:59:50.179709 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.180685 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469254 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
Avatar of WizRd-Linux
WizRd-Linux
Flag of Australia image

Iptables hasn't changed?
If you restart syslog, does it begin working?
Does 'netstat -al' show syslog listening?
Avatar of amigan_99

ASKER

Update: So syslog itself is working somewhat in that boot.log is working, /var/log/messages, /var/log/mail..

But it's the syslog traffic that can't find its way to facility 6.  But that appears set fine in /etc/syslog.conf.

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

As I showed before - the cisco router syslog traffic arrives at the interface of the linux host.  But for some reason the linux host (netmon)  sends back "udp port syslog unreachable".  

15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
I don't see syslog port 514 in play..

[root@netmon etc]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 *:832                       *:*                         LISTEN
tcp        0      0 *:mysql                     *:*                         LISTEN
tcp        0      0 *:sunrpc                    *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:ipp       *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:smtp      *:*                         LISTEN
tcp        0      0 *:8000                      *:*                         LISTEN
tcp        0      0 *:http                      *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 *:https                     *:*                         LISTEN
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     6134   /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     8331   /tmp/.gdm_socket
unix  2      [ ACC ]     STREAM     LISTENING     8450   /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     5937   /dev/gpmctl
unix  2      [ ACC ]     STREAM     LISTENING     5415   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     5726   /var/lib/mysql/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     6211   /var/run/dbus/system_bus_socket
Syslog should be listed as listening to port 514.

Restart syslog and check /var/log/messages or /var/log/syslog to see if any errors are reported about binding to the port, or specific interfaces.
[root@netmon etc]# /etc/rc.d/init.d/syslog restart
Shutting down kernel logger:                               [  OK  ]
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]

From /sys/log/messages:
Oct  1 17:53:45 netmon kernel: Kernel logging (proc) stopped.
Oct  1 17:53:45 netmon kernel: Kernel log daemon terminating.
Oct  1 17:53:46 netmon syslog: klogd shutdown succeeded
Oct  1 17:53:46 netmon exiting on signal 15
Oct  1 17:53:47 netmon syslogd 1.4.1: restart.
Oct  1 17:53:47 netmon syslog: syslogd startup succeeded
Oct  1 17:53:47 netmon syslog: klogd startup succeeded
Oct  1 17:53:47 netmon kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct  1 17:53:47 netmon syslog: syslogd shutdown succeeded

[root@netmon log]# ps -ef | grep syslogd
root      4368     1  0 17:53 ?        00:00:00 syslogd -m 0

[root@netmon etc]# netstat -l | grep udp
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*

Nothing in the router log file:
[root@netmon log]# ls -l /var/log/cisco/routers/router
-rw-rw-rw-  1 root root 0 Oct  1 04:02 /var/log/cisco/routers/router




ASKER CERTIFIED SOLUTION
Avatar of WizRd-Linux
WizRd-Linux
Flag of Australia image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
and then restart syslog with /etc/init.d/syslog restart (service syslog restart)
All kneel to the WizRd!  Woot!

[root@netmon routers]# ls -l
total 311000
-rw-rw-rw-  1 root root   103197 Oct  1 21:01 router
-rw-r--r--  1 root root  7899873 Sep 22 04:02 router.10.gz
-rw-r--r--  1 root root  8161644 Sep 21 04:02 router.11.gz
-rw-r--r--  1 root root 10608622 Sep 20 04:02 router.12.gz