amigan_99
asked on
Syslog is broken
Someone booted the host and now syslog's not working. Would appreciate trouble-shooting tips.
ps -ef | grep syslog
root 3954 1 0 15:50 ? 00:00:00 syslogd -m 0
/etc/syslog.conf snippet:
local6.warning /var/log/cisco/routers/rou ter
local6.debug /var/log/cisco/routers/rou ter
local6.info /var/log/cisco/routers/rou ter
tcpdump output:
15:59:50.179709 IP c3845-inet1.pcmt.local.529 31 > netmon.pooched.local.syslo g: UDP, length 142
15:59:50.180685 IP c3845-inet1.pcmt.local.529 31 > netmon.pooched.local.syslo g: UDP, length 142
15:59:50.469254 IP c3845-inet1.pcmt.local.529 31 > netmon.pooched.local.syslo g: UDP, length 142
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
ps -ef | grep syslog
root 3954 1 0 15:50 ? 00:00:00 syslogd -m 0
/etc/syslog.conf snippet:
local6.warning /var/log/cisco/routers/rou
local6.debug /var/log/cisco/routers/rou
local6.info /var/log/cisco/routers/rou
tcpdump output:
15:59:50.179709 IP c3845-inet1.pcmt.local.529
15:59:50.180685 IP c3845-inet1.pcmt.local.529
15:59:50.469254 IP c3845-inet1.pcmt.local.529
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
ASKER
Update: So syslog itself is working somewhat in that boot.log is working, /var/log/messages, /var/log/mail..
But it's the syslog traffic that can't find its way to facility 6. But that appears set fine in /etc/syslog.conf.
/etc/syslog.conf snippet:
local6.warning /var/log/cisco/routers/rou ter
local6.debug /var/log/cisco/routers/rou ter
local6.info /var/log/cisco/routers/rou ter
As I showed before - the cisco router syslog traffic arrives at the interface of the linux host. But for some reason the linux host (netmon) sends back "udp port syslog unreachable".
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
But it's the syslog traffic that can't find its way to facility 6. But that appears set fine in /etc/syslog.conf.
/etc/syslog.conf snippet:
local6.warning /var/log/cisco/routers/rou
local6.debug /var/log/cisco/routers/rou
local6.info /var/log/cisco/routers/rou
As I showed before - the cisco router syslog traffic arrives at the interface of the linux host. But for some reason the linux host (netmon) sends back "udp port syslog unreachable".
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
ASKER
I don't see syslog port 514 in play..
[root@netmon etc]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:832 *:* LISTEN
tcp 0 0 *:mysql *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 netmon.pcmt.local:ipp *:* LISTEN
tcp 0 0 netmon.pcmt.local:smtp *:* LISTEN
tcp 0 0 *:8000 *:* LISTEN
tcp 0 0 *:http *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 *:https *:* LISTEN
udp 0 0 *:826 *:*
udp 0 0 *:829 *:*
udp 0 0 *:tftp *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:ipp *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6134 /tmp/.font-unix/fs7100
unix 2 [ ACC ] STREAM LISTENING 8331 /tmp/.gdm_socket
unix 2 [ ACC ] STREAM LISTENING 8450 /tmp/.X11-unix/X0
unix 2 [ ACC ] STREAM LISTENING 5937 /dev/gpmctl
unix 2 [ ACC ] STREAM LISTENING 5415 /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 5726 /var/lib/mysql/mysql.sock
unix 2 [ ACC ] STREAM LISTENING 6211 /var/run/dbus/system_bus_s ocket
[root@netmon etc]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:832 *:* LISTEN
tcp 0 0 *:mysql *:* LISTEN
tcp 0 0 *:sunrpc *:* LISTEN
tcp 0 0 netmon.pcmt.local:ipp *:* LISTEN
tcp 0 0 netmon.pcmt.local:smtp *:* LISTEN
tcp 0 0 *:8000 *:* LISTEN
tcp 0 0 *:http *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 *:https *:* LISTEN
udp 0 0 *:826 *:*
udp 0 0 *:829 *:*
udp 0 0 *:tftp *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:ipp *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 6134 /tmp/.font-unix/fs7100
unix 2 [ ACC ] STREAM LISTENING 8331 /tmp/.gdm_socket
unix 2 [ ACC ] STREAM LISTENING 8450 /tmp/.X11-unix/X0
unix 2 [ ACC ] STREAM LISTENING 5937 /dev/gpmctl
unix 2 [ ACC ] STREAM LISTENING 5415 /var/run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 5726 /var/lib/mysql/mysql.sock
unix 2 [ ACC ] STREAM LISTENING 6211 /var/run/dbus/system_bus_s
Syslog should be listed as listening to port 514.
Restart syslog and check /var/log/messages or /var/log/syslog to see if any errors are reported about binding to the port, or specific interfaces.
Restart syslog and check /var/log/messages or /var/log/syslog to see if any errors are reported about binding to the port, or specific interfaces.
ASKER
[root@netmon etc]# /etc/rc.d/init.d/syslog restart
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
From /sys/log/messages:
Oct 1 17:53:45 netmon kernel: Kernel logging (proc) stopped.
Oct 1 17:53:45 netmon kernel: Kernel log daemon terminating.
Oct 1 17:53:46 netmon syslog: klogd shutdown succeeded
Oct 1 17:53:46 netmon exiting on signal 15
Oct 1 17:53:47 netmon syslogd 1.4.1: restart.
Oct 1 17:53:47 netmon syslog: syslogd startup succeeded
Oct 1 17:53:47 netmon syslog: klogd startup succeeded
Oct 1 17:53:47 netmon kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 1 17:53:47 netmon syslog: syslogd shutdown succeeded
[root@netmon log]# ps -ef | grep syslogd
root 4368 1 0 17:53 ? 00:00:00 syslogd -m 0
[root@netmon etc]# netstat -l | grep udp
udp 0 0 *:826 *:*
udp 0 0 *:829 *:*
udp 0 0 *:tftp *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:ipp *:*
Nothing in the router log file:
[root@netmon log]# ls -l /var/log/cisco/routers/rou ter
-rw-rw-rw- 1 root root 0 Oct 1 04:02 /var/log/cisco/routers/rou ter
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
From /sys/log/messages:
Oct 1 17:53:45 netmon kernel: Kernel logging (proc) stopped.
Oct 1 17:53:45 netmon kernel: Kernel log daemon terminating.
Oct 1 17:53:46 netmon syslog: klogd shutdown succeeded
Oct 1 17:53:46 netmon exiting on signal 15
Oct 1 17:53:47 netmon syslogd 1.4.1: restart.
Oct 1 17:53:47 netmon syslog: syslogd startup succeeded
Oct 1 17:53:47 netmon syslog: klogd startup succeeded
Oct 1 17:53:47 netmon kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct 1 17:53:47 netmon syslog: syslogd shutdown succeeded
[root@netmon log]# ps -ef | grep syslogd
root 4368 1 0 17:53 ? 00:00:00 syslogd -m 0
[root@netmon etc]# netstat -l | grep udp
udp 0 0 *:826 *:*
udp 0 0 *:829 *:*
udp 0 0 *:tftp *:*
udp 0 0 *:sunrpc *:*
udp 0 0 *:ipp *:*
Nothing in the router log file:
[root@netmon log]# ls -l /var/log/cisco/routers/rou
-rw-rw-rw- 1 root root 0 Oct 1 04:02 /var/log/cisco/routers/rou
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
and then restart syslog with /etc/init.d/syslog restart (service syslog restart)
ASKER
All kneel to the WizRd! Woot!
[root@netmon routers]# ls -l
total 311000
-rw-rw-rw- 1 root root 103197 Oct 1 21:01 router
-rw-r--r-- 1 root root 7899873 Sep 22 04:02 router.10.gz
-rw-r--r-- 1 root root 8161644 Sep 21 04:02 router.11.gz
-rw-r--r-- 1 root root 10608622 Sep 20 04:02 router.12.gz
[root@netmon routers]# ls -l
total 311000
-rw-rw-rw- 1 root root 103197 Oct 1 21:01 router
-rw-r--r-- 1 root root 7899873 Sep 22 04:02 router.10.gz
-rw-r--r-- 1 root root 8161644 Sep 21 04:02 router.11.gz
-rw-r--r-- 1 root root 10608622 Sep 20 04:02 router.12.gz
If you restart syslog, does it begin working?
Does 'netstat -al' show syslog listening?