?
Solved

Syslog is broken

Posted on 2008-10-01
8
Medium Priority
?
872 Views
Last Modified: 2013-12-16
Someone booted the host and now syslog's not working.  Would appreciate trouble-shooting tips.

ps -ef | grep syslog
root      3954     1  0 15:50 ?        00:00:00 syslogd -m 0

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

tcpdump output:
15:59:50.179709 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.180685 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469254 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
0
Comment
Question by:amigan_99
  • 4
  • 4
8 Comments
 
LVL 13

Expert Comment

by:WizRd-Linux
ID: 22620098
Iptables hasn't changed?
If you restart syslog, does it begin working?
Does 'netstat -al' show syslog listening?
0
 
LVL 1

Author Comment

by:amigan_99
ID: 22620162
Update: So syslog itself is working somewhat in that boot.log is working, /var/log/messages, /var/log/mail..

But it's the syslog traffic that can't find its way to facility 6.  But that appears set fine in /etc/syslog.conf.

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

As I showed before - the cisco router syslog traffic arrives at the interface of the linux host.  But for some reason the linux host (netmon)  sends back "udp port syslog unreachable".  

15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
0
 
LVL 1

Author Comment

by:amigan_99
ID: 22620276
I don't see syslog port 514 in play..

[root@netmon etc]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 *:832                       *:*                         LISTEN
tcp        0      0 *:mysql                     *:*                         LISTEN
tcp        0      0 *:sunrpc                    *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:ipp       *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:smtp      *:*                         LISTEN
tcp        0      0 *:8000                      *:*                         LISTEN
tcp        0      0 *:http                      *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 *:https                     *:*                         LISTEN
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     6134   /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     8331   /tmp/.gdm_socket
unix  2      [ ACC ]     STREAM     LISTENING     8450   /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     5937   /dev/gpmctl
unix  2      [ ACC ]     STREAM     LISTENING     5415   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     5726   /var/lib/mysql/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     6211   /var/run/dbus/system_bus_socket
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 
LVL 13

Expert Comment

by:WizRd-Linux
ID: 22620402
Syslog should be listed as listening to port 514.

Restart syslog and check /var/log/messages or /var/log/syslog to see if any errors are reported about binding to the port, or specific interfaces.
0
 
LVL 1

Author Comment

by:amigan_99
ID: 22620477
[root@netmon etc]# /etc/rc.d/init.d/syslog restart
Shutting down kernel logger:                               [  OK  ]
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]

From /sys/log/messages:
Oct  1 17:53:45 netmon kernel: Kernel logging (proc) stopped.
Oct  1 17:53:45 netmon kernel: Kernel log daemon terminating.
Oct  1 17:53:46 netmon syslog: klogd shutdown succeeded
Oct  1 17:53:46 netmon exiting on signal 15
Oct  1 17:53:47 netmon syslogd 1.4.1: restart.
Oct  1 17:53:47 netmon syslog: syslogd startup succeeded
Oct  1 17:53:47 netmon syslog: klogd startup succeeded
Oct  1 17:53:47 netmon kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct  1 17:53:47 netmon syslog: syslogd shutdown succeeded

[root@netmon log]# ps -ef | grep syslogd
root      4368     1  0 17:53 ?        00:00:00 syslogd -m 0

[root@netmon etc]# netstat -l | grep udp
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*

Nothing in the router log file:
[root@netmon log]# ls -l /var/log/cisco/routers/router
-rw-rw-rw-  1 root root 0 Oct  1 04:02 /var/log/cisco/routers/router




0
 
LVL 13

Accepted Solution

by:
WizRd-Linux earned 2000 total points
ID: 22620541
Syslog doesn't have the -r option... which means it can't receive messages from remote hosts, including routers.

Edit the file /etc/sysconfig/syslog and modify the line SYSLOGD_OPTIONS="-m 0" to be SYSLOGD_OPTIONS="-m 0 -r"

Sorry I didn't notice this earlier.
0
 
LVL 13

Expert Comment

by:WizRd-Linux
ID: 22620547
and then restart syslog with /etc/init.d/syslog restart (service syslog restart)
0
 
LVL 1

Author Closing Comment

by:amigan_99
ID: 31502214
All kneel to the WizRd!  Woot!

[root@netmon routers]# ls -l
total 311000
-rw-rw-rw-  1 root root   103197 Oct  1 21:01 router
-rw-r--r--  1 root root  7899873 Sep 22 04:02 router.10.gz
-rw-r--r--  1 root root  8161644 Sep 21 04:02 router.11.gz
-rw-r--r--  1 root root 10608622 Sep 20 04:02 router.12.gz
0

Featured Post

Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Google Drive is extremely cheap offsite storage, and it's even possible to get extra storage for free for two years.  You can use the free account 15GB, and if you have an Android device..when you install Google Drive for the first time it will give…
This article explains how to use the rsync command to create backups and sync data across hosts. Rsync is a very useful command that is often used to copy data, make backups, migrate hosts, and bridge the gap between site staging and production envi…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses
Course of the Month8 days, 1 hour left to enroll

616 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question