Solved

Syslog is broken

Posted on 2008-10-01
8
825 Views
Last Modified: 2013-12-16
Someone booted the host and now syslog's not working.  Would appreciate trouble-shooting tips.

ps -ef | grep syslog
root      3954     1  0 15:50 ?        00:00:00 syslogd -m 0

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

tcpdump output:
15:59:50.179709 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.180685 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469254 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
0
Comment
Question by:amigan_99
  • 4
  • 4
8 Comments
 
LVL 13

Expert Comment

by:WizRd-Linux
Comment Utility
Iptables hasn't changed?
If you restart syslog, does it begin working?
Does 'netstat -al' show syslog listening?
0
 
LVL 1

Author Comment

by:amigan_99
Comment Utility
Update: So syslog itself is working somewhat in that boot.log is working, /var/log/messages, /var/log/mail..

But it's the syslog traffic that can't find its way to facility 6.  But that appears set fine in /etc/syslog.conf.

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

As I showed before - the cisco router syslog traffic arrives at the interface of the linux host.  But for some reason the linux host (netmon)  sends back "udp port syslog unreachable".  

15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
0
 
LVL 1

Author Comment

by:amigan_99
Comment Utility
I don't see syslog port 514 in play..

[root@netmon etc]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 *:832                       *:*                         LISTEN
tcp        0      0 *:mysql                     *:*                         LISTEN
tcp        0      0 *:sunrpc                    *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:ipp       *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:smtp      *:*                         LISTEN
tcp        0      0 *:8000                      *:*                         LISTEN
tcp        0      0 *:http                      *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 *:https                     *:*                         LISTEN
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     6134   /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     8331   /tmp/.gdm_socket
unix  2      [ ACC ]     STREAM     LISTENING     8450   /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     5937   /dev/gpmctl
unix  2      [ ACC ]     STREAM     LISTENING     5415   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     5726   /var/lib/mysql/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     6211   /var/run/dbus/system_bus_socket
0
 
LVL 13

Expert Comment

by:WizRd-Linux
Comment Utility
Syslog should be listed as listening to port 514.

Restart syslog and check /var/log/messages or /var/log/syslog to see if any errors are reported about binding to the port, or specific interfaces.
0
Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

 
LVL 1

Author Comment

by:amigan_99
Comment Utility
[root@netmon etc]# /etc/rc.d/init.d/syslog restart
Shutting down kernel logger:                               [  OK  ]
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]

From /sys/log/messages:
Oct  1 17:53:45 netmon kernel: Kernel logging (proc) stopped.
Oct  1 17:53:45 netmon kernel: Kernel log daemon terminating.
Oct  1 17:53:46 netmon syslog: klogd shutdown succeeded
Oct  1 17:53:46 netmon exiting on signal 15
Oct  1 17:53:47 netmon syslogd 1.4.1: restart.
Oct  1 17:53:47 netmon syslog: syslogd startup succeeded
Oct  1 17:53:47 netmon syslog: klogd startup succeeded
Oct  1 17:53:47 netmon kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct  1 17:53:47 netmon syslog: syslogd shutdown succeeded

[root@netmon log]# ps -ef | grep syslogd
root      4368     1  0 17:53 ?        00:00:00 syslogd -m 0

[root@netmon etc]# netstat -l | grep udp
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*

Nothing in the router log file:
[root@netmon log]# ls -l /var/log/cisco/routers/router
-rw-rw-rw-  1 root root 0 Oct  1 04:02 /var/log/cisco/routers/router




0
 
LVL 13

Accepted Solution

by:
WizRd-Linux earned 500 total points
Comment Utility
Syslog doesn't have the -r option... which means it can't receive messages from remote hosts, including routers.

Edit the file /etc/sysconfig/syslog and modify the line SYSLOGD_OPTIONS="-m 0" to be SYSLOGD_OPTIONS="-m 0 -r"

Sorry I didn't notice this earlier.
0
 
LVL 13

Expert Comment

by:WizRd-Linux
Comment Utility
and then restart syslog with /etc/init.d/syslog restart (service syslog restart)
0
 
LVL 1

Author Closing Comment

by:amigan_99
Comment Utility
All kneel to the WizRd!  Woot!

[root@netmon routers]# ls -l
total 311000
-rw-rw-rw-  1 root root   103197 Oct  1 21:01 router
-rw-r--r--  1 root root  7899873 Sep 22 04:02 router.10.gz
-rw-r--r--  1 root root  8161644 Sep 21 04:02 router.11.gz
-rw-r--r--  1 root root 10608622 Sep 20 04:02 router.12.gz
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now