Solved

Syslog is broken

Posted on 2008-10-01
8
840 Views
Last Modified: 2013-12-16
Someone booted the host and now syslog's not working.  Would appreciate trouble-shooting tips.

ps -ef | grep syslog
root      3954     1  0 15:50 ?        00:00:00 syslogd -m 0

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

tcpdump output:
15:59:50.179709 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.180685 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469254 IP c3845-inet1.pcmt.local.52931 > netmon.pooched.local.syslog: UDP, length 142
15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
0
Comment
Question by:amigan_99
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 4
8 Comments
 
LVL 13

Expert Comment

by:WizRd-Linux
ID: 22620098
Iptables hasn't changed?
If you restart syslog, does it begin working?
Does 'netstat -al' show syslog listening?
0
 
LVL 1

Author Comment

by:amigan_99
ID: 22620162
Update: So syslog itself is working somewhat in that boot.log is working, /var/log/messages, /var/log/mail..

But it's the syslog traffic that can't find its way to facility 6.  But that appears set fine in /etc/syslog.conf.

/etc/syslog.conf snippet:
local6.warning                                          /var/log/cisco/routers/router
local6.debug                                            /var/log/cisco/routers/router
local6.info                                             /var/log/cisco/routers/router

As I showed before - the cisco router syslog traffic arrives at the interface of the linux host.  But for some reason the linux host (netmon)  sends back "udp port syslog unreachable".  

15:59:50.469283 IP netmon.pooched.local > c3845-inet1.pcmt.local: icmp 178: netmon.pooched.local udp port syslog unreachable
0
 
LVL 1

Author Comment

by:amigan_99
ID: 22620276
I don't see syslog port 514 in play..

[root@netmon etc]# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp        0      0 *:832                       *:*                         LISTEN
tcp        0      0 *:mysql                     *:*                         LISTEN
tcp        0      0 *:sunrpc                    *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:ipp       *:*                         LISTEN
tcp        0      0 netmon.pcmt.local:smtp      *:*                         LISTEN
tcp        0      0 *:8000                      *:*                         LISTEN
tcp        0      0 *:http                      *:*                         LISTEN
tcp        0      0 *:ssh                       *:*                         LISTEN
tcp        0      0 *:https                     *:*                         LISTEN
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  2      [ ACC ]     STREAM     LISTENING     6134   /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     8331   /tmp/.gdm_socket
unix  2      [ ACC ]     STREAM     LISTENING     8450   /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     5937   /dev/gpmctl
unix  2      [ ACC ]     STREAM     LISTENING     5415   /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     5726   /var/lib/mysql/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     6211   /var/run/dbus/system_bus_socket
0
Ransomware-A Revenue Bonanza for Service Providers

Ransomware – malware that gets on your customers’ computers, encrypts their data, and extorts a hefty ransom for the decryption keys – is a surging new threat.  The purpose of this eBook is to educate the reader about ransomware attacks.

 
LVL 13

Expert Comment

by:WizRd-Linux
ID: 22620402
Syslog should be listed as listening to port 514.

Restart syslog and check /var/log/messages or /var/log/syslog to see if any errors are reported about binding to the port, or specific interfaces.
0
 
LVL 1

Author Comment

by:amigan_99
ID: 22620477
[root@netmon etc]# /etc/rc.d/init.d/syslog restart
Shutting down kernel logger:                               [  OK  ]
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]
Starting kernel logger:                                    [  OK  ]

From /sys/log/messages:
Oct  1 17:53:45 netmon kernel: Kernel logging (proc) stopped.
Oct  1 17:53:45 netmon kernel: Kernel log daemon terminating.
Oct  1 17:53:46 netmon syslog: klogd shutdown succeeded
Oct  1 17:53:46 netmon exiting on signal 15
Oct  1 17:53:47 netmon syslogd 1.4.1: restart.
Oct  1 17:53:47 netmon syslog: syslogd startup succeeded
Oct  1 17:53:47 netmon syslog: klogd startup succeeded
Oct  1 17:53:47 netmon kernel: klogd 1.4.1, log source = /proc/kmsg started.
Oct  1 17:53:47 netmon syslog: syslogd shutdown succeeded

[root@netmon log]# ps -ef | grep syslogd
root      4368     1  0 17:53 ?        00:00:00 syslogd -m 0

[root@netmon etc]# netstat -l | grep udp
udp        0      0 *:826                       *:*
udp        0      0 *:829                       *:*
udp        0      0 *:tftp                      *:*
udp        0      0 *:sunrpc                    *:*
udp        0      0 *:ipp                       *:*

Nothing in the router log file:
[root@netmon log]# ls -l /var/log/cisco/routers/router
-rw-rw-rw-  1 root root 0 Oct  1 04:02 /var/log/cisco/routers/router




0
 
LVL 13

Accepted Solution

by:
WizRd-Linux earned 500 total points
ID: 22620541
Syslog doesn't have the -r option... which means it can't receive messages from remote hosts, including routers.

Edit the file /etc/sysconfig/syslog and modify the line SYSLOGD_OPTIONS="-m 0" to be SYSLOGD_OPTIONS="-m 0 -r"

Sorry I didn't notice this earlier.
0
 
LVL 13

Expert Comment

by:WizRd-Linux
ID: 22620547
and then restart syslog with /etc/init.d/syslog restart (service syslog restart)
0
 
LVL 1

Author Closing Comment

by:amigan_99
ID: 31502214
All kneel to the WizRd!  Woot!

[root@netmon routers]# ls -l
total 311000
-rw-rw-rw-  1 root root   103197 Oct  1 21:01 router
-rw-r--r--  1 root root  7899873 Sep 22 04:02 router.10.gz
-rw-r--r--  1 root root  8161644 Sep 21 04:02 router.11.gz
-rw-r--r--  1 root root 10608622 Sep 20 04:02 router.12.gz
0

Featured Post

WordPress Tutorial 1: Installation & Setup

WordPress is a very popular option for running your web site and can be used to get your content online quickly for the world to see. This guide will walk you through installing the WordPress server software and the initial setup process.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
If you're a developer or IT admin, you’re probably tasked with managing multiple websites, servers, applications, and levels of security on a daily basis. While this can be extremely time consuming, it can also be frustrating when systems aren't wor…
Suggested Courses

631 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question