Avatar of wbens
wbens asked on

How Does a Client Send Logs to a Remote Rsyslog Server?

Hello Experts:  I need to configure an rsyslog server where 1) all the clients send all their log files to it, and 2) where all the clients also send the auth.log files separate to an rsyslog server.  


I have part 1 working, but I cannot get part 2 working.  One of the requirements is that on the rsyslog server, the clients need to have their hostnames and not their FQDN.  


I have really run out of ideas on how to complete this task at work.


This is the part that is working, which is part 1:


 ##RULES##
$LocalHostname CLIENT1NAME

*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window


The configuration above produces what I want.  However, when I try to add this configuration:


 ##RULES##
$LocalHostname CLIENT1NAME

$Ruleset auth_test
auth,authpriv.* /log/$LocalHostname/auth.log
auth,authpriv.* @172.31.80.166:514
auth,authpriv.* @@172.31.80.166:514

$ModLoad imtcp
$InputTCPServerBindRuleset auth_test
$InputTCPServerRun 514

*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window


This configuration is not working.  I need the auth and authpriv logs to go to the remote rsyslog server as /logs/CLIENT1NAME/YEAR/MONTH/DAY/auth.log, and all others in /logs/CLIENT1NAME/YEAR/MONTH/DAY/


This is an example of what is working:


[root@ip-172-31-80-166 logs]# ls -l CLIENT1NAME/2021/09/21/
total 32
-rw-r--r--. 1 root root  425 Sep 21 17:57 dbus-daemon.log
-rw-r--r--. 1 root root  147 Sep 21 17:57 NetworkManager.log
-rw-r--r--. 1 root root  484 Sep 21 17:33 rsyslogd.log
-rw-r--r--. 1 root root  422 Sep 21 17:57 sshd.log
-rw-r--r--. 1 root root   96 Sep 21 17:33 sudo.log
-rw-r--r--. 1 root root 5982 Sep 21 17:55 systemd.log
-rw-r--r--. 1 root root  352 Sep 21 17:57 systemd-logind.log
[root@ip-172-31-80-166 logs]#

Open in new window


The ryslog server is the one with IP 172.31.80.166.


This is the rsyslog version:



[root@ip-172-31-80-166 logs]# rsyslogd -v
rsyslogd 8.24.0-57.el7_9.1, compiled with:
        PLATFORM:                               x86_64-redhat-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.
[root@ip-172-31-80-166 logs]#

Open in new window


This is the rsyslog server configuration in /etc/rsyslog.d/rsyslog_custom.conf:


$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

$DirCreateMode 0755
$FileCreateMode 0644
$umask 0022

$PrivDropToUser root
$PrivDropToGroup root
$PreserveFQDN off

$template TmplMsg, "/logs/%HOSTNAME:::uppercase%/%$YEAR%/%$MONTH%/%$DAY%/%PROGRAMNAME%.log
*.* ?TmplMsg


Open in new window

The rsyslog server is running on Red Hat 7.9.


Note:  As this is a work in progress, its requirements might change.  So far, this is what my manager has requested from me in this assignment.  


Thanks!

--Willie

Linux* rsyslogd

Avatar of undefined
Last Comment
arnold

8/22/2022 - Mon
SOLUTION
arnold

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
ASKER
wbens

Hello arnold:  I think this is what you suggested:

##RULES##
$LocalHostname ANSIBLECLIENT1

#auth,authpriv.* /log/auth_test.log

auth,authpriv.* @172.31.80.166:514
auth,authpriv.* @@172.31.80.166:514

*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window


However, it does not work. It does not even create the /log/HOSTNAME directory.  Sending all the logs to the remote server is working fine for me. I need to find a way to send the auth and authpriv logs independently from this configuration:

*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window



Thanks!
--Willie
arnold

Yes, usually, you do not need to provide a port, (:514) redirecting to a remote syslogger is through @remoteip the port is assumed to be 514 on the remote side you have to be receiving/accepting port 514 through the software firewall on the system if active as well as the local rsyslog/syslog has to be configured to listen on port 514

on the remote host confirm it is listening on port 514.
lsod -i:514 to see if you have a syslog/rsyslog listening.

Rules in the syslog.conf/rsyslog.conf are not exclusive.

You can and should have one entry saved locally to a file and one that routes the same event out to a central repository.
ASKER
wbens

Yes! It is listening on port 514:

tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN      -
tcp6       0      0 :::514                  :::*                    LISTEN      -
udp        0      0 0.0.0.0:514             0.0.0.0:*                           -
udp6       0      0 :::514                  :::*                                -

Open in new window


I have made a few changes to the rsyslog-server config file:

$ModLoad imudp
$UDPServerRun 514


$ModLoad imtcp
$InputTCPServerRun 514


$DirCreateMode 0755
$FileCreateMode 0644
$umask 0022

$PrivDropToUser root
$PrivDropToGroup root
$PreserveFQDN off


$template TmplMsg, "/logs/%HOSTNAME:::uppercase%/%$YEAR%/%$MONTH%/%$DAY%/%PROGRAMNAME%.log
*.* ?TmplMsg


if $programname == 'auth' then {
        action(type="omfile" file="/logs/%HOSTNAME:::uppercase%/%$YEAR%/%$MONTH%/%$DAY%/authTest.log")
}


if $programname == 'authpriv' then {
        action(type="omfile" file="/logs/%HOSTNAME:::uppercase%/%$YEAR%/%$MONTH%/%$DAY%/authTest.log")
}

Open in new window


However, this is not doing what I need.

Please, help!

Thanks!
--Willie
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
arnold

what OS are you running on?
Double check whether your have a local running firewall (iptables, firewalld, sfw) and whether the firewall allows traffic to pass through.

on another system since you have rsyslog configured for both TCP and UDP connection on port 514
try
telenet remotesyslogIP 514
does the connection establish?
Do you get
Trying remotesyslogIP_address...
telnet: connect to address remotesyslogIP_address: Connection refused

Do you get
Trying remotesyslogIP_address...

or do you get
Trying remotesyslogIP_address...
Connected to remotesyslogIP.
Escape character is '^]'.


Only the last one confirms the connection established all prior do not. The first one you should not get, because you confirmed it is listening on all interface (0.0.0.0) means all networks on the system.

The second means it is hitting something like a firewall that it is not getting a aresponse of disposition of the connection.
ASKER
wbens

The client and the server are Red Hat Enterprise Linux Server release 7.9 (Maipo).  

The client can connec to the server.  I used telnet to confirm that.  Also, when I use this client configuration:

##RULES##
$LocalHostname CLIENT1


*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window


I get logs sent from the client to the rsyslog server.  It is when I try to isolate the auth and authpriv logs that things break.

Thanks!
--Willie

ASKER CERTIFIED SOLUTION
arnold

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
See how we're fighting big data
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
ASKER
wbens

I tried this, but it did not work:

module(load="imfile" PollingInterval="5")
$LocalHostname ANSIBLECLIENT1
input(type="imfile"


      File="/var/log/messages"


      Tag="auth"


      Severity="*"


      Facility="local7")


local7.* @172.31.80.166:514
local7.* @@172.31.80.166:514

Open in new window


I have tried the other ones since 2 or three hours ago, but nothing is working. This is what works:
1) Sending all files over to the remote rsyslog server.
2) Sending the auth and authpriv files over to the remote rsyslog server.

However, when I try sending both together, it does not work. I need to send all the files, which works with this configuration on the client:

*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window


However, when I try sending all the logs, and the auth and authpriv logs separate, but at the same time, it does not work. I tried something like this one the client:

auth,authpriv.* @172.31.80.166:514
auth,authpriv.* @@172.31.80.166:514


*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window


It did not work.

I have also tried filtering the auth and authpriv logs on the server side with this code and similar:

if $programname == 'auth' then {         
action(type="omfile" file="/logs/%HOSTNAME:::uppercase%/%$YEAR%/%$MONTH%/%$DAY%/authTest.log") }

Open in new window


That has not worked either.


Thanks for sticking around, arnold!
--Willie

Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
arnold

IT does not send logs, it sends/forwards the events the local syslog receives.
Since you added a rule to the Local7 facility

try this
on the local
logger -p local7.err -t "testing stuff"  "this is a test"
see if it forwards to the remote.

What does the remote have that deals with Local7 if you do not have a handler defined, the data is ignored.

on the main rsyslog you could try using tcpdump to capture traffic comming from one of these forwarders to port 514.

and see if you see the traffic coming to your server.

Point just because you are forwarding events, if you do not have the proper handlers on the receiving syslog/rsyslog it might be
ASKER
wbens

I ran the command as you indicated on the client:

 logger -p local7.err -t "testing stuff"  "this is a test"

Open in new window


with a configuration file on the client as /etc/rsysd.d/auth_logs.conf:

module(load="imfile" PollingInterval="5")
$LocalHostname CLIENT1
input(type="imfile"

      File="/var/log/authtest.log"
      Tag="auth"
      Severity="*"
      Facility="local7")


local7.* @172.31.80.166:514
local7.* @@172.31.80.166:514

Open in new window


Nothing showed up on the rsyslog server.

Thanks!
--Willie
arnold

The setting
On the receiving server what is your configuration for local7.*?


Can you run tcpdump to see whether you have outgoing traffic to port 514 of the remote syslog server ip
On the local system.

The *.*[tab]@remoteserverip

Commonly would be in the rsyslog.conf and not in a subordinate location.

Once you can confirm the thing is being sent out

The issue to look at whether you are restricting outgoing traffic, oath to the remotesyslog, and firewall on the remote syslog blocking.


Can you run on the central server
iptables -L -t filter --line-numbers
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
ASKER
wbens

arnold:

I think we have this working.  After so much troubleshooting, and following your suggestions, my conclusion is that this is the configuration on the client as you indicated with this link:

https://support.hostway.com/hc/en-us/articles/115001717450-Configure-rsyslog-client-for-remote-logging-on-CentOS 

So, on the rsyslog client, I have:

 ##RULES##
$LocalHostname CLIENT1


#auth,authpriv.* /log/auth_test.log


auth,authpriv.* @172.31.80.166:514
auth,authpriv.* @@172.31.80.166:514


*.* @172.31.80.166:514
*.* @@172.31.80.166:514

Open in new window


After that, I just had to create a template in the rsyslog server with a path that is different than the one with all of other events. So, on the server, I have:

$ModLoad imudp
$UDPServerRun 514

$ModLoad imtcp
$InputTCPServerRun 514

$DirCreateMode 0755
$FileCreateMode 0644
$umask 0022

$PrivDropToUser root
$PrivDropToGroup root
$PreserveFQDN off

$template TmplMsg, "/logs/%HOSTNAME:::uppercase%/%$YEAR%/%$MONTH%/%$DAY%/%PROGRAMNAME%.log"
*.* ?TmplMsg

$template TmplAuth, "/logs/auth.d/%HOSTNAME:::uppercase%/%$YEAR%/%$MONTH%/%$DAY%/%PROGRAMNAME%.log"
auth,authpriv.* ?TmplAuth      

Open in new window


I am getting all the events in this path on the rsyslog server:
/logs/CLIENT1/2021/09/23

Open in new window


Then, I am getting all the auth and authpriv events in this path on the rsyslog server:

[root@ip-172-31-80-166 23]# cd /logs/auth.d/CLIENT1/2021/09/23/
[root@ip-172-31-80-166 23]# ls -l
total 16
-rw-r--r--. 1 root root    0 Sep 23 00:31 polkitd.log
-rw-r--r--. 1 root root 1728 Sep 23 00:34 sshd.log
-rw-r--r--. 1 root root 2099 Sep 23 00:36 sudo.log
-rw-r--r--. 1 root root 2864 Sep 23 00:36 su.log
-rw-r--r--. 1 root root    0 Sep 23 00:34 systemd.log
-rw-r--r--. 1 root root 1764 Sep 23 00:34 systemd-logind.log
[root@ip-172-31-80-166 23]#

Open in new window


I think this is a good solution.
I need to go back to my manager and talk with him about this.  I need to know if this is what he is looking for. If it is not, at least, I am in a better position to complete this assignment.

Probably, I will need to ask for more help, but for now, this is great!

Thanks!
--Willie

arnold

Glad you got further along.

usually when you send facility you should always have the priority auth.* if you want all to go.

I think, forwarding should be done from the main config file versus from the subordinate client as it will complicate things if you or someone else has to track things down.
or define a subordinate remote.conf

You can use this information and actually push it directly into a DB, mysql, mariadb
or triggering proactive notification if based on criteria.