syslog is not releasing disk-space

Your issue is common when syslog is not issued a reload sighup notice.
When syslog starts it attaches to file using a filehandle, the disk space to which this process writes remains until syslog gets notice to detach and attach to a new file.


Look at /etc/logrotate.conf /etc/logrotate.d/syslog

which system are you on? Svcadm

Lsof is a tool you can use resource
Lsof /var/log...

lsof is not there on server. pfiles can help ?

bash-3.2# cat /etc/logadm.conf | grep -v "#"
/var/cron/log -c -s 512k -t /var/cron/olog
/var/lp/logs/lpsched -C 2 -N -t '$file.$N'
/var/fm/fmd/errlog -M '/usr/sbin/fmadm -q rotate errlog && mv /var/fm/fmd/errlog.0- $nfile' -N -s 2m
/var/fm/fmd/fltlog -A 6m -M '/usr/sbin/fmadm -q rotate fltlog && mv /var/fm/fmd/fltlog.0- $nfile' -N -s 10m
smf_logs -C 8 -c -s 1m /var/svc/log/*.log
/var/adm/pacct -C 0 -N -a '/usr/lib/acct/accton pacct' -g adm -m 664 -o adm -p never
/var/log/pool/poold -N -a 'pkill -HUP poold; true' -s 512k
/var/adm/messages -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -z 0
/var/adm/dskmessages -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -p 1d -z 1
/var/log/authlog -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -g root -m 0640 -o root -z 0
/var/log/maillog -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -z 0
/var/log/locallog -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -z 0
/var/adm/wtmpx -A 1y -p 1m -z 0
/var/adm/sulog -A 1y -g root -m 0640 -o root -p 1m -z 0
bash-3.2#
bash-3.2# find /proc -type f -links 0 -ls | sort -n +6 | tail
find: stat() error /proc/11835/fd/43: No such file or directory
find: stat() error /proc/11820/fd/43: No such file or directory
find: stat() error /proc/11829/fd/43: No such file or directory
find: stat() error /proc/11821/fd/43: No such file or directory
find: stat() error /proc/11827/fd/43: No such file or directory
find: stat() error /proc/11825/fd/43: No such file or directory
find: stat() error /proc/11833/fd/43: No such file or directory
find: stat() error /proc/11826/fd/43: No such file or directory
find: stat() error /proc/11832/fd/43: No such file or directory
4213633025    4 -rw-------   0 root     root         2048 Apr 12 13:23 /proc/1222/fd/6
4213638852    4 -rw-------   0 root     root         2048 May 27 21:09 /proc/1222/fd/4
45240   73 -r--r--r--   0 adm      adm         73656 Apr 13 00:36 /proc/2140/fd/3
48364 48425 --w-------   0 root     root     49486355 Jun 21 18:37 /proc/3065/fd/6
48364 48425 --w-------   0 root     root     49486355 Jun 21 18:37 /proc/3065/fd/8
bash-3.2# pfiles 3065
3065:   /usr/sbin/syslogd
  Current rlimit: 65536 file descriptors
   0: S_IFDIR mode:0755 dev:181,65546 ino:5361 uid:0 gid:0 size:24
      O_RDONLY
      /
   1: S_IFDIR mode:0755 dev:181,65546 ino:5361 uid:0 gid:0 size:24
      O_RDONLY
      /
   2: S_IFDIR mode:0755 dev:181,65546 ino:5361 uid:0 gid:0 size:24
      O_RDONLY
      /
   3: S_IFDOOR mode:0444 dev:308,0 ino:30 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[1461]
      /var/run/name_service_door
   4: S_IFCHR mode:0000 dev:310,4 ino:50124 uid:0 gid:0 rdev:41,81
      O_RDWR
      /dev/udp
   5: S_IFCHR mode:0620 dev:310,4 ino:49206 uid:0 gid:7 rdev:164,7
      O_WRONLY|O_APPEND|O_NOCTTY|O_LARGEFILE
      /dev/zconsole
   6: S_IFREG mode:0644 dev:181,65546 ino:48364 uid:0 gid:0 size:49684790
      O_WRONLY|O_APPEND|O_NOCTTY|O_LARGEFILE
   7: S_IFCHR mode:0620 dev:310,4 ino:49206 uid:0 gid:7 rdev:164,7
      O_WRONLY|O_APPEND|O_NOCTTY|O_LARGEFILE
      /dev/zconsole
   8: S_IFREG mode:0644 dev:181,65546 ino:48364 uid:0 gid:0 size:49684790
      O_WRONLY|O_APPEND|O_NOCTTY|O_LARGEFILE
   9: S_IFCHR mode:0000 dev:310,4 ino:49348 uid:0 gid:0 rdev:21,88
      O_RDONLY
      /dev/log
  10: S_IFDOOR mode:0777 dev:307,0 ino:0 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC  door to syslogd[3065]
bash-3.2# svcs -a | grep -i syslog
online         16:41:47 svc:/site/system/syslog-ng:default
bash-3.2# svcprop svc:/site/system/syslog-ng:default
syslog-ng/config_file astring /etc/syslog-ng.conf
syslog-ng/executable astring /usr/local/sbin/syslog-ng
syslog-ng/pid_file astring /var/run/syslog-ng.pid
general/enabled boolean true
general/entity_stability astring Unstable
tm_common_name/C ustring Syslog-NG\ Server
tm_man_syslog-ng/manpath astring /usr/local/share/man
tm_man_syslog-ng/section astring 8
tm_man_syslog-ng/title astring syslog-ng
syslog-multi-user/entities fmri svc:/milestone/multi-user
syslog-multi-user/grouping astring require_all
syslog-multi-user/restart_on astring none
syslog-multi-user/type astring service
syslog_network/entities fmri svc:/milestone/network
syslog_network/grouping astring require_all
syslog_network/restart_on astring none
syslog_network/type astring service
dependents/syslog-multi-user-server fmri svc:/milestone/multi-user-server
start/exec astring /usr/local/lib/svc/method/syslog-ng
start/timeout_seconds count 60
start/type astring method
stop/exec astring :kill
stop/timeout_seconds count 60
stop/type astring method
refresh/exec astring :kill\ -HUP
refresh/timeout_seconds count 60
refresh/type astring method
restarter/logfile astring /var/svc/log/site-system-syslog-ng:default.log
restarter/contract count 60954
restarter/start_pid count 23874
restarter/start_method_timestamp time 1529624507.398667000
restarter/start_method_waitstatus integer 0
restarter/auxiliary_state astring none
restarter/next_state astring none
restarter/state astring online
restarter/state_timestamp time 1529624507.401978000
restarter_actions/restart integer
bash-3.2#

Select allOpen in new window

Issues with outputting the process from a pid file is that it might not be the right PID.
i.e. the process crashed and the pid file was not updated but the syslog was restarted by the monitoring.
cat /var/run/syslog-ng.pid
ps -ef | grep syslog

are the two providing/referencing the same PID?
kill -HUP <pid from the second line> and the space gets released immediately.

instead of writing into the /proc/fd, issue a HUP signal to the syslog process and it should release the filehandle.

Double check syslog.conf and the /var/log/
Partitioning such that /var/log has its own mount point often helps limit this issue.

I suspect that you have a log that was not setup to be rotated due to overisight.
it is much easier to find a large file

ls -li /proc/3065/fd/6 /proc/3065/fd/8
then use the inode to locate the file name

find / -inum <inode>
This way you can setup logadm to manage these logfiles as well.

Here is output

bash-3.2# cat /var/run/syslog-ng.pid
23881
bash-3.2# ps -ef | grep syslog
    root  3065  1140   0   Apr 12 ?         648:07 /usr/sbin/syslogd
    root  4518  4513   0 21:11:44 pts/9       0:00 grep syslog
    root 23881 23880   0 16:41:47 ?           2:14 /usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng.conf --pidfile /var/run/sysl
    root 23880  1140   0 16:41:47 ?           0:00 /usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng.conf --pidfile /var/run/sysl
bash-3.2# ls -li /proc/3065/fd/6 /proc/3065/fd/8
     48364 --w-------   0 root     root     115845590 Jun 21 21:12 /proc/3065/fd/6
     48364 --w-------   0 root     root     115845590 Jun 21 21:12 /proc/3065/fd/8
bash-3.2# find / -inum 48364
find: stat() error /proc/11835/fd/43: No such file or directory
/proc/3065/fd/6
/proc/3065/fd/8
find: stat() error /proc/11820/fd/43: No such file or directory
find: stat() error /proc/11829/fd/43: No such file or directory
find: stat() error /proc/11821/fd/43: No such file or directory
find: stat() error /proc/11827/fd/43: No such file or directory
find: stat() error /proc/11825/fd/43: No such file or directory
find: stat() error /proc/11833/fd/43: No such file or directory
find: stat() error /proc/11826/fd/43: No such file or directory
find: stat() error /proc/11832/fd/43: No such file or directory
/usr/share/gnome/javahelp/evolution-1.4/zh_HK/advancedinbox-108.html
bash-3.2#

Select allOpen in new window

kill -HUP <PID> can release it, but I have it do it every couple of days. Instead, I thought, ideally it should not hold this space

You seem have two syslog instances running, PID 3065 which is what consumed the space and the 23881 which logadm issues the reset to.

Determine which files are managed by syslog and make sure if they are under logadm management, that the correct PID is sent the HUP signal.

Note both large file/space consuming were under the 3065 PID /usr/sbin/syslogd

Here is syslog-ng.conf, which shows 5 logs and all are mentioned in logadm.conf. But, HUP signal is always going to cat /var/run/syslog-ng.pid, which is 23881. How should tell it to sent signal to 3065 PID as well ? I thought, 3065 is main PID of syslog.

bash-3.2# cat /etc/syslog-ng.conf
@version:3.0

options {
    flush_lines (1);
    time_reopen (10);
    log_fifo_size (1000);
    long_hostnames (off);
    use_dns (no);
    use_fqdn (no);
    keep_hostname (yes);
    stats_freq (0);
    mark_freq (600);
};

source s_sys {
    internal();
    sun-streams ("/dev/log" door("/etc/.syslog_door"));
    udp(ip("127.0.0.1") );
};

destination d_mesg   { file("/var/adm/messages"   perm(0644)); };
destination d_dskmesg { file("/var/adm/dskmessages" perm(0644)); };
destination d_auth   { file("/var/log/authlog"    perm(0640)); };
destination d_local  { file("/var/log/locallog"   perm(0644)); };
destination d_mail   { file("/var/log/maillog"    perm(0644)); };

filter f_auth   { facility(auth) and level(info..emerg); };
filter f_dsk     { facility(local0); };
filter f_local  { facility(local1,local2,local3,local4,local5,local6,local7); };
filter f_mail   { facility(mail) and level(info..emerg); };
filter f_mesg   {
        (facility(kern)          and level(debug..emerg))  or
        (facility(daemon,syslog) and level(info..emerg)) or
        (facility(cron,user)     and level(warning..emerg));
};

log { source(s_sys); filter(f_auth);  destination(d_auth);   };
log { source(s_sys); filter(f_mail);  destination(d_mail);   };
log { source(s_sys); filter(f_dsk);    destination(d_dskmesg); };
log { source(s_sys); filter(f_local); destination(d_local);  };
log { source(s_sys); filter(f_mesg);  destination(d_mesg);   };

filter f_remote { level(info..emerg); };
destination d_remote { tcp("dududududu.dududu.com" port(5142)); };
log { source(s_sys); filter(f_remote); destination(d_remote); };

bash-3.2# cat /etc/logadm.conf | grep -v "#"
/var/cron/log -c -s 512k -t /var/cron/olog
/var/lp/logs/lpsched -C 2 -N -t '$file.$N'
/var/fm/fmd/errlog -M '/usr/sbin/fmadm -q rotate errlog && mv /var/fm/fmd/errlog.0- $nfile' -N -s 2m
/var/fm/fmd/fltlog -A 6m -M '/usr/sbin/fmadm -q rotate fltlog && mv /var/fm/fmd/fltlog.0- $nfile' -N -s 10m
smf_logs -C 8 -c -s 1m /var/svc/log/*.log
/var/adm/pacct -C 0 -N -a '/usr/lib/acct/accton pacct' -g adm -m 664 -o adm -p never
/var/log/pool/poold -N -a 'pkill -HUP poold; true' -s 512k
/var/adm/messages -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -z 0
/var/adm/dskmessages -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -p 1d -z 1
/var/log/authlog -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -g root -m 0640 -o root -z 0
/var/log/maillog -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -z 0
/var/log/locallog -C 8 -a 'kill -HUP `cat /var/run/syslog-ng.pid`' -z 0
/var/adm/wtmpx -A 1y -p 1m -z 0
/var/adm/sulog -A 1y -g root -m 0640 -o root -p 1m -z 0
bash-3.2# cat /var/run/syslog-ng.pid
23881
bash-3.2#
bash-3.2# ps -ef | grep syslog
    root  3065  1140   0   Apr 12 ?         648:29 /usr/sbin/syslogd
    root 10593  9441   0 21:51:16 pts/13      0:00 grep syslog
    root 23881 23880   0 16:41:47 ?           2:35 /usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng.conf --pidfile /var/run/sysl
    root 23880  1140   0 16:41:47 ?           0:00 /usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng.conf --pidfile /var/run/sysl
bash-3.2#

Select allOpen in new window

your issue is not with syslog-ng but with the included syslogd from Solaris /etc/syslog.conf

run
fuser /var/log/*

This will identify the log files under /var/log/ to which there are active processes connected.
What you are looking for is which file is being accessed/managed by 3065
then compare your logadm.conf to make sure the file being rotated is sending the HUP notice to the correct process.

Commonly /var/adm/messages is managed by syslogd while you are sending HUP request when this log rotates to syslog-ng.

try kill -HUP 3065 and then rerun the fuser command.

Looking at the config /etc/syslog.conf would likely be enough.

I tried HUP to 3065 and still it seems to be bind with same files

bash-3.2# fuser /var/log/*
/var/log/VRTSpbx:
/var/log/Xorg.0.log:
/var/log/Xorg.0.log.old:
/var/log/authlog:    23881o
/var/log/authlog.0.gz:
/var/log/authlog.1.gz:
/var/log/authlog.2.gz:
/var/log/authlog.3.gz:
/var/log/authlog.4.gz:
/var/log/authlog.5.gz:
/var/log/authlog.6.gz:
/var/log/authlog.7.gz:
/var/log/locallog:
/var/log/maillog:
/var/log/maillog.0.gz:
/var/log/maillog.1.gz:
/var/log/maillog.2.gz:
/var/log/maillog.3.gz:
/var/log/maillog.4.gz:
/var/log/maillog.5.gz:
/var/log/maillog.6.gz:
/var/log/maillog.7.gz:
/var/log/snmpd.log:     3114o
/var/log/swupas:
/var/log/sysidconfig.log:
/var/log/syslog:
/var/log/tdr:
/var/log/webconsole:
bash-3.2#
bash-3.2#
bash-3.2# kill -HUP 3065
bash-3.2# fuser /var/log/*
/var/log/VRTSpbx:
/var/log/Xorg.0.log:
/var/log/Xorg.0.log.old:
/var/log/authlog:    23881o
/var/log/authlog.0.gz:
/var/log/authlog.1.gz:
/var/log/authlog.2.gz:
/var/log/authlog.3.gz:
/var/log/authlog.4.gz:
/var/log/authlog.5.gz:
/var/log/authlog.6.gz:
/var/log/authlog.7.gz:
/var/log/locallog:
/var/log/maillog:
/var/log/maillog.0.gz:
/var/log/maillog.1.gz:
/var/log/maillog.2.gz:
/var/log/maillog.3.gz:
/var/log/maillog.4.gz:
/var/log/maillog.5.gz:
/var/log/maillog.6.gz:
/var/log/maillog.7.gz:
/var/log/snmpd.log:     3114o
/var/log/swupas:
/var/log/sysidconfig.log:
/var/log/syslog:
/var/log/tdr:
/var/log/webconsole:
bash-3.2#
bash-3.2# cat /etc/syslog.conf | grep -v "#"
*.err;kern.notice;auth.notice                   /dev/sysmsg
*.err;kern.debug;daemon.notice;mail.crit        /var/adm/messages

*.alert;kern.err;daemon.err                     operator
*.alert                                         root

*.emerg                                         *


mail.debug                      ifdef(`LOGHOST', /var/log/syslog, @loghost)

ifdef(`LOGHOST', ,
user.err                                        /dev/sysmsg
user.err                                        /var/adm/messages
user.alert                                      `root, operator'
user.emerg                                      *
)
bash-3.2#

Select allOpen in new window

3065 is not bound to any log files in /var/log

Try the same fuser /var/adm/* potentially thus is syslogd but your Logadm directs notification to syslog-ng when messages are rotated.

Ran on /var/adm/* and here it shows -

bash-3.2# fuser /var/adm/*
/var/adm/acct:
/var/adm/aculog:
/var/adm/exacct:
/var/adm/lastlog:
/var/adm/log:
/var/adm/messages:    23881o    3065o
/var/adm/messages.0.gz:
/var/adm/messages.1.gz:
/var/adm/messages.2.gz:
/var/adm/messages.3.gz:
/var/adm/messages.4.gz:
/var/adm/messages.5.gz:
/var/adm/messages.6.gz:
/var/adm/messages.7.gz:
/var/adm/sa:
/var/adm/sm.bin:
/var/adm/streams:
/var/adm/sulog:
/var/adm/sulog.0.gz:
/var/adm/sulog.1.gz:
/var/adm/sulog.10.gz:
/var/adm/sulog.2.gz:
/var/adm/sulog.3.gz:
/var/adm/sulog.4.gz:
/var/adm/sulog.5.gz:
/var/adm/sulog.6.gz:
/var/adm/sulog.7.gz:
/var/adm/sulog.8.gz:
/var/adm/sulog.9.gz:
/var/adm/utmpx:
/var/adm/wtmpx:
/var/adm/wtmpx.0.gz:
/var/adm/wtmpx.1.gz:
/var/adm/wtmpx.10.gz:
/var/adm/wtmpx.11.gz:
/var/adm/wtmpx.2.gz:
/var/adm/wtmpx.3.gz:
/var/adm/wtmpx.4.gz:
/var/adm/wtmpx.5.gz:
/var/adm/wtmpx.6.gz:
/var/adm/wtmpx.7.gz:
/var/adm/wtmpx.8.gz:
/var/adm/wtmpx.9.gz:
/var/adm/dskmessages:    23881o
/var/adm/dskmessages.0:
/var/adm/dskmessages.1.gz:
/var/adm/dskmessages.2.gz:
/var/adm/dskmessages.3.gz:
/var/adm/dskmessages.4.gz:
/var/adm/dskmessages.5.gz:
/var/adm/dskmessages.6.gz:
/var/adm/dskmessages.7.gz:
bash-3.2#
bash-3.2# kill -HUP 3065
bash-3.2#
bash-3.2# fuser /var/adm/*
/var/adm/acct:
/var/adm/aculog:
/var/adm/exacct:
/var/adm/lastlog:
/var/adm/log:
/var/adm/messages:    23881o    3065o
/var/adm/messages.0.gz:
/var/adm/messages.1.gz:
/var/adm/messages.2.gz:
/var/adm/messages.3.gz:
/var/adm/messages.4.gz:
/var/adm/messages.5.gz:
/var/adm/messages.6.gz:
/var/adm/messages.7.gz:
/var/adm/sa:
/var/adm/sm.bin:
/var/adm/streams:
/var/adm/sulog:
/var/adm/sulog.0.gz:
/var/adm/sulog.1.gz:
/var/adm/sulog.10.gz:
/var/adm/sulog.2.gz:
/var/adm/sulog.3.gz:
/var/adm/sulog.4.gz:
/var/adm/sulog.5.gz:
/var/adm/sulog.6.gz:
/var/adm/sulog.7.gz:
/var/adm/sulog.8.gz:
/var/adm/sulog.9.gz:
/var/adm/utmpx:
/var/adm/wtmpx:
/var/adm/wtmpx.0.gz:
/var/adm/wtmpx.1.gz:
/var/adm/wtmpx.10.gz:
/var/adm/wtmpx.11.gz:
/var/adm/wtmpx.2.gz:
/var/adm/wtmpx.3.gz:
/var/adm/wtmpx.4.gz:
/var/adm/wtmpx.5.gz:
/var/adm/wtmpx.6.gz:
/var/adm/wtmpx.7.gz:
/var/adm/wtmpx.8.gz:
/var/adm/wtmpx.9.gz:
/var/adm/dskmessages:    23881o
/var/adm/dskmessages.0:
/var/adm/dskmessages.1.gz:
/var/adm/dskmessages.2.gz:
/var/adm/dskmessages.3.gz:
/var/adm/dskmessages.4.gz:
/var/adm/dskmessages.5.gz:
/var/adm/dskmessages.6.gz:
/var/adm/dskmessages.7.gz:
bash-3.2#

Select allOpen in new window

Here is what happens, you gave two loggers appending data into /var/adm/messages
When the size of the file meets your loagadm settings, messages is rolled to messages.0.gz
But the filehandle that 3065 and 23881 shared gets released by 23881 when it gets the HUP signal that syslog did never does.

Why are you running two instances of sysloger?

If I am comparing with other servers, each application is having its own syslog. But here, it is just one file - /etc/syslog-ng.conf. Not sure, if there should be two processes.
Looks like syslog is being used for OS related logs, while syslog-ng is for application logs (dskmessages, etc.)
Also, ps putput is truncating so I am assuming like this -

bash-3.2# ps -ef | grep syslog
    root  3065  1140   0   Apr 12 ?         649:15 /usr/sbin/syslogd
    root 23881 23880   0 16:41:47 ?           3:20 /usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng.conf --pidfile /var/run/sysl
    root 23880  1140   0 16:41:47 ?           0:00 /usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng.conf --pidfile /var/run/sysl
    root 24006 19661   0 23:21:59 pts/1       0:00 grep syslog
bash-3.2# cat /var/run/syslog.pid
3065
bash-3.2# cat /var/run/syslog-ng.pid
23881
bash-3.2#

Select allOpen in new window

Arnold is right: your are running syslog (native solaris) and you are running syslog-ng

syslog-ng gets notified during rollover, syslogd isn't.   It makes no sense to run 2 syslog daemons.
As you have diled in some stuff (logadm) to work with syslog-ng then  it seems logical to disable & stop syslogd.