logrotate not compressing logs

Is this the only log that doesn't compress?

Something to check, make sure the "compresscmd" and "uncompresscmd" are defined in /etc/logrotate.conf

It was not, so I added:

compresscmd=/bin/gzip


Will this still work even if compress is turned of in logrotate.conf, or should I put it in /etc/logrotate.d/ciscologs ?

/etc/logrotate.conf is read first before the individual files in logrotate.d/.  Put the default things you want all the scripts to use in logrotate.conf and then specifics and overrides in the others.

So, enabling compress in /etc/logrotate.d/ciscologs will compress the files you reference there in your definition.  Other definitions lacking the compress qualifier will not compress (following the default in logrotate.conf)

Ok, I had compress in my /etc/logrotate.d/ciscologs file and it still did not compress the rotated logs.  I thought logrotate would use gzip by default.

I added this to my /etc/logrotate.d/ciscologs file, let's see if that help.

/var/log/cisco/*/*.log
{
     daily
     rotate 5000
     missingok
     compresscmd=/bin/gzip
     compressext=.gz
     compress
     sharedscripts
#    postrotate
#    endscript
}

Select allOpen in new window

Unfortunately there is not way to test this until it runs daily.

How logs are written?
If the file is open (like syslog does) you need to re-open it to compress.

So it is still not compressing files.

I do not think the file is open all the time, these are remote systems.

here is my /etc/logrotate.d/ciscologs file:

/var/log/cisco/*/*.log
{
     daily
     rotate 5000
     nodateext
     nocreate
     missingok
     notifempty
     compresscmd=/bin/gzip
     delaycompress
     compress
     sharedscripts
#    postrotate
#    endscript
}

Select allOpen in new window

and here is an example of one of the directories:

[root@logserver firewall-in]# ls -lrt
total 627968
-rw-------. 1 root root 115704880 Jul  4 23:59 firewall-in-07-04-2013.log-20130705
-rw-------. 1 root root   6449900 Jul  5 03:36 firewall-in-07-02-2013.log-20130704.gz
-rw-------. 1 root root  12349378 Jul  5 03:36 firewall-in-07-03-2013.log-20130704.gz
-rw-------. 1 root root 181189143 Jul  5 23:59 firewall-in-07-05-2013.log.1
-rw-------. 1 root root 136139178 Jul  6 23:59 firewall-in-07-06-2013.log.1
-rw-------. 1 root root 122355969 Jul  7 23:59 firewall-in-07-07-2013.log.1
-rw-------. 1 root root  68817641 Jul  8 08:52 firewall-in-07-08-2013.log

Select allOpen in new window

As you can see it compressed 2 of the files back on the 5th.

That is delaycompress directive.
I see that you actually rotate with (r) syslogd and two tools dont make friends.
You should make fixed name log and probably add your log file to system log rotation (which knows how to get along with rsyslogd)

@ghiest,
Which two tools do not get along?  I am using rsyslogd only.

rsyslogd is rotating your logs and renaming them which does not match your filter in the logrotate configuration file.  You'll need to change the rsyslog config to not rotate the log files if you want logrotate to handle this for you...  rsyslog and logrotate are conflicting.

Ahh ok...

So I told rsyslog to not add a date, which mean it will write to /var/log/ciscologs/hostname/hostname.log  (not rotated).

Is this what you guys mean?

I'm figuring that the use of wildcards, changing filenames, nocreate and delaycompress at the same time are causing you problems.

I'm speculating here, but what I figure has been happening is this:


To fix this, have rsyslog create the log file with no date.  Remove the nodateext from the logrotate rule so your rotated files have dates.  If rsyslog is set up to create new files, then simply reloading rsyslog should force it to create a new file and continue.  If rsyslog requires the creation of a file before writing to it (standard syslog would not log if the file didn't already exist, thus the create keyword in logrotate).  To have your rule reload syslog, you'll need to enable the postrotate/endscript and add the command to reload syslog.  The init script for syslog is typically an easy way to handle this.  On my system (OpenSuSE) it looks like this:

/var/log/network/router
{
    compress
    dateext
    rotate 99
    missingok
    notifempty
    create 640 root root
    sharedscripts
    postrotate
        /etc/init.d/syslog reload > /dev/null
    endscript
}  

Select allOpen in new window

If the script doesn't exist, you can do something like "killall -HUP rsyslogd".  You'll need to specify the path where killall lives on your system.  You could probably do away with the delaycompress command as well since the reloading of rsyslog will stop it writing to the rotated log.

as long as logrotate still says the files do not need to be rotated, the problem is not linked to compression.

i think giest had the right hunch, but i'm unsure about the conclusion. if different logs are created every day or if another tool rotates them, wether the names match or not (and they do if you have a debug), it won't work because logrotate will assume it is not supposed to do anything.

Ok, first let me say thank you to everyone.  You have been a great deal of help.

So now I have a situation where the compression was done fine, but since rsyslog did not know to change the filename (add date at the end) it needed to be restarted in order to write to the logs this morning.

So how can I tell logrotate to restart rsyslog?

I see this:
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true

and this:
/etc/init.d/syslog reload > /dev/null

Which is correct?

Also, since I am rotating several logs (using wildcards in conf file) would this restart rsyslog for every file?  I have about 30 different logs going to this system.

Sending the -HUP (hangup signal or signal 1) to syslog doesn't restart syslog.  SIGHUP sent to syslog tells syslog to close all open files, re-read the configuration file(s) and then continue.  That way when you move the old logs around, it knows to stop writing to the old ones and start on the new ones.

As to which is "correct", there are many ways to accomplish the goal.  Some are more correct than others.  Linux variants use different methods to start/stop services, so it's best to use the one recommended for your distribution.  gheist above mentions that RHEL6 documentation recommends the use of the "service" command:

service rsyslog reload

The service command is a wrapper for running the init scripts (which are often located in /etc/init.d/).  These scripts take a standard command and hide the specifics of starting/stopping/reloading along with prerequisite checks, setting environment variables and such like.

service rsyslog reload
is functionally equivalent to
/etc/init.d/rsyslog reload

Although it might be syslog rather than rsyslog as syslog is "generic" and rsyslog is a specific implementation, you'll have to look and see which is used in your case.

The command
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
is what the scripted commands above are doing.

rsyslog creates a process ID file (.pid) in the /var/run directory so that scripts wanting to send signals to a running process have a generic way of finding out what the process ID for it is.  The command above uses /bin/kill to send SIGHUP to the process ID identified in the syslogd.pid file (the backquote `` executes a command and sends the output to the command line)  The rest of the command redirects the error output to the null device and the true command at the end makes sure that no matter what happens, a success return value is left for logrotate to consume so it will happily keep running and not stop if the kill command doesn't work as expected.

By using the:

    
     sharedscripts
     postrotate
          -fill in your method here-
     endscript

Select allOpen in new window

Logrotate will take care of the syslog reloads as needed for rules including these commands

Thanks everybody, it is working as expected now.