Logrotate challenges

We have Oracle Linux Server release 7.4, with kernel version: 4.1.12-103.7.1.el7uek.x86_64

We have an application (Oracle RAC) that creates and/or adds to about 1,500 log files every day.  These log files are all in a single subdirectory of the Oracle executables.  Oracle does not offer a configuration option or parameter to direct these log files to a different volume (like a log volume, where we wouldn't risk filling up the executable volume).

I'm trying to use logrotate to help me manage these logs, but I haven't found a combination of parameters yet for logrotate that works reliably.  One of the complicating factors is the fact that these log files are produced by three different Linux user accounts (root, oracle and emagent).  These log files do all get created with same group name: oinstall.  Another complicating factor is the permissions in the parent directory: "rwxrwxr-x".  The Oracle software install creates this directory and sets these permissions.  Many of these log files are very small and get created with unique file names (that include a five or six digit number).  Some of these log files though keep the same (non-unique) name every day and can grow rather large (at least 10s of mbytes).

Here is my latest attempt at a config file (named: crs_logs.conf in: /etc/logrotate.d) for logrotate:

/u01/app/oracle/diag/crs/zvm186-g-qa-db3/crs/trace/* {
        su oracle oinstall
        create 0640 oracle oinstall
        copytruncate
        rotate 2
        daily
        dateyesterday
        dateext
        compress
        delaycompress
        maxage 4
}

I plan to add a "postrotate" command to this, to move files more than two days old out to a log volume, if I can get this basic logrotate script to work.

When I try this manually (with; "logrotate -f crs_logs.conf") it gives me lots of errors like these two:
error: error setting owner of /u01/app/oracle/diag/crs/zvm186-g-qa-db3/crs/trace/olsnodes_18754.trm-20171211 to uid 54320 and gid 54321: Operation not permitted
error: error setting owner of /u01/app/oracle/diag/crs/zvm186-g-qa-db3/crs/trace/olsnodes_19441.trc-20171211 to uid 54320 and gid 54321: Operation not permitted

But, it seems to do most of what I want when I run it manually.  Overnight though, when it runs automatically, it seems to do nothing, but it does update the: /var/lib/logrotate/logrote.status file each night with some details about every file in the log directory.

I'm an Oracle DBA, not a Linux expert.  It looks to me like logrotate should be able to help me with these files, but it is not working for me yet.
LVL 36
Mark GeerlingsDatabase AdministratorAsked:
Who is Participating?
 
David FavorConnect With a Mentor Linux/LXD/WordPress/Hosting SavantCommented:
Ah... I see what might be a fix... Change your su to...

su root root

Open in new window


This will allow logrotate to deal with log files owned by anyone.

Leave your stanza...

create 0640 oracle oinstall

Open in new window


So any newly created logs will be owned by Oracle, so Oracle can actually write to your log files.

See if this works for you.
0
 
Mark GeerlingsDatabase AdministratorAuthor Commented:
That appears to help.  At least when I try: "logrotate -d crs_logs.conf" it returns lots of lines like this:
considering log /u01/app/oracle/diag/crs/zvm186-g-qa-db3/crs/trace/osysmond.trm
  log does not need rotating (log has been rotated at 2017-12-12 10:26, that is not day ago yet)

Open in new window


Those lines look reasonable to me.

I just tried: "logrotate -f crs_logs.conf" and that finished without reporting errors.  (It took a minute or two.)

Then I checked the log directory, and I see some 0-byte files now and some with double or triple date extensions.  Here are a couple examples:
-rw-rw---- 1 oracle  oinstall 10486007 Dec 12 11:17 ohasd_oraagent_oracle_65.trc-20171211-20171211-20171211
-rw-rw---- 1 oracle  oinstall        0 Dec 12 11:17 ohasd_oraagent_oracle_65.trc-20171211-20171211

Open in new window


I wasn't expecting either of these things to happen.  I can probably clean these up with a postrotate command if necessary.
0
 
David FavorLinux/LXD/WordPress/Hosting SavantCommented:
Yep. You do require a post rotate.

Normally you'll do something like...

service oracle reload (or equivalent)

Open in new window


This will likely maintain current Oracle connections + simply reload your config + reopen your log files, which will mean opening the new log created daily by logrotate.

Be sure to check your specific version of Oracle docs, to ensure this works as expected.

Looks like you're close to having every thing work.

Congrats!
0
Cloud Class® Course: Amazon Web Services - Basic

Are you thinking about creating an Amazon Web Services account for your business? Not sure where to start? In this course you’ll get an overview of the history of AWS and take a tour of their user interface.

 
Mark GeerlingsDatabase AdministratorAuthor Commented:
No, I don't need to reload or restart any services (as far as I know).  The services stay running and they continue to write to log files in this directory. I assume the "copytruncate" option allows the services to keep writing to their existing log/trace files, after logrotate copies the contents to new files that include a date extension.

Do you know why logrotate doubled or tripled the date extension on some files?  I certainly did not expect that.  Maybe I need to change the trailing "/*" on the first line of my config file to: "/tr?", since all of the default log (or trace) files in this directory have extensions of: ".trc" or "trm".

I will also need to see what happens overnight when this runs automatically because in the past it often seemed to do nothing then, even though when I run it manually, it seems to do at least most of what I want it to do.
0
 
arnoldCommented:
To do what you want, you should use prerotate to move an older file that would be overwritten if exists in the log folder to another destination.
postrorate would potentially be too late if the rotation overwrites/deletes the file you would have moved/copied...... to a central log repository.
0
 
Mark GeerlingsDatabase AdministratorAuthor Commented:
Yes, I may need both prerotate and postrotate steps to manage:
1. moving files more than two days old out of the default directory to the secondary directory
2. deleting the tiny (less than 60 bytes) files more than a day old that have no useful information
3. purging files more than a month old from the secondary directory
4. deleting the 0-byte files that logrotate creates in the secondary directory

I'll try adding those steps in today and will then have to check tomorrow to see if they work as intended.

Now my /etc/logrotate.d/crs_logs.conf file looks like this:
/u01/app/oracle/diag/crs/zvm186-g-qa-db3/crs/trace/*.tr? {
        sharedscripts
        prerotate
                find /u01/app/oracle/diag/crs/zvm186-g-qa-db3/crs/trace -maxdepth 1 -type f  -name '*.tr?' -mmin +1440 -size -60c -delete
                find /u01/app/oracle/diag/crs/zvm186-g-qa-db3/crs/trace -maxdepth 1 -type f -mmin +2880 -exec mv {} /s10/crs_logs/node3 \;
        endscript
        su root root
        create 0640 oracle oinstall
        copytruncate
        rotate 2
        daily
        dateyesterday
        dateext
        compress
        delaycompress
        maxage 4
        postrotate
                find /s10/crs_logs/node3/* -maxdepth 1 -type f -mtime +30 -delete
                find /s10/crs_logs/node3/* -maxdepth 1 -type f -empty -delete
        endscript
}

Open in new window

0
 
Mark GeerlingsDatabase AdministratorAuthor Commented:
Changing the "su ..." line to:
su root root

seems to have this working as hoped now.  I don't understand why that was needed.  When I tested this manually by running logrotate -f, I was logged in as root, and my understanding is that the automatic overnight run of logrotate is also done as root.  But, as I indicated in the question, I'm not a Linux/Unix expert, just somewhat familiar with Linux now after managing Oracle databases running on Linux for the past 12 years.

After today, I'm heading out for a week of vacation, so I won't be able to follow up on this until after Christmas.  I'll open a new question then if necessary.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.