• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 675
  • Last Modified:

logrotate - logs are rotating, but apache won't write to new log

hi,

i'm using logrotate to rotate my apache logs, and the rotation works fine.
the problem is, after rotating, apache keeps writing to the old log.
i tried 3 different methods of apache restart (through logrotate script) but none helped.

example:
i have http.log, and it's rotating to http.log.1
apache keeps writing to http.log.1
and the new http.log stays empty.

the three methods i tried to restart apache with are:
        /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
        /usr/local/apache/bin/apachectl graceful
        /bin/kill -USR1 `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true

do i need to say that the restart should be quick/easy so the users will not feel it ?

thanks
gonen.
0
gonenra
Asked:
gonenra
  • 13
  • 12
  • 4
  • +2
1 Solution
 
periwinkleCommented:
Is it possible that you have multiple instances of Apache started?  I.e. try bringing down apache entirely - is there still apache instances running?

What does your full logrotate entry look like?

I have one that works ok that looks like: looks like:

/var/log/httpd/access_log /var/log/httpd/error_log {
        missingok
        sharedscripts
        postrotate
        /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
        endscript
        rotate 365
        compress
        notifempty
}

0
 
gonenraAuthor Commented:
thanks for replying

the script look like this:

var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
    endscript
}

(the other commands i tried, currently not in the script)

notice that it's working on al logs, and not specific one.

what is 'rotate 365' ?

does others like 'notifempty' have meaning to the location in the script ?

thanks again
0
 
periwinkleCommented:
notifempty means don't rotate if the log file is empty.  the rotate 365 means keep 365 log files.

Did you find out if you have other instances of Apache running?  Can you bring down all instances of Apache, and then restart it, to see if this will then work?
0
NFR key for Veeam Agent for Linux

Veeam is happy to provide a free NFR license for one year.  It allows for the non‑production use and valid for five workstations and two servers. Veeam Agent for Linux is a simple backup tool for your Linux installations, both on‑premises and in the public cloud.

 
sleep_furiouslyCommented:
How long does Apache keep writing to the old log file?  Graceful restart needs to keep writing to the old log until all the requests it was serving at the time of rotation have completed.

See:
http://httpd.apache.org/docs/2.0/logs.html#rotation
"By using a graceful restart, the server can be instructed to open new log files without losing any existing or pending connections from clients. However, in order to accomplish this, the server must continue to write to the old log files while it finishes serving old requests. It is therefore necessary to wait for some time after the restart before doing any processing on the log files."
0
 
gonenraAuthor Commented:
periwinkle, as for instances of Apache - we are talking about REDHAT EL4,
it's obvious that i have a few httpd all the time.
let's say i do kill all of them manualy and the i run logrotate - what am i going to do if i want to run that automatically ?
besides, killing all the httpd, will be the exact oposite solution because i need a way to make apache write to the new log file without interrupting surfers.

sleep_furiously, as for the time that apache keeps writing to the old log, will this ls help ?

ls -l /var/log/httpd/www3*
-rwxr-x---  1 root ramia        0 Apr  9 04:08 /var/log/httpd/www3diburim_log
-rwxr-x---  1 root ramia 13927103 Apr 10 12:23 /var/log/httpd/www3diburim_log.1
-rwxr-x---  1 root ramia  3096651 Mar 19 08:35 /var/log/httpd/www3diburim_log.4

as you can see, we are talking about something like 24 hrs

any ideas ?

thanks again
0
 
periwinkleCommented:
You misunderstand me.  I'm wondering if you have multiple parent instances of Apache, not if you have many children.  I'm suggesting for one time only that we shut down Apache manually, to see if there are still processes hanging around.  My theory is that you are not actually restarting the correct instance of apache -- you are restarting a different one that is writing someplace else.
0
 
gonenraAuthor Commented:
periwinkle, thanks for the quick reply,

when you say shut down apache, do you mean :
/etc/init.d/httpd stop ?

i did - top - and saw that i currently have about 5 httpd running
user of 4 of them is apache
the user of the fifth is root

i guess the one of root is the instance that's starts on boot

suppose i stop apache as mentiond above, and there are more instances ? whats next ?
suppose i stop apache as mentiond above, and there are no more instances ? whats next ?

thanks again for helping
0
 
periwinkleCommented:
If there are still processes, lyou can kill them off by brute force:

killall -KILL httpd

It sounds like there are rogues out there.

Then, start Apache properly using apachectl - and let's see if everything gets cleared up!
0
 
periwinkleCommented:
You should also look into /etc/init.d/httpd to see HOW it is stopping apache.  Is it using apachectl?
0
 
gonenraAuthor Commented:
excuse me but i still dont get that...
let's pretend /etc/init.d/httpd stop is killing all instances of httpd
how is that have to do with logrotate ? i'm not going to totaly stop and start my web server for the rotating logs.
i checked it on a VM and found that all instances are killed if i stop the httpd,
so what's next from here ?

btw, i can put here the httpd script from /etc/init.d/httpd if you'd like.


0
 
periwinkleCommented:
Did you, or did you not, have instances of httpd remaining when you issues /etc/init.d/httpd stop?

If you did, not all instances of httpd were shutdown - which means that you have multiple parent instances of apache running.  This is not normal, nor is it desirable!  You want to shut them all down and get back to the state of having only one parent instance of apache running.  There is only one PID (process identifier) in /var/run/httpd.pid, and that is the process that is being restarted.  If that is the incorrect process, then when you restart during logrotate, you'll get the symptoms that you are observing - that the old file is still being written to!

If you are in the abnormal state of having multiple parent processes running, you need to fix this ONCE by stopping all of apache processes, and starting it up as you normally would.  Then you will only have one parent process running, /var/run/httpd.pid will be correct, and log rotate will work properly.
0
 
gonenraAuthor Commented:
well, on httpd stop EVERY instance of httpd is being killed.
there is only ONE root httpd process
and FEW apache httpd processes

the problem still exists,

is there a possibility that the kill -HUP command works fine when called manualy (shell)
but not working fine when called automatically through logrotate script ?
0
 
gonenraAuthor Commented:
any ideas guys ?
i'm a new user here so i dont have so many points to offer
(trying to earn some)
0
 
periwinkleCommented:

If you do this from the command line:

cat /var/run/httpd.pid

You can find out what the process id is currently being reported as.

Are you able to find the httpd process with that ID when you do:

ps ax | grep -i processid

(where processid is the number that you got back from the httpd.pid file?)

0
 
gonenraAuthor Commented:
yep, i can find that pid

 2419 ?        Ss     0:02 /usr/sbin/httpd


i even did
ps aux
to find that the user of that httpd is root and not apache
meaning this is the parent instance
0
 
gonenraAuthor Commented:
anyone else has any thought ?
points goes up to 150
0
 
kenfcampCommented:
why don't you simply change "/bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true"
to " /bin/killall -HUP httpd"  It's worked for me
0
 
periwinkleCommented:
killall is simply a front end program to kill - it goes through and greps all the instances of a process, and kills them with the signal on the command line.  Although it seems to work for you, I'm not certain if it will properly handle the child processes;  basically, for apache to restart, it needs to restart the parent process.
0
 
kenfcampCommented:
killall -HUP httpd, will restart all "httpd" processess

:: man killall ::

       killall - kill processes by name
       killall sends a signal to all processes running any of the
       specified  commands.
0
 
periwinkleCommented:
I'm well aware of that - but you don't WANT to restart the child processes - only the parent process.  The parent process will start its own new child processes.
0
 
periwinkleCommented:
See:

http://httpd.apache.org/docs/1.3/stopping.html

which states in part:

"You will notice many httpd executables running on your system, but you should not send signals to any of them except the parent, whose pid is in the PidFile."

Incidentally, back to the question - this article lets us know that instead of using kill -HUP in your graceful restart script, you should use kill -USR1 instead, which is the graceful restart:

"The USR1 signal causes the parent process to advise the children to exit after their current request (or to exit immediately if they're not serving anything). The parent re-reads its configuration files and re-opens its log files. As each child dies off the parent replaces it with a child from the new generation of the configuration, which begins serving new requests immediately."

0
 
kenfcampCommented:
Hmmm

heh didn't think of that good point
0
 
gonenraAuthor Commented:
hi guys,

the USR1 was one of the option i used and didnt work for me.

i tried changing the > /dev/null to >> /anotherdir
so i might get any errors to help solve the problem, but nothing got from there,
anotherdir is empty.

is there any way the command isnt working due to permissions or something like that ?
restarting manualy - apache writes to new log
automaticaly - writes to old log
0
 
sleep_furiouslyCommented:
Is there anything that indicates to you that the postrotate scripts/commands are running at all?  Perhaps you could put a test command in that section to make sure they do actually run.

Also, logrotate doesn't like it if /tmp is secured to prevent execution (either mounted as noexec or ACL) so you might check that.
0
 
gonenraAuthor Commented:
sleep_furiously, thanks for replying,
i know that logrotate is running cause i get a new log with 0 bytes size.
the problem is apache is not wrting to that file after logrotate restart command. it keeps writing to the old log.

about /tmp i'll check it out but i think it wouldn't change anyway cause i'm not talking about logrotate not running...
0
 
sleep_furiouslyCommented:
I'm not talking about logrotate not running either, I'm just talking about the "postrotate" section of logrotate failing ...
0
 
sleep_furiouslyCommented:
Another random thought ... since 'cat' is not a built in shell command, is it possible that the necessary PATH is not present when cat command is run by logrotate, and thus the kill command is not getting the process number?

Maybe try:

        /bin/kill -HUP `/bin/cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true

or whatever the appropriate path to cat is on your system.

0
 
gonenraAuthor Commented:
well, about the /tmp and postrotate failing,
i did find this :
http://forum.ev1servers.net/showthread.php?t=61207
which says that there is a bug in logrotate-3.7.1-5.RHEL4
causing this exact problem
not sure how i'm gonna handle this

about cat not in PATH, i doubt it, but will give it a try. coudnt hurt.
0
 
periwinkleCommented:
AH!  That looks like good advice at the bottom - they are setting up a temporary logrotate tmp directory in that script near the bottom of that page - why not give it a try?
0
 
gonenraAuthor Commented:
about the new temp dir solution,
i wouldnt mind trying that,
but isnt /tmp and /var/tmp the same ? why should it work with /vat/tmp/logrotate
and not /tmp ?
0
 
periwinkleCommented:
I think it is really a workaround of a non-logical RedHat bug;  Konrad Frye basically explains it in the link you have given as redhat is reassured by the fact that you are redefining the TMPDIR variable with a directory that you have just created.
0
 
gonenraAuthor Commented:
hi,
first i find it a little odd that periwinkle is a Cleanup volunteer and decides to give to points to himself.
seems like a dirty way to get alot of expert points.

second, none of the above answers helped to fix the problem,
finally i added a cron job just before the awstat that restarts apache, and not through the logrotate script.
seems like the cause is really the bug, and for security reasons i didnt want to make /tmp executable or dealing with mkdir through the logrotate script.

there should be an option to close a question without an accepted answer,  cause at least in this case (and i guess that many other) none of the answers helped. (i found out about the bug myself).
0
 
GranModCommented:
PAQed with points refunded (150)

GranMod
Community Support Moderator
0

Featured Post

Upgrade your Question Security!

Add Premium security features to your question to ensure its privacy or anonymity. Learn more about your ability to control Question Security today.

  • 13
  • 12
  • 4
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now