bradclif
asked on
Apache Slowdown and Piped Logs
I currently have websites on a Windows Server 2003 box, served by Apache 2.0.59 and Tomcat 5.0.28 for dynamic content. Most of our servers are not open to the internet and experience fairly low load. We have been noticing tremendous slowdowns in serving requests and intermittent loss of logging. We currently rotate logs with rotatelogs.exe that ships with Apache.
Here's a sample of how I've been setting up virtual hosts
ServerName app_name
CustomLog "|bin/rotatelogs.exe logs/app_name-access-%Y%m. log 86400" common env=!dont_log
ErrorLog "|bin/rotatelogs.exe logs/app_name-error-%Y%m.l og 86400"
JkLogFile "|bin/rotatelogs.exe logs/app_name-modjk-%Y%m.l og 86400"
<Location "/*">
JkMount app_name
</Location>
One of our servers has 8 virtual hosts. I see 50 rotatelogs processes associated with the parent and child processes and many more rotatelogs processes not associated with Apache any more. I've checked, they are orphaned processes that weren't killed when Apache restarted some time in the past.
After a while the server starts to slow down to the point where it effectively stops serving requests. A restart of the Apache HTTPD service solves this problem, at least temporarily. The Apache HTTPD service needed to be restarted 5 times in the last 3 business days.
Can the orphaned rotatelogs processes still affect the Apache HTTPD service?
Can anything be done to reduce the number of orphaned or zombie rotatelogs processes? From my reading, it doesn't appear possible.
Can mod_log_rotate or any other Apache module rotate the Error or mod_jk logs?
I can provide some additional configuration information.
Here's a sample of how I've been setting up virtual hosts
ServerName app_name
CustomLog "|bin/rotatelogs.exe logs/app_name-access-%Y%m.
ErrorLog "|bin/rotatelogs.exe logs/app_name-error-%Y%m.l
JkLogFile "|bin/rotatelogs.exe logs/app_name-modjk-%Y%m.l
<Location "/*">
JkMount app_name
</Location>
One of our servers has 8 virtual hosts. I see 50 rotatelogs processes associated with the parent and child processes and many more rotatelogs processes not associated with Apache any more. I've checked, they are orphaned processes that weren't killed when Apache restarted some time in the past.
After a while the server starts to slow down to the point where it effectively stops serving requests. A restart of the Apache HTTPD service solves this problem, at least temporarily. The Apache HTTPD service needed to be restarted 5 times in the last 3 business days.
Can the orphaned rotatelogs processes still affect the Apache HTTPD service?
Can anything be done to reduce the number of orphaned or zombie rotatelogs processes? From my reading, it doesn't appear possible.
Can mod_log_rotate or any other Apache module rotate the Error or mod_jk logs?
I can provide some additional configuration information.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
> So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.
That's simple, just turn off logrotate and see what happens. I also tend to think there is another problem (may be some kind of resource leak). I guess performance and debugging is more impotent then rotating logs, that can be temporary stopped.
> Could there be another mechanism at work?
Hardly, but they also may lock files.
> Have you had any experience with mod_log_rotate
No.
That's simple, just turn off logrotate and see what happens. I also tend to think there is another problem (may be some kind of resource leak). I guess performance and debugging is more impotent then rotating logs, that can be temporary stopped.
> Could there be another mechanism at work?
Hardly, but they also may lock files.
> Have you had any experience with mod_log_rotate
No.
Forced accept.
Computer101
EE Admin
Computer101
EE Admin
ASKER
Unfortunately, I've never been able to recreate the problems we have on that one production server. Any solution I suggest, manual or mod_log_rotate, would receive its true test when implemented. So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.
I still had a few questions.
> >Can the orphaned rotatelogs processes still affect the Apache HTTPD service?
>Yes, if they eat memory.
The thing I find odd is that they consume memory, but they only took up about 5% of the available memory, no CPU time, and other server processes were unaffected. Could there be another mechanism at work?
> > So If you can, do not use logrotate and pipe feature at all. May be a better solution would be stopping the server at the off-load hours (may be once a week), copying logs and starting server again. It takes less then a second, but your webserver performance will always be the same.
mod_log_rotate is supposed to run in-process. If this would be more reliable than piping to an external process, I have a feeling it would be better received than a strictly manual process. I've tested it, and it does run in the Apache process. Have you had any experience with mod_log_rotate (http://www.hexten.net/wiki/index.php/Mod-log-rotate)?