Link to home
Start Free TrialLog in
Avatar of bradclif
bradclif

asked on

Apache Slowdown and Piped Logs

I currently have websites on a Windows Server 2003 box, served by Apache 2.0.59 and Tomcat 5.0.28 for dynamic content.  Most of our servers are not open to the internet and experience fairly low load.  We have been noticing tremendous slowdowns in serving requests and intermittent loss of logging.  We currently rotate logs with rotatelogs.exe that ships with Apache.

Here's a sample of how I've been setting up virtual hosts

    ServerName app_name
    CustomLog "|bin/rotatelogs.exe logs/app_name-access-%Y%m.log 86400" common env=!dont_log
    ErrorLog "|bin/rotatelogs.exe logs/app_name-error-%Y%m.log 86400"
    JkLogFile "|bin/rotatelogs.exe logs/app_name-modjk-%Y%m.log 86400"
    <Location "/*">
        JkMount app_name
    </Location>

One of our servers has 8 virtual hosts.  I see 50 rotatelogs processes associated with the parent and child processes and many more rotatelogs processes not associated with Apache any more.  I've checked, they are orphaned processes that weren't killed when Apache restarted some time in the past.

After a while the server starts to slow down to the point where it effectively stops serving requests.  A restart of the Apache HTTPD service solves this problem, at least temporarily.  The Apache HTTPD service needed to be restarted 5 times in the last 3 business days.

Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

Can anything be done to reduce the number of orphaned or zombie rotatelogs processes?  From my reading, it doesn't appear possible.

Can mod_log_rotate or any other Apache module rotate the Error or mod_jk logs?

I can provide some additional configuration information.
ASKER CERTIFIED SOLUTION
Avatar of Arty K
Arty K
Flag of Kazakhstan image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of bradclif
bradclif

ASKER

Thank you for the reply.  I very well may need to rotate the logs manually, but it will be a hard sell.  Though I see a correlation, I cannot prove a causal relationship between the piping process and the Apache problems we have been having.  Piping has been part of the process here before I started working with the servers.  Of course we've always had problems with slow downs, even under WebSphere.

Unfortunately, I've never been able to recreate the problems we have on that one production server.  Any solution I suggest, manual or mod_log_rotate, would receive its true test when implemented.  So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.

I still had a few questions.

> >Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

>Yes, if they eat memory.

The thing I find odd is that they consume memory, but they only took up about 5% of the available memory, no CPU time, and other server processes were unaffected.  Could there  be another mechanism at work?

> > So If you can, do not use logrotate and pipe feature at all. May be a better solution would be stopping the server at the off-load hours (may be once a week), copying logs and starting server again. It takes less then a second, but your webserver performance will always be the same.

mod_log_rotate is supposed to run in-process.  If this would be more reliable than piping to an external process, I have a feeling it would be better received than a strictly manual process.  I've tested it, and it does run in the Apache process.  Have you had any experience with mod_log_rotate (http://www.hexten.net/wiki/index.php/Mod-log-rotate)?
> So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.

That's simple, just turn off logrotate and see what happens. I also tend to think there is another problem (may be some kind of resource leak). I guess performance and debugging is more impotent then rotating logs, that can be temporary stopped.

> Could there  be another mechanism at work?
Hardly, but they also may lock files.

> Have you had any experience with mod_log_rotate
No.
Forced accept.

Computer101
EE Admin