• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2253
  • Last Modified:

Apache Slowdown and Piped Logs

I currently have websites on a Windows Server 2003 box, served by Apache 2.0.59 and Tomcat 5.0.28 for dynamic content.  Most of our servers are not open to the internet and experience fairly low load.  We have been noticing tremendous slowdowns in serving requests and intermittent loss of logging.  We currently rotate logs with rotatelogs.exe that ships with Apache.

Here's a sample of how I've been setting up virtual hosts

    ServerName app_name
    CustomLog "|bin/rotatelogs.exe logs/app_name-access-%Y%m.log 86400" common env=!dont_log
    ErrorLog "|bin/rotatelogs.exe logs/app_name-error-%Y%m.log 86400"
    JkLogFile "|bin/rotatelogs.exe logs/app_name-modjk-%Y%m.log 86400"
    <Location "/*">
        JkMount app_name
    </Location>

One of our servers has 8 virtual hosts.  I see 50 rotatelogs processes associated with the parent and child processes and many more rotatelogs processes not associated with Apache any more.  I've checked, they are orphaned processes that weren't killed when Apache restarted some time in the past.

After a while the server starts to slow down to the point where it effectively stops serving requests.  A restart of the Apache HTTPD service solves this problem, at least temporarily.  The Apache HTTPD service needed to be restarted 5 times in the last 3 business days.

Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

Can anything be done to reduce the number of orphaned or zombie rotatelogs processes?  From my reading, it doesn't appear possible.

Can mod_log_rotate or any other Apache module rotate the Error or mod_jk logs?

I can provide some additional configuration information.
0
bradclif
Asked:
bradclif
  • 2
1 Solution
 
NopiusCommented:
> Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

Yes, if they eat memory.

> Can anything be done to reduce the number of orphaned or zombie rotatelogs processes?  From my reading, it doesn't appear possible.

I wouldn't recommend you to use any kind of piped logging (including rotatelog.exe). That really impacts Apache performance (not only dead instances).
 
As we may read on Apache's site http://httpd.apache.org/docs/2.2/logs.html :
"As with conditional logging, piped logs are a very powerful tool, but they should not be used where a simpler solution like off-line post-processing is available."

> I've checked, they are orphaned processes that weren't killed when Apache restarted some time in the past.

They say "Apache will start the piped-log process when the server starts, and will restart it if it crashes while the server is running. (This last feature is why we can refer to this technique as "reliable piped logging".)". So, Apache never kills that processes. While for Unix it's OK (they are killed by OS when apache restarts), for Windows, behavior is quite different, as you can see.

So If you can, do not use logrotate and pipe feature at all. May be a better solution would be stopping the server at the off-load hours (may be once a week), copying logs and starting server again. It takes less then a second, but your webserver performance will always be the same.

0
 
bradclifAuthor Commented:
Thank you for the reply.  I very well may need to rotate the logs manually, but it will be a hard sell.  Though I see a correlation, I cannot prove a causal relationship between the piping process and the Apache problems we have been having.  Piping has been part of the process here before I started working with the servers.  Of course we've always had problems with slow downs, even under WebSphere.

Unfortunately, I've never been able to recreate the problems we have on that one production server.  Any solution I suggest, manual or mod_log_rotate, would receive its true test when implemented.  So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.

I still had a few questions.

> >Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

>Yes, if they eat memory.

The thing I find odd is that they consume memory, but they only took up about 5% of the available memory, no CPU time, and other server processes were unaffected.  Could there  be another mechanism at work?

> > So If you can, do not use logrotate and pipe feature at all. May be a better solution would be stopping the server at the off-load hours (may be once a week), copying logs and starting server again. It takes less then a second, but your webserver performance will always be the same.

mod_log_rotate is supposed to run in-process.  If this would be more reliable than piping to an external process, I have a feeling it would be better received than a strictly manual process.  I've tested it, and it does run in the Apache process.  Have you had any experience with mod_log_rotate (http://www.hexten.net/wiki/index.php/Mod-log-rotate)?
0
 
NopiusCommented:
> So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.

That's simple, just turn off logrotate and see what happens. I also tend to think there is another problem (may be some kind of resource leak). I guess performance and debugging is more impotent then rotating logs, that can be temporary stopped.

> Could there  be another mechanism at work?
Hardly, but they also may lock files.

> Have you had any experience with mod_log_rotate
No.
0
 
Computer101Commented:
Forced accept.

Computer101
EE Admin
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Microsoft Exchange Server

The MCTS: Microsoft Exchange Server 2010 certification validates your skills in supporting the maintenance and administration of the Exchange servers in an enterprise environment. Learn everything you need to know with this course.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now