[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Apache Slowdown and Piped Logs

Posted on 2007-07-24
5
Medium Priority
?
2,184 Views
Last Modified: 2008-01-14
I currently have websites on a Windows Server 2003 box, served by Apache 2.0.59 and Tomcat 5.0.28 for dynamic content.  Most of our servers are not open to the internet and experience fairly low load.  We have been noticing tremendous slowdowns in serving requests and intermittent loss of logging.  We currently rotate logs with rotatelogs.exe that ships with Apache.

Here's a sample of how I've been setting up virtual hosts

    ServerName app_name
    CustomLog "|bin/rotatelogs.exe logs/app_name-access-%Y%m.log 86400" common env=!dont_log
    ErrorLog "|bin/rotatelogs.exe logs/app_name-error-%Y%m.log 86400"
    JkLogFile "|bin/rotatelogs.exe logs/app_name-modjk-%Y%m.log 86400"
    <Location "/*">
        JkMount app_name
    </Location>

One of our servers has 8 virtual hosts.  I see 50 rotatelogs processes associated with the parent and child processes and many more rotatelogs processes not associated with Apache any more.  I've checked, they are orphaned processes that weren't killed when Apache restarted some time in the past.

After a while the server starts to slow down to the point where it effectively stops serving requests.  A restart of the Apache HTTPD service solves this problem, at least temporarily.  The Apache HTTPD service needed to be restarted 5 times in the last 3 business days.

Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

Can anything be done to reduce the number of orphaned or zombie rotatelogs processes?  From my reading, it doesn't appear possible.

Can mod_log_rotate or any other Apache module rotate the Error or mod_jk logs?

I can provide some additional configuration information.
0
Comment
Question by:bradclif
  • 2
4 Comments
 
LVL 27

Accepted Solution

by:
Nopius earned 2000 total points
ID: 19562578
> Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

Yes, if they eat memory.

> Can anything be done to reduce the number of orphaned or zombie rotatelogs processes?  From my reading, it doesn't appear possible.

I wouldn't recommend you to use any kind of piped logging (including rotatelog.exe). That really impacts Apache performance (not only dead instances).
 
As we may read on Apache's site http://httpd.apache.org/docs/2.2/logs.html :
"As with conditional logging, piped logs are a very powerful tool, but they should not be used where a simpler solution like off-line post-processing is available."

> I've checked, they are orphaned processes that weren't killed when Apache restarted some time in the past.

They say "Apache will start the piped-log process when the server starts, and will restart it if it crashes while the server is running. (This last feature is why we can refer to this technique as "reliable piped logging".)". So, Apache never kills that processes. While for Unix it's OK (they are killed by OS when apache restarts), for Windows, behavior is quite different, as you can see.

So If you can, do not use logrotate and pipe feature at all. May be a better solution would be stopping the server at the off-load hours (may be once a week), copying logs and starting server again. It takes less then a second, but your webserver performance will always be the same.

0
 

Author Comment

by:bradclif
ID: 19564264
Thank you for the reply.  I very well may need to rotate the logs manually, but it will be a hard sell.  Though I see a correlation, I cannot prove a causal relationship between the piping process and the Apache problems we have been having.  Piping has been part of the process here before I started working with the servers.  Of course we've always had problems with slow downs, even under WebSphere.

Unfortunately, I've never been able to recreate the problems we have on that one production server.  Any solution I suggest, manual or mod_log_rotate, would receive its true test when implemented.  So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.

I still had a few questions.

> >Can the orphaned rotatelogs processes still affect the Apache HTTPD service?

>Yes, if they eat memory.

The thing I find odd is that they consume memory, but they only took up about 5% of the available memory, no CPU time, and other server processes were unaffected.  Could there  be another mechanism at work?

> > So If you can, do not use logrotate and pipe feature at all. May be a better solution would be stopping the server at the off-load hours (may be once a week), copying logs and starting server again. It takes less then a second, but your webserver performance will always be the same.

mod_log_rotate is supposed to run in-process.  If this would be more reliable than piping to an external process, I have a feeling it would be better received than a strictly manual process.  I've tested it, and it does run in the Apache process.  Have you had any experience with mod_log_rotate (http://www.hexten.net/wiki/index.php/Mod-log-rotate)?
0
 
LVL 27

Expert Comment

by:Nopius
ID: 19565402
> So I'm trying to gather as much information as I can before moving forward just in case there's an issue unrelated to logging involved here.

That's simple, just turn off logrotate and see what happens. I also tend to think there is another problem (may be some kind of resource leak). I guess performance and debugging is more impotent then rotating logs, that can be temporary stopped.

> Could there  be another mechanism at work?
Hardly, but they also may lock files.

> Have you had any experience with mod_log_rotate
No.
0
 
LVL 1

Expert Comment

by:Computer101
ID: 20657548
Forced accept.

Computer101
EE Admin
0

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article is intended for those who are new to PHP error handling (https://www.experts-exchange.com/articles/11769/And-by-the-way-I-am-New-to-PHP.html).  It addresses one of the most common problems that plague beginning PHP develop…
What You Need to Know when Searching for a Webhost Provider
Look below the covers at a subform control , and the form that is inside it. Explore properties and see how easy it is to aggregate, get statistics, and synchronize results for your data. A Microsoft Access subform is used to show relevant calcul…
Whether it be Exchange Server Crash Issues, Dirty Shutdown Errors or Failed to mount error, Stellar Phoenix Mailbox Exchange Recovery has always got your back. With the help of its easy to understand user interface and 3 simple steps recovery proced…

834 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question