We have a RedHat Linux 7.3 site running apache 1.3.27, with MaxClients=256. Occasionally have seen the number of httpd processes go to 256, stay like that for a few minutes, then back down to well under 100. During those few minutes some connections are not accepted. The site is running about 80-100 sites across about 500 virtual domains (most of which are unused). About 5 of the sites do 90% of the traffic.
We recompiled httpd to enable settingMaxClients higher, and set it to 512 (I know it is dangerous to do so, but we have 1GB RAM and thus took the risk). MaxRequestsPerChild = 200. KeepAlive is On, MaxKeepAliveRequests = 200, KeepAliveTimeout = 15.
MySQL connections was increased from 400 to 600 when we increased MaxClients to 512.
After all this we still see the same activity. Running a script to count httpd processes every 15 seconds. Here is one such episode. Note the sudden jump at 10:17:05, climb to MaxClient processes for a few minutes and then back down.
42 09/17/04 10:15:34
42 09/17/04 10:15:49
40 09/17/04 10:16:04
36 09/17/04 10:16:19
36 09/17/04 10:16:34
36 09/17/04 10:16:50
36 09/17/04 10:17:05
178 09/17/04 10:17:37
242 09/17/04 10:17:53
243 09/17/04 10:18:11
243 09/17/04 10:18:27
272 09/17/04 10:18:43
397 09/17/04 10:19:14
405 09/17/04 10:19:34
513 09/17/04 10:20:07
514 09/17/04 10:20:23
514 09/17/04 10:20:41
514 09/17/04 10:20:58
(... repeats every 15 seconds)
514 09/17/04 10:24:49
486 09/17/04 10:25:48
72 09/17/04 10:26:09
73 09/17/04 10:26:24
72 09/17/04 10:26:39
65 09/17/04 10:26:54
56 09/17/04 10:27:10
(Note, when it says 514 it's actually 512, we just use "ps aux | fgrep httpd | wc -l" which results in 2 extra counts).
We have looked at traffic logs on all sites and can't see anything that would cause this type of traffic, IF it is traffic that's the problem. But, most of our sites are PHP+MySQL based, so maybe something is going on with some web site's programming that would cause this behavior. Anyway, we see max of around 5000-10,000 hits (from all combined server logs) in a 10-minute interval. (I will verify this number again shortly). That doesn't seem like too much but, maybe it is?
Other details:
* Seems to happen about 2-3 times a day.
* Does seem to happen during times when you would expect more use, i.e., doesn't happen in early a.m. hours.
* None of the httpd processes, when maxed out, are in any sort of spin loop. They all have 0 to 5 seconds of CPU time.
* System load average stays under 2.
* When this happens we have looked carefully at top, memory, swap etc., and all seem fine. System is not thrashing or going to disk for pages while these MaxClients processes are working. It's just... "stuck" for a while, then spontaneously resolves and apache processes die off rapidly.
Whatever help you can provide to help us track through this problem would be appreciated. I will award points if I feel significant help is rendered even if it doesn't result in a solution per-se.
Thanks!
Andrew
Start Free Trial
