[x]
Posted via EE Mobile

Search, ask, and monitor your questions on the go with EE Mobile. Visit Experts Exchange from your mobile device and never be out of touch again.
Question
[x]
Attachment Details
[x]
The Solution Rating System

With so many solutions, how can you tell which solutions are most likely to help you and which ones are not? To provide you with a tool to use, we rate our solutions based on various elements that most accurately determine if a solution is a quality solution. To explain what factors affect the solution rating, here are the elements we take into consideration when formulating our solution rating.

  • The Grade of the Solution
  • The Zone Rank of the Expert Providing the Solution
  • The Number of Author and Expert Comments
  • The Number of Experts Contributing
  • The Feedback of the Community

Your Input Matters
Because of the way the system is set up, the most important variable in this equation is you. As a member of Experts Exchange, you are able to cast your vote on the quality of the solutions in regard to how complete, accurate, helpful and easy to understand each solution is. When you provide your feedback, each rating is adjusted accordingly. So, if you see a solution that has a poor rating that you think is a good solution, let us know by rating it. As you do, the rating will be adjusted and will become more accurate for other members of our site.

If you have any suggestions that you would like to make for our rating system, please ask a question in the Suggestions Zone of Community Support.

Thank you!
8.1

How Can I Determine the Root Cause of Apache Threads Maxing Out?

Asked by GoSox2008 in Apache Web Server

Tags: Apache HTTP Project, Apache, 2.0.48, Running on AIX

I've run in to a rather red-herring-like situation that I'm hoping some of you good folks can shed some light on.  First some background:

This Apache Web Server is being used as a front end for a J2EE application that accepts incoming connections from clients out in the field rather frequently.  Every 60 seconds or so the client will send a message to the J2EE application just to say "Hi, I'm here.".  There also can be remote interaction with these clients as well.

The actual issue I've been chasing down for months is that from the J2EE application's point of view, some clients (a random subset) will stop communicating with the J2EE application.  However viewing a client log from a client that is allegedly not reporting, it shows unknown host errors for the Apache Server.

Taking a thread count of the Apache threads yields that the threads have maxed out at 503.  I'm just getting a thread count with your standard "ps -ef | grep http | wc -l" command.   My results are below.
I've also placed an httpd.conf snippet below as well.

Lastly, we have the following compiled in Apache:

  core.c
  mod_log_config.c
  mod_setenvif.c
  mod_ssl.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_alias.c
  mod_so.c

Any help would be much, much appreciated.

Best Regards,
Josh

PS: I have changed all of the addresses in the netstat for security purposes, of course. 
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
43:
44:
45:
46:
47:
48:
49:
50:
51:
52:
53:
54:
55:
56:
57:
58:
59:
60:
61:
62:
63:
64:
65:
66:
67:
68:
69:
70:
71:
72:
73:
74:
75:
76:
77:
78:
79:
80:
81:
82:
83:
84:
85:
86:
87:
88:
89:
90:
91:
92:
93:
94:
95:
96:
97:
98:
99:
100:
101:
102:
103:
104:
105:
106:
107:
108:
109:
110:
111:
112:
113:
114:
115:
116:
117:
118:
119:
120:
121:
122:
123:
124:
125:
126:
127:
128:
129:
130:
131:
132:
133:
134:
135:
136:
137:
138:
139:
140:
141:
142:
143:
144:
145:
146:
147:
148:
149:
150:
151:
152:
153:
154:
155:
156:
157:
158:
159:
160:
161:
162:
163:
164:
165:
166:
167:
168:
169:
170:
171:
172:
173:
174:
175:
176:
177:
178:
179:
180:
181:
182:
183:
184:
185:
186:
187:
188:
189:
190:
191:
192:
193:
194:
195:
196:
197:
198:
199:
200:
201:
202:
203:
204:
205:
206:
207:
208:
209:
210:
211:
212:
213:
214:
215:
216:
217:
218:
219:
220:
221:
222:
223:
224:
225:
226:
227:
228:
229:
230:
231:
232:
233:
234:
235:
236:
237:
238:
239:
240:
241:
242:
243:
244:
245:
246:
247:
248:
249:
250:
251:
252:
253:
254:
255:
256:
257:
258:
259:
260:
261:
262:
263:
264:
265:
266:
267:
268:
269:
270:
271:
272:
273:
274:
275:
276:
277:
278:
279:
280:
281:
282:
283:
284:
285:
286:
287:
288:
289:
290:
291:
292:
293:
294:
295:
296:
297:
298:
299:
300:
301:

================================================================================================================
Netstat Results
================================================================================================================
Local Address,         Foreign Address,            (state),
local_address_1.www,   foreign_address_1.16194     , CLOSE_WAIT	
local_address_1.www,   foreign_address_2.60599     , CLOSE_WAIT	
local_address_1.443,   foreign_address_2.60839     , CLOSE_WAIT	
local_address_1.443,   foreign_address_3.20613     , CLOSE_WAIT
local_address_1.443,   foreign_address_3.45379     , CLOSE_WAIT
local_address_1.www,   foreign_address_3.45745     , CLOSE_WAIT
local_address_1.www,   foreign_address_3.45775     , CLOSE_WAIT
local_address_1.www,   foreign_address_3.46061     , CLOSE_WAIT
local_address_1.www,   foreign_address_4.11653     , CLOSE_WAIT
local_address_1.443,   foreign_address_4.11923     , CLOSE_WAIT
local_address_1.443,   foreign_address_4.12028     , CLOSE_WAIT
local_address_1.443,   foreign_address_5.35389     , CLOSE_WAIT
local_address_1.443,   foreign_address_6.10749     , CLOSE_WAIT
local_address_1.www,   foreign_address_6.10960     , CLOSE_WAIT
local_address_1.443,   foreign_address_6.11440     , CLOSE_WAIT
local_address_1.443,   foreign_address_7.31565     , CLOSE_WAIT
local_address_1.443,   foreign_address_7.32095     , CLOSE_WAIT
local_address_1.443,   foreign_address_8.27290     , CLOSE_WAIT
local_address_1.443,   foreign_address_8.28009     , CLOSE_WAIT
local_address_1.www,   foreign_address_8.28033     , CLOSE_WAIT
local_address_1.www,   foreign_address_8.3556      , CLOSE_WAIT
local_address_1.www,   foreign_address_9.21388     , CLOSE_WAIT
local_address_1.www,   foreign_address_10.55432    , CLOSE_WAIT
local_address_1.www,   foreign_address_11.55803    , CLOSE_WAIT
local_address_1.www,   foreign_address_12.26118    , CLOSE_WAIT
local_address_1.www,   foreign_address_13.49844    , CLOSE_WAIT
local_address_1.443,   foreign_address_14.s.54361  , CLOSE_WAIT
local_address_1.443,   foreign_address_14.s.55122  , CLOSE_WAIT
local_address_1.443,   foreign_address_15.20497    , CLOSE_WAIT
local_address_1.www,   foreign_address_15.20763    , CLOSE_WAIT
local_address_1.www,   foreign_address_15.45526    , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3557     , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3623     , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3660     , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3691     , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3693     , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3696     , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3705     , CLOSE_WAIT
local_address_1.443,   foreign_address_16.3707     , CLOSE_WAIT
local_address_1.www,   foreign_address_16.3709     , CLOSE_WAIT
local_address_1.www,   foreign_address_16.3712     , CLOSE_WAIT
local_address_1.www,   foreign_address_17.4908     , CLOSE_WAIT
local_address_1.www,   foreign_address_17.54341    , CLOSE_WAIT
local_address_1.www,   foreign_address_18.l.22196  , CLOSE_WAIT
local_address_1.www,   foreign_address_18.l.22356  , CLOSE_WAIT
local_address_1.www,   foreign_address_18.l.46641  , CLOSE_WAIT
local_address_1.www,   foreign_address_19.59834    , CLOSE_WAIT
local_address_1.www,   foreign_address_20.6521     , CLOSE_WAIT
local_address_1.443,   foreign_address_20.6665     , CLOSE_WAIT
local_address_1.443,   foreign_address_20.6701     , CLOSE_WAIT
local_address_1.443,   foreign_address_21.36324    , CLOSE_WAIT
local_address_1.443,   foreign_address_21.36661    , CLOSE_WAIT
local_address_1.443,   foreign_address_22.cp.26188 , CLOSE_WAIT
local_address_1.443,   foreign_address_23.13950    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14043    , CLOSE_WAIT
local_address_1.www,   foreign_address_23.14109    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14120    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14133    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14150    , CLOSE_WAIT
local_address_1.www,   foreign_address_23.14165    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14201    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14226    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14269    , CLOSE_WAIT
local_address_1.www,   foreign_address_23.14292    , CLOSE_WAIT
local_address_1.www,   foreign_address_23.14322    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14336    , CLOSE_WAIT
local_address_1.443,   foreign_address_23.14338    , CLOSE_WAIT
local_address_1.443,   foreign_address_24.39104    , CLOSE_WAIT
local_address_1.443,   foreign_address_25.38173    , CLOSE_WAIT
local_address_1.443,   foreign_address_26.58945    , CLOSE_WAIT
local_address_1.443,   foreign_address_26.2175     , CLOSE_WAIT
local_address_1.443,   foreign_address_26.2183     , CLOSE_WAIT
local_address_1.443,   foreign_address_26.2190     , CLOSE_WAIT
local_address_1.443,   foreign_address_26.2224     , CLOSE_WAIT
local_address_1.443,   foreign_address_26.2231     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2141     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2144     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2162     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2173     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2176     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2179     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2186     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2190     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2199     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2206     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2209     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2212     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2219     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2221     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2222     , CLOSE_WAIT
local_address_1.www,   foreign_address_27.2227     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2231     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.2234     , CLOSE_WAIT
local_address_1.443,   foreign_address_27.51784    , CLOSE_WAIT
local_address_1.443,   foreign_address_28.44566    , CLOSE_WAIT
local_address_1.443,   foreign_address_29.44592    , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4008     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4016     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4037     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4041     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4044     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4047     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4052     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4058     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4065     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4067     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4068     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4071     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4074     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4078     , CLOSE_WAIT
local_address_1.443,   foreign_address_30.4087     , CLOSE_WAIT
local_address_1.443,   foreign_address_31.14485    , CLOSE_WAIT
local_address_1.443,   foreign_address_32.4196     , CLOSE_WAIT
local_address_1.443,   foreign_address_32.4198     , CLOSE_WAIT
local_address_1.443,   foreign_address_33.15291    , CLOSE_WAIT
local_address_1.443,   foreign_address_33.15597    , CLOSE_WAIT
local_address_1.www,   foreign_address_34.50549    , CLOSE_WAIT
local_address_1.443,   foreign_address_35.20621    , CLOSE_WAIT
local_address_1.443,   foreign_address_36.47187    , CLOSE_WAIT
local_address_1.443,   foreign_address_37.31364    , CLOSE_WAIT
local_address_1.443,   foreign_address_38.39666    , CLOSE_WAIT
local_address_1.443,   foreign_address_5.35950     , ESTABLISHE
local_address_1.443,   foreign_address_8.27605     , ESTABLISHE
local_address_1.443,   foreign_address_38.59375    , ESTABLISHE
local_address_1.443,   foreign_address_39.cp.26337 , ESTABLISHE
local_address_1.443,   foreign_address_23.14369    , ESTABLISHE
local_address_1.443,   foreign_address_27.2237     , ESTABLISHE
local_address_1.443,   foreign_address_40.56387    , LAST_ACK
local_address_1.443,   foreign_address_16.3585     , LAST_ACK
local_address_1.443,   foreign_address_16.3589     , LAST_ACK
local_address_1.www,   foreign_address_16.3625     , LAST_ACK
local_address_1.443,   foreign_address_16.3631     , LAST_ACK
local_address_1.443,   foreign_address_41.13230    , LAST_ACK
local_address_1.www,   foreign_address_20.7237     , LAST_ACK
local_address_1.443,   foreign_address_21.36334    , LAST_ACK
local_address_1.www,   foreign_address_23.13854    , LAST_ACK
local_address_1.443,   foreign_address_42.58060    , LAST_ACK
local_address_1.443,   foreign_address_43.59057    , LAST_ACK
local_address_1.443,   foreign_address_27.2094     , LAST_ACK
local_address_1.443,   foreign_address_27.2103     , LAST_ACK
local_address_1.443,   foreign_address_27.2113     , LAST_ACK
local_address_1.443,   foreign_address_27.2124     , LAST_ACK
local_address_1.443,   foreign_address_27.2156     , LAST_ACK
local_address_1.443,   foreign_address_27.2184     , LAST_ACK
local_address_1.www,   foreign_address_30.3944     , LAST_ACK
local_address_1.www,   foreign_address_30.3983     , LAST_ACK
local_address_1.www,   foreign_address_30.3985     , LAST_ACK
local_address_1.443,   foreign_address_30.4018     , LAST_ACK
local_address_1.443,   foreign_address_30.4032     , LAST_ACK
local_address_1.www,   foreign_address_30.4039     , LAST_ACK
local_address_1.www,   foreign_address_30.4042     , LAST_ACK
local_address_1.www,   foreign_address_30.4043     , LAST_ACK
local_address_1.www,   foreign_address_30.4048     , LAST_ACK
local_address_1.443,   foreign_address_44.4176     , LAST_ACK
local_address_1.443,   foreign_address_45.58708    , LAST_ACK
local_address_1.www,   foreign_address_45.59205    , LAST_ACK
 
================================================================================================================
httpd.conf
================================================================================================================
 
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300
 
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
 
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100
 
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15
 
##
## Server-Pool Size Regulation (MPM specific)
## 
 
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers         5
MinSpareServers      5
MaxSpareServers     10
ServerLimit	   512
MaxClients         500
MaxRequestsPerChild  500000
</IfModule>
 
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75 
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>
 
# perchild MPM
# NumServers: constant number of server processes
# StartThreads: initial number of worker threads in each server process
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# MaxThreadsPerChild: maximum number of worker threads in each server process
# MaxRequestsPerChild: maximum number of connections per server process
<IfModule perchild.c>
NumServers           5
StartThreads         5
MinSpareThreads      5
MaxSpareThreads     10
MaxThreadsPerChild  20
MaxRequestsPerChild  0
</IfModule>
 
# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum  number of requests a server process serves
<IfModule mpm_winnt.c>
ThreadsPerChild 250
MaxRequestsPerChild  0
</IfModule>
 
# BeOS MPM
# StartThreads: how many threads do we initially spawn?
# MaxClients:   max number of threads we can have (1 thread == 1 client)
# MaxRequestsPerThread: maximum number of requests each thread will process
<IfModule beos.c>
StartThreads               10
MaxClients                 50
MaxRequestsPerThread       10000
</IfModule>    
 
# NetWare MPM
# ThreadStackSize: Stack size allocated for each worker thread
# StartThreads: Number of worker threads launched at server startup
# MinSpareThreads: Minimum number of idle threads, to handle request spikes
# MaxSpareThreads: Maximum number of idle threads
# MaxThreads: Maximum number of worker threads alive at the same time
# MaxRequestsPerChild: Maximum  number of requests a thread serves. It is 
#                      recommended that the default value of 0 be set for this
#                      directive on NetWare.  This will allow the thread to 
#                      continue to service requests indefinitely.                          
<IfModule mpm_netware.c>
ThreadStackSize      65536
StartThreads           250
MinSpareThreads         25
MaxSpareThreads        250
MaxThreads            1000
MaxRequestsPerChild      0
</IfModule>
 
# OS/2 MPM
# StartServers: Number of server processes to maintain
# MinSpareThreads: Minimum number of idle threads per process, 
#                  to handle request spikes
# MaxSpareThreads: Maximum number of idle threads per process
# MaxRequestsPerChild: Maximum number of connections per server process
<IfModule mpmt_os2.c>
StartServers           2
MinSpareThreads        5
MaxSpareThreads       10
MaxRequestsPerChild    0
</IfModule>
 
#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
 
Listen 80
Listen 443
[+][-]10/06/08 07:33 AM, ID: 22650408Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10/06/08 08:19 AM, ID: 22650926Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10/06/08 08:22 AM, ID: 22650962Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10/06/08 09:28 AM, ID: 22651735Expert Comment

At Experts Exchange, members can ask their questions to thousands of technology professionals, also known as Experts. Experts compete and collaborate to answer those questions by leaving comments like this one.

Start your 30-day free trial to view this Expert Comment or ask the Experts your question.

 
[+][-]10/06/08 09:57 AM, ID: 22652017Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10/06/08 10:05 AM, ID: 22652092Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
[+][-]10/06/08 11:30 AM, ID: 22652927Accepted Solution

View this solution now by starting your 30-day free trial. Setting up your free trial is quick, easy, and secure. We will return you to this solution, unlocked, when you're done.

About this solution

Zone: Apache Web Server
Tags: Apache HTTP Project, Apache, 2.0.48, Running on AIX
Sign Up Now!
Solution Provided By: urgoll
Participating Experts: 1
Solution Grade: A
 
[+][-]10/13/08 10:28 AM, ID: 22704277Author Comment

Often, when Experts are collaborating with members who have asked questions, they will request additional information about the problem. Askers respond with an author comment like this one.

Start your 30-day free trial to view this Author Comment or ask the Experts your question.

 
 
Loading Advertisement...
20091111-EE-VQP-89 / EE_QW_2_20070628