• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 381
  • Last Modified:

Why do my processes not get older?

Hi all, I have a server running and something weird is going on with Apache-- the httpd threads are going to sleep, not waking up, and (strangest of all) aren't aging.

What I mean is, here's part of my 'top' display, sorted by time:


  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  356 root       9   0   416  340   304 S     0.0  0.0 236:35 ipband
29911 daemon     9   0  4644 4416  4024 S     0.0  0.4  54:58 named
    4 root       9   0     0    0     0 SW    0.0  0.0  43:56 kswapd
   26 root       9   0     0    0     0 SW    0.0  0.0   8:57 kjournald
18672 nobody     9   0 17004 4844  4724 S     0.0  0.5   8:13 squid
10891 nobody     9   0  9068 7836  3824 S     0.0  0.8   5:35 httpd
 7309 nobody     9   0  9016 7748  3836 S     0.0  0.8   5:19 httpd
13665 nobody     9   0 16576  15M  4036 S     0.0  1.7   4:46 httpd
13666 nobody     9   0  9140 8076  3980 S     0.0  0.8   4:43 httpd
   24 root       9   0     0    0     0 SW    0.0  0.0   4:11 kjournald
   10 root       9   0     0    0     0 SW    0.0  0.0   4:11 kjournald
 5327 nobody     9   0  8584 7332  3792 S     0.0  0.8   3:52 httpd
 5320 nobody     9   0  8568 7324  3812 S     0.0  0.8   3:38 httpd
 5345 nobody     9   0  9116 7836  3620 S     0.0  0.8   3:35 httpd
 5336 nobody     9   0  9792 8332  3548 S     0.0  0.9   3:24 httpd
 5322 nobody     9   0  8896 7620  3788 S     0.0  0.8   3:24 httpd
 5318 nobody     9   0  9512 8012  3516 S     0.0  0.8   3:24 httpd
 5302 nobody     9   0  8740 7408  3784 S     0.0  0.8   3:21 httpd
 5311 nobody     9   0  9832 8552  3764 S     0.0  0.9   3:15 httpd
 5330 nobody     9   0  8408 7136  3560 S     0.0  0.7   3:14 httpd

There's dozens more httpd threads... all of them don't get older. As I sit there with 'top' and tap the space bar to refresh the display, the 'time' column never goes up.

So what's up with that?

(I'd love to get to the bottom of where these threads come from in the first place, but barring that, I've written a quick-and-VERY-dirty hack, trigger by cron, that just kills threads older than 4 minutes. Yes, awful, I know, but it at least keeps the server moderately clean while I work on solving the "real" issue of their genesis.)

Appreciate any help!

best
Eric


0
pnoeric
Asked:
pnoeric
  • 8
  • 5
  • 4
  • +3
2 Solutions
 
sunnycoderCommented:
Hi pnoeric,

Apache has a thread based architecture which keeps a pool of free threads to be able to service http requests as and when they come.

There is no point in killing these from cron as apache will generate more thread to keep a minimum amuont of free threads ... If you are not running a webserver, then shut down apache ... If you are running a webserver and want to reduce the number of free threads, look in httpd.conf file and modify the MaxSpareServers directive in that file ... refer to apache man page for more details

Sunnycoder
0
 
yuzhCommented:
What do tou mean by "processes not get older"?

do a:

ps -ef | grep httpd

it should show you when the process is start etc. httpd should keep runing unless you
shult it down.

man ps

BTW, which version of OS are you using?

0
 
pnoericAuthor Commented:
I did ps -ef | grep httpd | less

and found 62 lines of:

nobody    5309 22396  0 12:55 ?        00:02:54 /usr/local/apache/bin/httpd -DSSL
nobody    5310 22396  0 12:55 ?        00:02:51 /usr/local/apache/bin/httpd -DSSL
nobody    5311 22396  0 12:55 ?        00:03:26 /usr/local/apache/bin/httpd -DSSL
nobody    5312 22396  0 12:55 ?        00:03:13 /usr/local/apache/bin/httpd -DSSL
nobody    5313 22396  0 12:55 ?        00:02:58 /usr/local/apache/bin/httpd -DSSL
nobody    5314 22396  0 12:55 ?        00:02:39 /usr/local/apache/bin/httpd -DSSL
nobody    5315 22396  0 12:55 ?        00:02:40 /usr/local/apache/bin/httpd -DSSL
nobody    5316 22396  0 12:55 ?        00:02:39 /usr/local/apache/bin/httpd -DSSL
nobody    5317 22396  0 12:55 ?        00:02:52 /usr/local/apache/bin/httpd -DSSL

But when I do it again, I get exactly the same results.

Shouldn't the "time" column be going up every second?

I'm not sure how to tell what version of the OS I'm using. (I know-- crazy, right? I'm not the sysadmin, I'm just a coder... but tell me what the command is to report the current os version etc and I'll tell ya.)

 ---

Sunny - re: extra free threads - hmm. If my machine is getting slow when I have these very old httpd threads going, the problem could be just an out-of-memory situation. So I should just lower my open threads. Good idea. I'll try it.

best
E
0
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

 
yuzhCommented:
to find out your OS version, type in:

uname -a

0
 
pnoericAuthor Commented:
btw my MaxSpareServers is set to 50 (the default in the conf file, incidentally). So if I have 62 of them right now, is something going wrong?
0
 
sunnycoderCommented:
No, you have 50 idle spare servers and rest might be actively serving requests or doing some other chores
0
 
pnoericAuthor Commented:
uname -a gives me

Linux svr1 2.4.24 #1 Mon Jan 5 13:46:16 PST 2004 i686 unknown

0
 
pnoericAuthor Commented:
Yeah, I think there's something weird going on-- I now have 90 httpd processes going on, and it continues to climb. :-( Any thoughts on how to debug that, i.e., tell what URLs are triggering the processes and how they are getting stuck on?
0
 
sunnycoderCommented:
try running netstat -a  .. that will give you all open connections ... The number of threads maintained by apache vary according to the load on the server. It tried to maintain some free server threads and kills them if they exceed the set limit
0
 
yuzhCommented:
PS: top and ps show you the process starting time, for the same running process it is
      always the SAME, unless you kill it and restart another one!

      man top
      man ps
      To learn more details
0
 
yuzhCommented:
For Linux, you can use:

ps -uax | grep httpd

to check it out
0
 
tfewsterCommented:
Just to clarify, the TIME column for both `top` and `ps` is the amount of CPU time used by the process; If the process is sleeping, the time  won't increase. Now back to sunnycoder & yuzh, who can help on the underlying problem ;-)
0
 
pnoericAuthor Commented:
Thanks tfewster, I didn't realize that. So the threads I had with the time equal to 160 minutes and stuff... those were basically runaway processes, it sounds like. Or threads that were being reused, and reused, and reused...
0
 
sunnycoderCommented:
Reused is a possibility
0
 
gheistCommented:
Could you post output of $ vmstat 5 5 ; uptime ; free
, to see if there is ANY problem
0
 
pnoericAuthor Commented:
good idea... the results of vmstat 5 5 ; uptime ; free:

   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0 105412  67288  60844 324448   0   2     2     3    1     4   3   4   3
 0  0  0 105412  65988  60876 324696   5   0    51   115  373   857  56   3  41
 2  0  0 105412  58836  60904 324820   2   0    25   109  270   450  26   3  71
 0  0  0 105412  54828  60936 324852   0   0     3   129  248   483  41   2  57
 0  0  0 105412  57296  60960 324944   0   0    17   113  261   442  28   3  70

 00:55:36 up 88 days, 13:59,  1 user,  load average: 0.55, 0.89, 0.99

             total       used       free     shared    buffers     cached
Mem:        904356     847060      57296          0      60960     324956
-/+ buffers/cache:     461144     443212
Swap:      2457924     105412    2352512

(sorry, I can't tell if threre's a way to force Courier for the above)
0
 
yuzhCommented:
Check you httpd.conf file for, please have a look at the answer for the following question:

http:Q_20299553.html

PS: If you are rung apache 1.x, consider upgrade it to apache 2.x, the newer version
      of apache handle the system resource better than the older one!


0
 
gheistCommented:
You should reduce MaxClients a bit to avoid swapping, but otherwise your system looks fine
0
 
pnoericAuthor Commented:
Uh oh. I adjusted my httpd.conf  per that other thread and now my server is very, very slow.

netstat -a shows a LOT: dozens and dozens of these:

tcp        0      0 localhost:3306          localhost:40334         TIME_WAIT
tcp        0      0 localhost:3306          localhost:40335         TIME_WAIT
tcp        0      0 localhost:3306          localhost:40398         TIME_WAIT


and dozens and dozens of these:

tcp        0      0 62-113-98-39.digit:http d47-69-100-204.try:2190 TIME_WAIT
tcp        0      0 62-113-98-39.digit:http d47-69-100-204.try:2189 TIME_WAIT


I ended up re-adjusting the conf back to where it was and restarted mysql and apache. I think the moral is: don't fool with what works. :-)

My server load also jumped to like 19, 21, 24 etc... now it's drifting back down quickly and I hope to get it under 1 again. Eek.

E


This looks like
0
 
ahoffmannCommented:
> my processes not get older
the time displayed by ps and top is the "system time" not the "user time" or "elapsed time" used by the process.
Be happy that it does not increase ;-)
0
 
gheistCommented:
netstat -a shows a LOT: dozens and dozens of these:

tcp        0      0 localhost:3306          localhost:40334         TIME_WAIT

this is normal, when you run out of sockets (which is unlikely), you may reduce interval between tcp keepalive probes.
0
 
pnoericAuthor Commented:
thanks all for the help.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 8
  • 5
  • 4
  • +3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now