Solved

Why do my processes not get older?

Posted on 2004-09-12
22
367 Views
Last Modified: 2013-12-16
Hi all, I have a server running and something weird is going on with Apache-- the httpd threads are going to sleep, not waking up, and (strangest of all) aren't aging.

What I mean is, here's part of my 'top' display, sorted by time:


  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  356 root       9   0   416  340   304 S     0.0  0.0 236:35 ipband
29911 daemon     9   0  4644 4416  4024 S     0.0  0.4  54:58 named
    4 root       9   0     0    0     0 SW    0.0  0.0  43:56 kswapd
   26 root       9   0     0    0     0 SW    0.0  0.0   8:57 kjournald
18672 nobody     9   0 17004 4844  4724 S     0.0  0.5   8:13 squid
10891 nobody     9   0  9068 7836  3824 S     0.0  0.8   5:35 httpd
 7309 nobody     9   0  9016 7748  3836 S     0.0  0.8   5:19 httpd
13665 nobody     9   0 16576  15M  4036 S     0.0  1.7   4:46 httpd
13666 nobody     9   0  9140 8076  3980 S     0.0  0.8   4:43 httpd
   24 root       9   0     0    0     0 SW    0.0  0.0   4:11 kjournald
   10 root       9   0     0    0     0 SW    0.0  0.0   4:11 kjournald
 5327 nobody     9   0  8584 7332  3792 S     0.0  0.8   3:52 httpd
 5320 nobody     9   0  8568 7324  3812 S     0.0  0.8   3:38 httpd
 5345 nobody     9   0  9116 7836  3620 S     0.0  0.8   3:35 httpd
 5336 nobody     9   0  9792 8332  3548 S     0.0  0.9   3:24 httpd
 5322 nobody     9   0  8896 7620  3788 S     0.0  0.8   3:24 httpd
 5318 nobody     9   0  9512 8012  3516 S     0.0  0.8   3:24 httpd
 5302 nobody     9   0  8740 7408  3784 S     0.0  0.8   3:21 httpd
 5311 nobody     9   0  9832 8552  3764 S     0.0  0.9   3:15 httpd
 5330 nobody     9   0  8408 7136  3560 S     0.0  0.7   3:14 httpd

There's dozens more httpd threads... all of them don't get older. As I sit there with 'top' and tap the space bar to refresh the display, the 'time' column never goes up.

So what's up with that?

(I'd love to get to the bottom of where these threads come from in the first place, but barring that, I've written a quick-and-VERY-dirty hack, trigger by cron, that just kills threads older than 4 minutes. Yes, awful, I know, but it at least keeps the server moderately clean while I work on solving the "real" issue of their genesis.)

Appreciate any help!

best
Eric


0
Comment
Question by:pnoeric
  • 8
  • 5
  • 4
  • +3
22 Comments
 
LVL 45

Accepted Solution

by:
sunnycoder earned 75 total points
ID: 12041450
Hi pnoeric,

Apache has a thread based architecture which keeps a pool of free threads to be able to service http requests as and when they come.

There is no point in killing these from cron as apache will generate more thread to keep a minimum amuont of free threads ... If you are not running a webserver, then shut down apache ... If you are running a webserver and want to reduce the number of free threads, look in httpd.conf file and modify the MaxSpareServers directive in that file ... refer to apache man page for more details

Sunnycoder
0
 
LVL 38

Assisted Solution

by:yuzh
yuzh earned 50 total points
ID: 12041464
What do tou mean by "processes not get older"?

do a:

ps -ef | grep httpd

it should show you when the process is start etc. httpd should keep runing unless you
shult it down.

man ps

BTW, which version of OS are you using?

0
 

Author Comment

by:pnoeric
ID: 12041559
I did ps -ef | grep httpd | less

and found 62 lines of:

nobody    5309 22396  0 12:55 ?        00:02:54 /usr/local/apache/bin/httpd -DSSL
nobody    5310 22396  0 12:55 ?        00:02:51 /usr/local/apache/bin/httpd -DSSL
nobody    5311 22396  0 12:55 ?        00:03:26 /usr/local/apache/bin/httpd -DSSL
nobody    5312 22396  0 12:55 ?        00:03:13 /usr/local/apache/bin/httpd -DSSL
nobody    5313 22396  0 12:55 ?        00:02:58 /usr/local/apache/bin/httpd -DSSL
nobody    5314 22396  0 12:55 ?        00:02:39 /usr/local/apache/bin/httpd -DSSL
nobody    5315 22396  0 12:55 ?        00:02:40 /usr/local/apache/bin/httpd -DSSL
nobody    5316 22396  0 12:55 ?        00:02:39 /usr/local/apache/bin/httpd -DSSL
nobody    5317 22396  0 12:55 ?        00:02:52 /usr/local/apache/bin/httpd -DSSL

But when I do it again, I get exactly the same results.

Shouldn't the "time" column be going up every second?

I'm not sure how to tell what version of the OS I'm using. (I know-- crazy, right? I'm not the sysadmin, I'm just a coder... but tell me what the command is to report the current os version etc and I'll tell ya.)

 ---

Sunny - re: extra free threads - hmm. If my machine is getting slow when I have these very old httpd threads going, the problem could be just an out-of-memory situation. So I should just lower my open threads. Good idea. I'll try it.

best
E
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12041582
to find out your OS version, type in:

uname -a

0
 

Author Comment

by:pnoeric
ID: 12041587
btw my MaxSpareServers is set to 50 (the default in the conf file, incidentally). So if I have 62 of them right now, is something going wrong?
0
 
LVL 45

Expert Comment

by:sunnycoder
ID: 12041596
No, you have 50 idle spare servers and rest might be actively serving requests or doing some other chores
0
 

Author Comment

by:pnoeric
ID: 12041603
uname -a gives me

Linux svr1 2.4.24 #1 Mon Jan 5 13:46:16 PST 2004 i686 unknown

0
 

Author Comment

by:pnoeric
ID: 12041621
Yeah, I think there's something weird going on-- I now have 90 httpd processes going on, and it continues to climb. :-( Any thoughts on how to debug that, i.e., tell what URLs are triggering the processes and how they are getting stuck on?
0
 
LVL 45

Expert Comment

by:sunnycoder
ID: 12041627
try running netstat -a  .. that will give you all open connections ... The number of threads maintained by apache vary according to the load on the server. It tried to maintain some free server threads and kills them if they exceed the set limit
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12041628
PS: top and ps show you the process starting time, for the same running process it is
      always the SAME, unless you kill it and restart another one!

      man top
      man ps
      To learn more details
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12041647
For Linux, you can use:

ps -uax | grep httpd

to check it out
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 20

Expert Comment

by:tfewster
ID: 12041723
Just to clarify, the TIME column for both `top` and `ps` is the amount of CPU time used by the process; If the process is sleeping, the time  won't increase. Now back to sunnycoder & yuzh, who can help on the underlying problem ;-)
0
 

Author Comment

by:pnoeric
ID: 12041818
Thanks tfewster, I didn't realize that. So the threads I had with the time equal to 160 minutes and stuff... those were basically runaway processes, it sounds like. Or threads that were being reused, and reused, and reused...
0
 
LVL 45

Expert Comment

by:sunnycoder
ID: 12041832
Reused is a possibility
0
 
LVL 61

Expert Comment

by:gheist
ID: 12041878
Could you post output of $ vmstat 5 5 ; uptime ; free
, to see if there is ANY problem
0
 

Author Comment

by:pnoeric
ID: 12041912
good idea... the results of vmstat 5 5 ; uptime ; free:

   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0 105412  67288  60844 324448   0   2     2     3    1     4   3   4   3
 0  0  0 105412  65988  60876 324696   5   0    51   115  373   857  56   3  41
 2  0  0 105412  58836  60904 324820   2   0    25   109  270   450  26   3  71
 0  0  0 105412  54828  60936 324852   0   0     3   129  248   483  41   2  57
 0  0  0 105412  57296  60960 324944   0   0    17   113  261   442  28   3  70

 00:55:36 up 88 days, 13:59,  1 user,  load average: 0.55, 0.89, 0.99

             total       used       free     shared    buffers     cached
Mem:        904356     847060      57296          0      60960     324956
-/+ buffers/cache:     461144     443212
Swap:      2457924     105412    2352512

(sorry, I can't tell if threre's a way to force Courier for the above)
0
 
LVL 38

Expert Comment

by:yuzh
ID: 12041977
Check you httpd.conf file for, please have a look at the answer for the following question:

http:Q_20299553.html

PS: If you are rung apache 1.x, consider upgrade it to apache 2.x, the newer version
      of apache handle the system resource better than the older one!


0
 
LVL 61

Expert Comment

by:gheist
ID: 12042416
You should reduce MaxClients a bit to avoid swapping, but otherwise your system looks fine
0
 

Author Comment

by:pnoeric
ID: 12049099
Uh oh. I adjusted my httpd.conf  per that other thread and now my server is very, very slow.

netstat -a shows a LOT: dozens and dozens of these:

tcp        0      0 localhost:3306          localhost:40334         TIME_WAIT
tcp        0      0 localhost:3306          localhost:40335         TIME_WAIT
tcp        0      0 localhost:3306          localhost:40398         TIME_WAIT


and dozens and dozens of these:

tcp        0      0 62-113-98-39.digit:http d47-69-100-204.try:2190 TIME_WAIT
tcp        0      0 62-113-98-39.digit:http d47-69-100-204.try:2189 TIME_WAIT


I ended up re-adjusting the conf back to where it was and restarted mysql and apache. I think the moral is: don't fool with what works. :-)

My server load also jumped to like 19, 21, 24 etc... now it's drifting back down quickly and I hope to get it under 1 again. Eek.

E


This looks like
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12063221
> my processes not get older
the time displayed by ps and top is the "system time" not the "user time" or "elapsed time" used by the process.
Be happy that it does not increase ;-)
0
 
LVL 61

Expert Comment

by:gheist
ID: 12149715
netstat -a shows a LOT: dozens and dozens of these:

tcp        0      0 localhost:3306          localhost:40334         TIME_WAIT

this is normal, when you run out of sockets (which is unlikely), you may reduce interval between tcp keepalive probes.
0
 

Author Comment

by:pnoeric
ID: 12174267
thanks all for the help.
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Every server (virtual or physical) needs a console: and the console can be provided through hardware directly connected, software for remote connections, local connections, through a KVM, etc. This document explains the different types of consol…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now