Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

Appropiate Configuration for Startup and Standby Servers

Posted on 2009-04-02
17
Medium Priority
?
414 Views
Last Modified: 2012-05-06
I have been having an issue with my apache server at random times stop responding.  I would restart it and it would be on its merry way.  After the last time I turned up logging on the production machine and kept an eye on it, once it happened again I went into the error logs and picked up this gem.

"[Wed Apr 01 19:07:38 2009] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 7 idle, and 48 total children"

there were several of these and then a few more lines of "[Wed Apr 01 19:07:38 2009] [debug] proxy_util.c(1778): proxy: initialized single connection worker 0 in child 21005 for (*)" and then nothing until I restarted.  From the looks of it I should up the StartServers and Min/MaxSpareServers, but this is where I dont really understand a whole lot about the httpd.conf file and what to set, so if someone could look over my settings and information and point me in the right direction or even just tell my dense headed self I would appreciate it .. oh and shower you with Points.  :D
httpd.conf
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers       16
MinSpareServers    10
MaxSpareServers   20
ServerLimit      512
MaxClients       512
MaxRequestsPerChild  4000
</IfModule>
 
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>
 
#free -mt
             total       used       free     shared    buffers     cached
Mem:           998        967         31          0        173        560
-/+ buffers/cache:        233        765
Swap:         2008         28       1979
Total:        3006        995       2011
 
#/sbin/swapon -s
Filename                                Type            Size    Used    Priority
/dev/sda5                               partition       2056280 28876   -1
 
#/usr/sbin/httpd -v
Server version: Apache/2.2.8 (Unix)
 
# /usr/sbin/httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

Open in new window

0
Comment
Question by:iceman19330
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 8
17 Comments
 
LVL 29

Expert Comment

by:fosiul01
ID: 24050423
1. does your Server get too much hits ??
you need to think to edit prefork.c if your sites gets too much hits


2. when this happen, have you took the "top" report ??( to see cpu uses)
3)  your Ram looks like less

4. can you enable server_status report from apaache to see whats going on in peack time
have a look  this one ,how to enable server_status report
http://www.cyberciti.biz/faq/apache-server-status/
example :
http://httpd.apache.org/server-status 

from server_status report you will see, how much worker proces you have and what the status of your apache

also please read this article you will get a good idea

http://www.devside.net/articles/apache-performance-tuning

0
 

Author Comment

by:iceman19330
ID: 24050888
1.  I would have to dig through the logs and see what the overall traffic was for that time, most of our statistics are per domain, we have about 6 domains of varying size.

2.  No I didn't, I will try that next time.

3.  Ram is 1GB

4.  I have server status enabled, I just enabled extended status.
0
 

Author Comment

by:iceman19330
ID: 24050954
I did run top just now, nothing has gone wrong.  There was 1 httpd process, then 10 more popped up.
Top:
 
29234 apache    20   0 72080  21m  12m S  4.3  2.1   0:00.45 httpd                                                                  
 2016 mysql     20   0  145m  32m 4932 S  4.0  3.3 123:44.71 mysqld                                                                 
29052 apache    20   0 72096  21m  12m S  2.7  2.2   0:00.79 httpd                                                                  
29056 apache    20   0 74636  30m  18m S  2.7  3.0   0:01.23 httpd                                                                  
29048 apache    20   0 76772  31m  17m S  2.3  3.1   0:01.10 httpd                                                                  
29051 apache    20   0 72088  25m  16m S  1.7  2.6   0:00.95 httpd                                                                  
29060 apache    20   0 72644  27m  18m S  0.7  2.8   0:00.66 httpd                                                                  
29061 apache    20   0 72484  22m  12m S  0.7  2.2   0:01.05 httpd                                                                  
29049 apache    20   0 76164  30m  17m S  0.3  3.0   0:00.91 httpd                                                                  
29053 apache    20   0 72128  27m  18m S  0.3  2.7   0:00.83 httpd                                                                  
29054 apache    20   0 72112  27m  18m S  0.3  2.7   0:01.01 httpd                                                                  
29055 apache    20   0 72124  31m  22m S  0.3  3.1   0:01.16 httpd                                                                  
29058 apache    20   0 72120  27m  18m S  0.3  2.7   0:00.71 httpd                                                                  
29059 apache    20   0 72072  21m  12m S  0.3  2.2   0:00.68 httpd                                                                  
29062 apache    20   0 74000  28m  17m S  0.3  2.8   0:01.02 httpd                                                                  
29063 apache    20   0 72484  22m  12m S  0.3  2.2   0:00.86 httpd                                                                  
29064 apache    20   0 72480  26m  16m S  0.3  2.6   0:00.77 httpd                                                                  
29107 apache    20   0 72628  23m  14m S  0.3  2.4   0:00.66 httpd                                                                  
29185 apache    20   0 72484  17m 8516 S  0.3  1.8   0:00.51 httpd                                                                  
29191 apache    20   0 72080  17m 8544 S  0.3  1.7   0:00.38 httpd 
 
 
# ps -ylC httpd --sort:rss
S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S    48 29047 29045  0  80   0  5488  6761 -      ?        00:00:00 httpd
S     0 29045     1  0  80   0 16444 17771 -      ?        00:00:00 httpd
S    48 29191 29045  0  80   0 17720 18020 semtim ?        00:00:00 httpd
S    48 29185 29045  0  80   0 18096 18121 semtim ?        00:00:00 httpd
S    48 29234 29045  0  80   0 21912 18020 semtim ?        00:00:00 httpd
S    48 29059 29045  0  80   0 22064 18018 semtim ?        00:00:00 httpd
S    48 29052 29045  0  80   0 22072 18024 semtim ?        00:00:00 httpd
S    48 29061 29045  0  80   0 22440 18026 semtim ?        00:00:01 httpd
S    48 29063 29045  0  80   0 22792 18121 semtim ?        00:00:00 httpd
S    48 29051 29045  0  80   0 26284 18022 semtim ?        00:00:00 httpd
S    48 29064 29045  0  80   0 26668 18120 semtim ?        00:00:00 httpd
S    48 29054 29045  0  80   0 27700 18028 -      ?        00:00:01 httpd
S    48 29058 29045  0  80   0 27708 18030 semtim ?        00:00:00 httpd
S    48 29060 29045  0  80   0 28360 18161 semtim ?        00:00:00 httpd
S    48 29062 29045  0  80   0 28800 18500 semtim ?        00:00:01 httpd
S    48 29049 29045  0  80   0 31052 19041 semtim ?        00:00:01 httpd
S    48 29056 29045  0  80   0 31160 18659 -      ?        00:00:01 httpd
S    48 29048 29045  0  80   0 31888 19193 semtim ?        00:00:01 httpd
S    48 29053 29045  0  80   0 31928 18032 semtim ?        00:00:00 httpd
S    48 29055 29045  0  80   0 31960 18031 semtim ?        00:00:01 httpd
S    48 29107 29045  0  80   0 32756 18157 -      ?        00:00:00 httpd
S    48 29087 29045  0  80   0 34904 19316 semtim ?        00:00:00 httpd

Open in new window

0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 29

Expert Comment

by:fosiul01
ID: 24051051
1. First off all, problem is prefork, server is runing out of worker process.

so you need to increase that.

but one question :

1.why suddently server is running out of worker process, that mean, some time, your serer is getting too much its from outside

why ??

is those realy hits or some one tryign to mess with the server such as brutal attack ??

only way you can check this to see Server log file always to see the hits is getting are those real hits ..
2. when this problem happennding, check how many apache process is running on the server .

3. when this problem is happenning, check top command for memory user and processor use

if you run top, you wil see this kind of output

Cpu(s):  1.2%us,  0.0%sy,  0.0%ni, 94.7%id,  0.3%wa,  0.0%hi,  0.0%si,  3.8%st
Mem:    524464k total,   506864k used,    17600k free,   123764k buffers
Swap:   262136k total,       32k used,   262104k free,   264220k cached

also : how often this problem happenning ??



0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24051194
have a read about apache article about prefork setting

http://httpd.apache.org/docs/2.2/mod/prefork.html

default prefork setting is able to hand 256 simultaneous request..
0
 

Author Comment

by:iceman19330
ID: 24051211
The problem happens at different times, sometimes it might be a couple of weeks, other times it might happen on back to back days.  I dont think it is necessarily malicious as these are a collection of decent (200-300 student) private religious schools, several of them average 30k visits a month, and the problems (not 100% sure) seem to occur early to mid evening.

i will check on the other stuff when a problem occurs and report back.
0
 

Author Comment

by:iceman19330
ID: 24051248
BTW this is what top has now.

top - 10:52:13 up 64 days,  1:15,  1 user,  load average: 0.06, 0.03, 0.02
Tasks: 116 total,   2 running, 114 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1022880k total,  1005824k used,    17056k free,   136916k buffers
Swap:  2056280k total,    28952k used,  2027328k free,   608676k cached

My other question would be, is there an alert system that I can setup that will tell me when this occurs, rather then having to keep checking the sites every little while.  I tried a system that would wget the html pages and if it returned a blank page then it would text me, but then that seemed a bit sketchy since it was dependent on another server to run and etc.
0
 
LVL 29

Accepted Solution

by:
fosiul01 earned 2000 total points
ID: 24051360

You can use Monit to monitor httpd service it will alert you if there is any problem and will restart the server

http://www.howtoforge.com/server-monitoring-with-munin-and-monit-on-centos-5.2

omm i dont think its memory, as the log said , you need to increate the prefork beause you are runnign out

so when this problem happen, try to incrase your prefork setting , see if that help

but your free ram is low... when this problem happended, check the Ram and processor utilization, just check if they are not running out ...
0
 

Author Comment

by:iceman19330
ID: 24055677
So it happened tonight and here is what I was able to get.

top - 18:18:05 up 64 days,  8:41,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 107 total,   1 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1022880k total,   884356k used,   138524k free,   180332k buffers
Swap:  2056280k total,    28952k used,  2027328k free,   523780k cached

2466 apache    20   0 72632  27m  18m S  2.0  2.8   0:01.20 httpd

ps -ylC httpd --sort:rss
S   UID   PID  PPID  C PRI  NI   RSS  S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S    48 32472 32470  0  80   0  5488  6761 -      ?        00:00:00 httpd
S    48  3034 32470  0  80   0 10108 17805 semtim ?        00:00:00 httpd
S     0 32470     1  0  80   0 16440 17771 -      ?        00:00:00 httpd
S    48  2974 32470  0  80   0 17892 18114 semtim ?        00:00:00 httpd
S    48  2978 32470  0  80   0 17928 18114 semtim ?        00:00:00 httpd
S    48  2979 32470  0  80   0 17932 18119 semtim ?        00:00:00 httpd
S    48  2976 32470  0  80   0 17976 18118 semtim ?        00:00:00 httpd
S    48  2453 32470  0  80   0 22008 18020 -      ?        00:00:01 httpd
S    48  2715 32470  0  80   0 22420 18122 semtim ?        00:00:00 httpd
S    48  2355 32470  0  80   0 27652 18028 semtim ?        00:00:00 httpd
S    48  2149 32470  0  80   0 27676 18028 semtim ?        00:00:01 httpd
S    48  2243 32470  0  80   0 28148 18032 semtim ?        00:00:01 httpd
S    48  2464 32470  0  80   0 28188 18161 semtim ?        00:00:01 httpd
S    48  2466 32470  0  80   0 28188 18158 semtim ?        00:00:01 httpd

I could not get into server-status, but that is because I was logging on from home and I only allow access from work.  *head slap*.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24058488
Ok
have you tryed to increase the prefork setting ?? [ its wired is not it at middle of the night apache is running out of worker process ?? ]

allow Server-status report to get access from your Home IP, so you would be able to see report from home

when this problem will happended, will you able to check, how much Established connection in your server ??

lsof -i TCP:80   : to check status of port 80

lsof -i TCP:80 | grep ESTABLISHED  = to see established connection

to get continuous you might consider to run this command in crontab to run every minutes and insert report in a text file ..

/var/log/message/httpd/website log  = see what kind of hits you getting



0
 

Author Comment

by:iceman19330
ID: 24059821
what would you suggest for an increase in the prefork settings?
0
 

Author Comment

by:iceman19330
ID: 24060137
So I got monit working because I am on a Plesk server I had it already however it was mis-configured so I redid some of the settings and it seems to be going.

[CDT Apr  3 08:34:07] 'apache' total mem amount of 584312kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:39:09] 'apache' total mem amount of 607332kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:44:11] 'apache' total mem amount of 611556kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:49:13] 'apache' total mem amount of 615944kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:54:16] 'apache' total mem amount of 608428kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:54:16] Sendmail error: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
[CDT Apr  3 08:54:16] 'apache' trying to restart
[CDT Apr  3 08:54:16] 'apache' stop: /usr/local/psa/admin/bin/websrvmng
[CDT Apr  3 08:54:17] 'apache' start: /usr/local/psa/admin/bin/websrvmng
[CDT Apr  3 08:54:17] 'apache' failed, cannot open a connection to INET[www.fakedomain.com:80]
[CDT Apr  3 08:54:17] Sendmail error: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
[CDT Apr  3 08:54:17] 'apache' trying to restart
[CDT Apr  3 08:59:19] 'apache' resource passed
[CDT Apr  3 08:59:19] Sendmail error: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
[CDT Apr  3 08:59:19] 'apache' connection passed
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24060229
Ok
I am not aware of this error
apache' total mem amount of 608428kB matches resource limit [total mem amount>512001kB]
i will have to have a look in google what does it mean

but  your prefork setting is allright
the default prefork setting apache can handle 256 simulatanous connection a  a time.  which is a lot, only site who gets more then 256 simlatanous connection need to modify prefork setting

i dont think you get 256 simaltanous true hit in apache

so why this problem ??

1. keep monitor your server, check log, i belived yoru server is under brutal attack.
2. what about apache log ?? hvae you check the hit log, does those log is comming from true user ??
3. if you see you are not gettign too much hits still apache is giving this troblel , then have to think something else

but before that you need to identify , what causing problem

1. attack or apache

if you cant seperate this issue , i would not be able to help you too much rather then just giving some theoritical idea..
0
 

Author Comment

by:iceman19330
ID: 24060273
should I turn on any additional logging or something to trace back what network this traffic is coming from.  most of the site on here are only really known in this region.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24060327
in /var/log/httpd/

cat mydomain_log    [ here i dont know how you configure your apache log for individual domain]

so check all access log to carefully to see what IP you see , what pages are browsing ...


also  lsof -i TCP:80 | wc -l
or lsof -i TCP:80 | grep ESTABLISHED | wc -l

it will show you how many connections are established at a time  , may be when this problem is happeening at that time there is too many connection to your server ..
0
 

Author Comment

by:iceman19330
ID: 24060470
I have a main access_log and then each domain is configured separately.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24060576
you know when your system went down

check every log prerior that time
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you've heard about htaccess and it sounds like it does what you want, but you're not sure how it works... well, you're in the right place. Read on. Some Basics #1. It's a file and its filename is .htaccess (yes, with a dot in the front). #…
In Solr 4.0 it is possible to atomically (or partially) update individual fields in a document. This article will show the operations possible for atomic updating as well as setting up your Solr instance to be able to perform the actions. One major …
Add bar graphs to Access queries using Unicode block characters. Graphs appear on every record in the color you want. Give life to numbers. Hopes this gives you ideas on visualizing your data in new ways ~ Create a calculated field in a query: …
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…
Suggested Courses

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question