Solved

Appropiate Configuration for Startup and Standby Servers

Posted on 2009-04-02
17
402 Views
Last Modified: 2012-05-06
I have been having an issue with my apache server at random times stop responding.  I would restart it and it would be on its merry way.  After the last time I turned up logging on the production machine and kept an eye on it, once it happened again I went into the error logs and picked up this gem.

"[Wed Apr 01 19:07:38 2009] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 7 idle, and 48 total children"

there were several of these and then a few more lines of "[Wed Apr 01 19:07:38 2009] [debug] proxy_util.c(1778): proxy: initialized single connection worker 0 in child 21005 for (*)" and then nothing until I restarted.  From the looks of it I should up the StartServers and Min/MaxSpareServers, but this is where I dont really understand a whole lot about the httpd.conf file and what to set, so if someone could look over my settings and information and point me in the right direction or even just tell my dense headed self I would appreciate it .. oh and shower you with Points.  :D
httpd.conf

# prefork MPM

# StartServers: number of server processes to start

# MinSpareServers: minimum number of server processes which are kept spare

# MaxSpareServers: maximum number of server processes which are kept spare

# ServerLimit: maximum value for MaxClients for the lifetime of the server

# MaxClients: maximum number of server processes allowed to start

# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule prefork.c>

StartServers       16

MinSpareServers    10

MaxSpareServers   20

ServerLimit      512

MaxClients       512

MaxRequestsPerChild  4000

</IfModule>
 

# worker MPM

# StartServers: initial number of server processes to start

# MaxClients: maximum number of simultaneous client connections

# MinSpareThreads: minimum number of worker threads which are kept spare

# MaxSpareThreads: maximum number of worker threads which are kept spare

# ThreadsPerChild: constant number of worker threads in each server process

# MaxRequestsPerChild: maximum number of requests a server process serves

<IfModule worker.c>

StartServers         2

MaxClients         150

MinSpareThreads     25

MaxSpareThreads     75

ThreadsPerChild     25

MaxRequestsPerChild  0

</IfModule>
 

#free -mt

             total       used       free     shared    buffers     cached

Mem:           998        967         31          0        173        560

-/+ buffers/cache:        233        765

Swap:         2008         28       1979

Total:        3006        995       2011
 

#/sbin/swapon -s

Filename                                Type            Size    Used    Priority

/dev/sda5                               partition       2056280 28876   -1
 

#/usr/sbin/httpd -v

Server version: Apache/2.2.8 (Unix)
 

# /usr/sbin/httpd -l

Compiled in modules:

  core.c

  prefork.c

  http_core.c

  mod_so.c

Open in new window

0
Comment
Question by:iceman19330
  • 9
  • 8
17 Comments
 
LVL 29

Expert Comment

by:fosiul01
ID: 24050423
1. does your Server get too much hits ??
you need to think to edit prefork.c if your sites gets too much hits


2. when this happen, have you took the "top" report ??( to see cpu uses)
3)  your Ram looks like less

4. can you enable server_status report from apaache to see whats going on in peack time
have a look  this one ,how to enable server_status report
http://www.cyberciti.biz/faq/apache-server-status/
example :
http://httpd.apache.org/server-status

from server_status report you will see, how much worker proces you have and what the status of your apache

also please read this article you will get a good idea

http://www.devside.net/articles/apache-performance-tuning

0
 

Author Comment

by:iceman19330
ID: 24050888
1.  I would have to dig through the logs and see what the overall traffic was for that time, most of our statistics are per domain, we have about 6 domains of varying size.

2.  No I didn't, I will try that next time.

3.  Ram is 1GB

4.  I have server status enabled, I just enabled extended status.
0
 

Author Comment

by:iceman19330
ID: 24050954
I did run top just now, nothing has gone wrong.  There was 1 httpd process, then 10 more popped up.
Top:
 

29234 apache    20   0 72080  21m  12m S  4.3  2.1   0:00.45 httpd                                                                  

 2016 mysql     20   0  145m  32m 4932 S  4.0  3.3 123:44.71 mysqld                                                                 

29052 apache    20   0 72096  21m  12m S  2.7  2.2   0:00.79 httpd                                                                  

29056 apache    20   0 74636  30m  18m S  2.7  3.0   0:01.23 httpd                                                                  

29048 apache    20   0 76772  31m  17m S  2.3  3.1   0:01.10 httpd                                                                  

29051 apache    20   0 72088  25m  16m S  1.7  2.6   0:00.95 httpd                                                                  

29060 apache    20   0 72644  27m  18m S  0.7  2.8   0:00.66 httpd                                                                  

29061 apache    20   0 72484  22m  12m S  0.7  2.2   0:01.05 httpd                                                                  

29049 apache    20   0 76164  30m  17m S  0.3  3.0   0:00.91 httpd                                                                  

29053 apache    20   0 72128  27m  18m S  0.3  2.7   0:00.83 httpd                                                                  

29054 apache    20   0 72112  27m  18m S  0.3  2.7   0:01.01 httpd                                                                  

29055 apache    20   0 72124  31m  22m S  0.3  3.1   0:01.16 httpd                                                                  

29058 apache    20   0 72120  27m  18m S  0.3  2.7   0:00.71 httpd                                                                  

29059 apache    20   0 72072  21m  12m S  0.3  2.2   0:00.68 httpd                                                                  

29062 apache    20   0 74000  28m  17m S  0.3  2.8   0:01.02 httpd                                                                  

29063 apache    20   0 72484  22m  12m S  0.3  2.2   0:00.86 httpd                                                                  

29064 apache    20   0 72480  26m  16m S  0.3  2.6   0:00.77 httpd                                                                  

29107 apache    20   0 72628  23m  14m S  0.3  2.4   0:00.66 httpd                                                                  

29185 apache    20   0 72484  17m 8516 S  0.3  1.8   0:00.51 httpd                                                                  

29191 apache    20   0 72080  17m 8544 S  0.3  1.7   0:00.38 httpd 
 
 

# ps -ylC httpd --sort:rss

S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD

S    48 29047 29045  0  80   0  5488  6761 -      ?        00:00:00 httpd

S     0 29045     1  0  80   0 16444 17771 -      ?        00:00:00 httpd

S    48 29191 29045  0  80   0 17720 18020 semtim ?        00:00:00 httpd

S    48 29185 29045  0  80   0 18096 18121 semtim ?        00:00:00 httpd

S    48 29234 29045  0  80   0 21912 18020 semtim ?        00:00:00 httpd

S    48 29059 29045  0  80   0 22064 18018 semtim ?        00:00:00 httpd

S    48 29052 29045  0  80   0 22072 18024 semtim ?        00:00:00 httpd

S    48 29061 29045  0  80   0 22440 18026 semtim ?        00:00:01 httpd

S    48 29063 29045  0  80   0 22792 18121 semtim ?        00:00:00 httpd

S    48 29051 29045  0  80   0 26284 18022 semtim ?        00:00:00 httpd

S    48 29064 29045  0  80   0 26668 18120 semtim ?        00:00:00 httpd

S    48 29054 29045  0  80   0 27700 18028 -      ?        00:00:01 httpd

S    48 29058 29045  0  80   0 27708 18030 semtim ?        00:00:00 httpd

S    48 29060 29045  0  80   0 28360 18161 semtim ?        00:00:00 httpd

S    48 29062 29045  0  80   0 28800 18500 semtim ?        00:00:01 httpd

S    48 29049 29045  0  80   0 31052 19041 semtim ?        00:00:01 httpd

S    48 29056 29045  0  80   0 31160 18659 -      ?        00:00:01 httpd

S    48 29048 29045  0  80   0 31888 19193 semtim ?        00:00:01 httpd

S    48 29053 29045  0  80   0 31928 18032 semtim ?        00:00:00 httpd

S    48 29055 29045  0  80   0 31960 18031 semtim ?        00:00:01 httpd

S    48 29107 29045  0  80   0 32756 18157 -      ?        00:00:00 httpd

S    48 29087 29045  0  80   0 34904 19316 semtim ?        00:00:00 httpd

Open in new window

0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24051051
1. First off all, problem is prefork, server is runing out of worker process.

so you need to increase that.

but one question :

1.why suddently server is running out of worker process, that mean, some time, your serer is getting too much its from outside

why ??

is those realy hits or some one tryign to mess with the server such as brutal attack ??

only way you can check this to see Server log file always to see the hits is getting are those real hits ..
2. when this problem happennding, check how many apache process is running on the server .

3. when this problem is happenning, check top command for memory user and processor use

if you run top, you wil see this kind of output

Cpu(s):  1.2%us,  0.0%sy,  0.0%ni, 94.7%id,  0.3%wa,  0.0%hi,  0.0%si,  3.8%st
Mem:    524464k total,   506864k used,    17600k free,   123764k buffers
Swap:   262136k total,       32k used,   262104k free,   264220k cached

also : how often this problem happenning ??



0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24051194
have a read about apache article about prefork setting

http://httpd.apache.org/docs/2.2/mod/prefork.html

default prefork setting is able to hand 256 simultaneous request..
0
 

Author Comment

by:iceman19330
ID: 24051211
The problem happens at different times, sometimes it might be a couple of weeks, other times it might happen on back to back days.  I dont think it is necessarily malicious as these are a collection of decent (200-300 student) private religious schools, several of them average 30k visits a month, and the problems (not 100% sure) seem to occur early to mid evening.

i will check on the other stuff when a problem occurs and report back.
0
 

Author Comment

by:iceman19330
ID: 24051248
BTW this is what top has now.

top - 10:52:13 up 64 days,  1:15,  1 user,  load average: 0.06, 0.03, 0.02
Tasks: 116 total,   2 running, 114 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1022880k total,  1005824k used,    17056k free,   136916k buffers
Swap:  2056280k total,    28952k used,  2027328k free,   608676k cached

My other question would be, is there an alert system that I can setup that will tell me when this occurs, rather then having to keep checking the sites every little while.  I tried a system that would wget the html pages and if it returned a blank page then it would text me, but then that seemed a bit sketchy since it was dependent on another server to run and etc.
0
 
LVL 29

Accepted Solution

by:
fosiul01 earned 500 total points
ID: 24051360

You can use Monit to monitor httpd service it will alert you if there is any problem and will restart the server

http://www.howtoforge.com/server-monitoring-with-munin-and-monit-on-centos-5.2

omm i dont think its memory, as the log said , you need to increate the prefork beause you are runnign out

so when this problem happen, try to incrase your prefork setting , see if that help

but your free ram is low... when this problem happended, check the Ram and processor utilization, just check if they are not running out ...
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:iceman19330
ID: 24055677
So it happened tonight and here is what I was able to get.

top - 18:18:05 up 64 days,  8:41,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 107 total,   1 running, 106 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1022880k total,   884356k used,   138524k free,   180332k buffers
Swap:  2056280k total,    28952k used,  2027328k free,   523780k cached

2466 apache    20   0 72632  27m  18m S  2.0  2.8   0:01.20 httpd

ps -ylC httpd --sort:rss
S   UID   PID  PPID  C PRI  NI   RSS  S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S    48 32472 32470  0  80   0  5488  6761 -      ?        00:00:00 httpd
S    48  3034 32470  0  80   0 10108 17805 semtim ?        00:00:00 httpd
S     0 32470     1  0  80   0 16440 17771 -      ?        00:00:00 httpd
S    48  2974 32470  0  80   0 17892 18114 semtim ?        00:00:00 httpd
S    48  2978 32470  0  80   0 17928 18114 semtim ?        00:00:00 httpd
S    48  2979 32470  0  80   0 17932 18119 semtim ?        00:00:00 httpd
S    48  2976 32470  0  80   0 17976 18118 semtim ?        00:00:00 httpd
S    48  2453 32470  0  80   0 22008 18020 -      ?        00:00:01 httpd
S    48  2715 32470  0  80   0 22420 18122 semtim ?        00:00:00 httpd
S    48  2355 32470  0  80   0 27652 18028 semtim ?        00:00:00 httpd
S    48  2149 32470  0  80   0 27676 18028 semtim ?        00:00:01 httpd
S    48  2243 32470  0  80   0 28148 18032 semtim ?        00:00:01 httpd
S    48  2464 32470  0  80   0 28188 18161 semtim ?        00:00:01 httpd
S    48  2466 32470  0  80   0 28188 18158 semtim ?        00:00:01 httpd

I could not get into server-status, but that is because I was logging on from home and I only allow access from work.  *head slap*.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24058488
Ok
have you tryed to increase the prefork setting ?? [ its wired is not it at middle of the night apache is running out of worker process ?? ]

allow Server-status report to get access from your Home IP, so you would be able to see report from home

when this problem will happended, will you able to check, how much Established connection in your server ??

lsof -i TCP:80   : to check status of port 80

lsof -i TCP:80 | grep ESTABLISHED  = to see established connection

to get continuous you might consider to run this command in crontab to run every minutes and insert report in a text file ..

/var/log/message/httpd/website log  = see what kind of hits you getting



0
 

Author Comment

by:iceman19330
ID: 24059821
what would you suggest for an increase in the prefork settings?
0
 

Author Comment

by:iceman19330
ID: 24060137
So I got monit working because I am on a Plesk server I had it already however it was mis-configured so I redid some of the settings and it seems to be going.

[CDT Apr  3 08:34:07] 'apache' total mem amount of 584312kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:39:09] 'apache' total mem amount of 607332kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:44:11] 'apache' total mem amount of 611556kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:49:13] 'apache' total mem amount of 615944kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:54:16] 'apache' total mem amount of 608428kB matches resource limit [total mem amount>512001kB]
[CDT Apr  3 08:54:16] Sendmail error: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
[CDT Apr  3 08:54:16] 'apache' trying to restart
[CDT Apr  3 08:54:16] 'apache' stop: /usr/local/psa/admin/bin/websrvmng
[CDT Apr  3 08:54:17] 'apache' start: /usr/local/psa/admin/bin/websrvmng
[CDT Apr  3 08:54:17] 'apache' failed, cannot open a connection to INET[www.fakedomain.com:80]
[CDT Apr  3 08:54:17] Sendmail error: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
[CDT Apr  3 08:54:17] 'apache' trying to restart
[CDT Apr  3 08:59:19] 'apache' resource passed
[CDT Apr  3 08:59:19] Sendmail error: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)
[CDT Apr  3 08:59:19] 'apache' connection passed
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24060229
Ok
I am not aware of this error
apache' total mem amount of 608428kB matches resource limit [total mem amount>512001kB]
i will have to have a look in google what does it mean

but  your prefork setting is allright
the default prefork setting apache can handle 256 simulatanous connection a  a time.  which is a lot, only site who gets more then 256 simlatanous connection need to modify prefork setting

i dont think you get 256 simaltanous true hit in apache

so why this problem ??

1. keep monitor your server, check log, i belived yoru server is under brutal attack.
2. what about apache log ?? hvae you check the hit log, does those log is comming from true user ??
3. if you see you are not gettign too much hits still apache is giving this troblel , then have to think something else

but before that you need to identify , what causing problem

1. attack or apache

if you cant seperate this issue , i would not be able to help you too much rather then just giving some theoritical idea..
0
 

Author Comment

by:iceman19330
ID: 24060273
should I turn on any additional logging or something to trace back what network this traffic is coming from.  most of the site on here are only really known in this region.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24060327
in /var/log/httpd/

cat mydomain_log    [ here i dont know how you configure your apache log for individual domain]

so check all access log to carefully to see what IP you see , what pages are browsing ...


also  lsof -i TCP:80 | wc -l
or lsof -i TCP:80 | grep ESTABLISHED | wc -l

it will show you how many connections are established at a time  , may be when this problem is happeening at that time there is too many connection to your server ..
0
 

Author Comment

by:iceman19330
ID: 24060470
I have a main access_log and then each domain is configured separately.
0
 
LVL 29

Expert Comment

by:fosiul01
ID: 24060576
you know when your system went down

check every log prerior that time
0

Featured Post

Control application downtime with dependency maps

Visualize the interdependencies between application components better with Applications Manager's automated application discovery and dependency mapping feature. Resolve performance issues faster by quickly isolating problematic components.

Join & Write a Comment

In my time as an SEO for the last 2 years and in the questions I have assisted with on here I have always seen the need to redirect from non-www urls to their www versions. For instance redirecting http://domain.com (http://domain.com) to http…
Hi, in this article I'm going to teach you how to run your own site, and how to let people in (without IP). I'll talk about and explain each step... :) By the way, everything in this Tutorial is completely free and legal. This article is for …
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now