http connection failure

Hi experts,

I would appreciate if you could help. Currently we're experiencing problems in our website. The website is load balanced with cisco ACE module, Virtual IP has been NAT'ed outside and was working well till now. There are 2 web servers in serverfarm ( both are in DMZ). Ping is working ok to all these separate servers. (note that web servers are on AIX)

ACE is configured with Http probe, when i'm troubleshooting the problem from ACE, what i notice is that when connection number goes up, ACE stops receiving 200 ok message from both servers (sometimes one will fail, other will fail later). After http probe failure, Virtual IP becomes outofservice and at the result website becomes unavailable.

When one of our admins restart apache and tomcat in that server, http probe will go back to success state and Virtual IP will come back again. Failures happen periodically.

Could you please help me finding out why http connection fails with individual servers? How can I fix the problem? Is there anything to do with ACE configuration?


Thank you very much,
Appreciate your efforts.
LVL 1
aliyevAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

giltjrCommented:
The problem appears to have nothing to do with ACE.  

If the solution to the problem is to restart Apache/Tomcat then they are having a problem and get to the point where they can't respond to requests.

ACE is doing exactly what it should do.  Check to see if the server is responding.  If it is, leave it marked as up and send requests to it.  If it stops responding, then there is a problem and ACE marks it as down.   If all servers in that pool stop responding, then mark them all down and mark the VIP down.
0
aliyevAuthor Commented:
Yeah, it's exactly what i've reported to sys admin team. However, they've failed to correctly identify and fix the problem. By the way apache and tomcat are on AIX servers. Any recommended troubleshooting steps?
0
Bernard S.CTOCommented:
First step would probably to carefully inspect the logs...
0
Newly released Acronis True Image 2019

In announcing the release of the 15th Anniversary Edition of Acronis True Image 2019, the company revealed that its artificial intelligence-based anti-ransomware technology – stopped more than 200,000 ransomware attacks on 150,000 customers last year.

giltjrCommented:
I  agree with fibo,  I would look at the tomcat logs first. Apache is most likely hanging because of something in tomcat.  If you application accesses a data base, tomcat may end up pointing to the database.
0
aliyevAuthor Commented:
Dear giltjr and fibo, thanks very much for your comments and help. As i'm not an expert in unix, apache tomcat systems i'm not sure whether I will be able to send you correct logs.

Could you please guide me on this? Which commands to use? where to look. Also to my knowledge problem could occur because of AIX OS also. Could you also please send me some troubleshooting steps on AIX also ( commands to lookup for logs etc.)

Thanks in advance
0
Bernard S.CTOCommented:
1 - Usually, most logs are placed in /var/log and apache2 logs are in /var/log/apache2
(your web site is probably in /var/www/ or in /var/www/mysite or similar.)

2 - depending on your own config, you might (or not) access to them with your ftp client, better would be a terminal access (ie, ssh or putty). otherwise you'll need to ask some assistance to get your files

3 - the files that would be helpful at first would be for apache access.log and error.log (go to the end of the files!)
if you have a terminal access, you can directly type
tail /var/log/apache2/access.log
and
tail /var/log/apache2/error.log
0
aliyevAuthor Commented:
Dear fibo thanks for your guide, i could not find any apache folder in var/log path, hence i could find it in another path. Please see below pasted log messages. Also please see screenshot to have an idea what folders we have in /usr/apache path. Thanks in advance

bash-3.00# tail /usr/apache/logs/error_log  
[Thu Oct 03 19:18:57 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Fri Oct 04 08:55:10 2013] [notice] caught SIGTERM, shutting down
[Fri Oct 04 08:57:31 2013] [notice] Apache/2.2.20 (Unix) mod_jk/1.2.32 configured -- resuming normal operations
[Fri Oct 04 11:31:32 2013] [notice] SIGHUP received.  Attempting to restart
[Fri Oct 04 11:31:32 2013] [notice] Apache/2.2.20 (Unix) mod_jk/1.2.32 configured -- resuming normal operations
[Fri Oct 04 11:47:12 2013] [notice] caught SIGTERM, shutting down
[Fri Oct 04 16:39:55 2013] [notice] Apache/2.2.20 (Unix) mod_jk/1.2.32 configured -- resuming normal operations
[Tue Oct 08 12:52:40 2013] [notice] caught SIGTERM, shutting down
[Tue Oct 08 12:53:32 2013] [notice] Apache/2.2.20 (Unix) mod_jk/1.2.32 configured -- resuming normal operations
[Tue Oct 08 15:51:23 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
You have mail in /usr/spool/mail/root

Open in new window


This is from tomcat.

bash-3.00# tail -f catalina.2013-10-09.log 
Oct 9, 2013 10:58:17 PM org.apache.tomcat.util.http.Parameters processParameters
WARNING: Parameters: Invalid chunk '' ignored.
Oct 9, 2013 11:26:12 PM org.apache.tomcat.util.http.Parameters processParameters
WARNING: Parameters: Invalid chunk '' ignored.
Oct 9, 2013 11:34:54 PM org.apache.tomcat.util.http.Parameters processParameters
WARNING: Parameters: Invalid chunk '' ignored.
Oct 9, 2013 11:54:05 PM org.apache.tomcat.util.http.Parameters processParameters
WARNING: Parameters: Invalid chunk '' ignored.
Oct 9, 2013 11:57:11 PM org.apache.tomcat.util.http.Parameters processParameters
WARNING: Parameters: Invalid chunk '' ignored.

Open in new window

ScreenClip.png
0
giltjrCommented:
First log from Apache shows exactly what I expected:

server reached MaxClients setting, consider raising the MaxClients setting

This means Apache had the maximum number of concurrent connections that it was configured to have and once it hits that it stops accepting new connections.  Depending on your hardware (CPU speed, number of CPU's, and amount of RAM) you might be able to increase the number of max connections.

As for the tomcat log, you may need to look at more of the log.  The last few lines really don't say a lot other than an application is trying to do something on an empty array/string.

Is tomcat running a custom written program or an off the shelf app?

If custom, then the application development team needs to look at the tomcat logs.
0
aliyevAuthor Commented:
Hi giltjr, can you please help me on increasing maxclient settings? what commands should i use? where should i type it?
0
giltjrCommented:
I've never run Apache under AIX, but I think the configuration file should be:

/usr/local/apache2/conf/httpd.conf

However, you can't just change this number.  We would need to know:

1) CPU type/speed.
2) Number of CPU's
3) Amount or RAM
4) Avg. CPU utlization
5) Avg. amount of RAM used that is NOT cache/buffers.
6) Expected number of concurrent users.
0
Bernard S.CTOCommented:
Seems that it is not an apache2 but apache 1
you'll probably find apache http.conf under /usr/apache/
0
aliyevAuthor Commented:
this is what can i tell you

 prtconf
System Model: IBM,8203-E4A
Machine Serial Number: 06788A5
Processor Type: PowerPC_POWER6
Processor Implementation Mode: POWER 6
Processor Version: PV_6_Compat
Number Of Processors: 4
Processor Clock Speed: 4695 MHz
CPU Type: 64-bit
Kernel Type: 64-bit
LPAR Info: 2 WBS1
Memory Size: 3712 MB
Good Memory Size: 3712 MB
Platform Firmware level: EL350_039
Firmware Version: IBM,EL350_039
Console Login: enable
Auto Restart: true
Full Core: false

cpu usage is 1 % and please see attached file for memory. Number of expected concurrent users changes from 50 - 120

Thanks in advance
ScreenClips.png
0
giltjrCommented:
Do you monitor the CPU utilization with product so you can get avg. as well as peak?

Does tomcat run on this same server?
0
aliyevAuthor Commented:
I just used nmon. cpu seems to be idle always. Tomcat runs on the same server.
0
giltjrCommented:
O.K, just need the httpd.conf file then.  Make sure you mask out any information you don't want the public to see (userid, public IP addresses, ect.).
0
aliyevAuthor Commented:
Dear giltjr, here is my httpd.conf file in usr/apache/conf folder. I've changed folder names and ip addresses to pseudo addresses and names.

#
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.2> for detailed information.
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "logs/foo_log"
# with ServerRoot set to "/usr/apache" will be interpreted by the
# server as "/usr/apache/logs/foo_log".

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# Do not add a slash at the end of the directory path.  If you point
# ServerRoot at a non-local disk, be sure to point the LockFile directive
# at a local disk.  If you wish to share the same ServerRoot for multiple
# httpd daemons, you will need to change at least LockFile and PidFile.
#
ServerRoot "/usr/apache"

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses.
#
#Listen 12.34.56.78:80
Listen 192.168.0.1:80

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
LoadModule jk_module modules/mod_jk.so
JkWorkersFile /usr/apache/conf/workers.properties
JkShmFile /usr/apache/logs/mod_jk.shm
JkLogFile /usr/apache/logs/mod_jk.log
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
#JkMount /examples/* worker1
#JKMount /mywebsite/* worker1
JKMount /* worker1

<IfModule !mpm_netware_module>
<IfModule !mpm_winnt_module>
#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.  
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User nobody 
Group nobody 

</IfModule>
</IfModule>

# 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition.  These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. admin@your-domain.com
#
ServerAdmin you@example.com

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
#ServerName www.example.com:80
ServerName 192.168.0.1
#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
#DocumentRoot "/usr/apache/htdocs"

DocumentRoot "/usr/local/tomcat/webapps/mywebsite"

#
# Each directory to which Apache has access can be configured with respect
# to which services and features are allowed and/or disabled in that
# directory (and its subdirectories). 
#
# First, we configure the "default" to be a very restrictive set of 
# features.  
#
<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# This should be changed to whatever you set DocumentRoot to.
#
<Directory "/usr/apache/htdocs">
    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.2/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks

    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride None

    #
    # Controls who can get stuff from this server.
    #
    Order allow,deny
    Allow from all

</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<FilesMatch "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</FilesMatch>

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog "logs/error_log"

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    # CustomLog "logs/access_log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog "logs/access_log" combined
</IfModule>

<IfModule alias_module>
    #
    # Redirect: Allows you to tell clients about documents that used to 
    # exist in your server's namespace, but do not anymore. The client 
    # will make a new request for the document at its new location.
    # Example:
    # Redirect permanent /foo http://www.example.com/bar

    #
    # Alias: Maps web paths into filesystem paths and is used to
    # access content that does not live under the DocumentRoot.
    # Example:
    # Alias /webpath /full/filesystem/path
    #
    # If you include a trailing / on /webpath then the server will
    # require it to be present in the URL.  You will also likely
    # need to provide a <Directory> section to allow access to
    # the filesystem path.

    #
    # ScriptAlias: This controls which directories contain server scripts. 
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the target directory are treated as applications and
    # run by the server when requested rather than as documents sent to the
    # client.  The same rules about trailing "/" apply to ScriptAlias
    # directives as to Alias.
    #
    ScriptAlias /cgi-bin/ "/usr/apache/cgi-bin/"

</IfModule>

<IfModule cgid_module>
    #
    # ScriptSock: On threaded servers, designate the path to the UNIX
    # socket used to communicate with the CGI daemon of mod_cgid.
    #
    #Scriptsock logs/cgisock
</IfModule>

#
# "/usr/apache/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/usr/apache/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

#
# DefaultType: the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain

<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig conf/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    # For type maps (negotiated resources):
    #AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    #AddType text/html .shtml
    #AddOutputFilter INCLUDES .shtml
</IfModule>

#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type.  The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
#MIMEMagicFile conf/magic

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# EnableMMAP and EnableSendfile: On systems that support it, 
# memory-mapping or the sendfile syscall is used to deliver
# files.  This usually improves server performance, but must
# be turned off when serving from networked-mounted 
# filesystems or if support for these functions is otherwise
# broken on your system.
#
#EnableMMAP off
#EnableSendfile off

# Supplemental configuration
#
# The configuration files in the conf/extra/ directory can be 
# included to add extra features or to modify the default configuration of 
# the server, or you may simply copy their contents here and change as 
# necessary.

# Server-pool management (MPM specific)
#Include conf/extra/httpd-mpm.conf

# Multi-language error messages
#Include conf/extra/httpd-multilang-errordoc.conf

# Fancy directory listings
#Include conf/extra/httpd-autoindex.conf

# Language settings
#Include conf/extra/httpd-languages.conf

# User home directories
#Include conf/extra/httpd-userdir.conf

# Real-time info on requests and configuration
#Include conf/extra/httpd-info.conf

# Virtual hosts
#Include conf/extra/httpd-vhosts.conf

# Local access to the Apache HTTP Server Manual
#Include conf/extra/httpd-manual.conf

# Distributed authoring and versioning (WebDAV)
#Include conf/extra/httpd-dav.conf

# Various default settings
#Include conf/extra/httpd-default.conf

# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>

Open in new window

0
aliyevAuthor Commented:
Guys, i also found httpd-mpm.conf file under usr/apache/conf/extra folder. I'm inserting it as it is.  Might be useful. To my knowledge this module should be inserted into httpd.conf.
But i couldn't find any line about this particular module. May this cause a problem?

#
# Server-Pool Management (MPM specific)
# 

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
#
# Note that this is the default PidFile for most MPMs.
#
<IfModule !mpm_netware_module>
    PidFile "logs/httpd.pid"
</IfModule>

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
<IfModule !mpm_winnt_module>
<IfModule !mpm_netware_module>
LockFile "logs/accept.lock"
</IfModule>
</IfModule>

#
# Only one of the below sections will be relevant on your
# installed httpd.  Use "apachectl -l" to find out the
# active mpm.
#

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          350
    MaxRequestsPerChild   0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          2
    MaxClients          350
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

# BeOS MPM
# StartThreads: how many threads do we initially spawn?
# MaxClients:   max number of threads we can have (1 thread == 1 client)
# MaxRequestsPerThread: maximum number of requests each thread will process
<IfModule mpm_beos_module>
    StartThreads            10
    MaxClients              50
    MaxRequestsPerThread 10000
</IfModule>

# NetWare MPM
# ThreadStackSize: Stack size allocated for each worker thread
# StartThreads: Number of worker threads launched at server startup
# MinSpareThreads: Minimum number of idle threads, to handle request spikes
# MaxSpareThreads: Maximum number of idle threads
# MaxThreads: Maximum number of worker threads alive at the same time
# MaxRequestsPerChild: Maximum  number of requests a thread serves. It is 
#                      recommended that the default value of 0 be set for this
#                      directive on NetWare.  This will allow the thread to 
#                      continue to service requests indefinitely.                          
<IfModule mpm_netware_module>
    ThreadStackSize      65536
    StartThreads           250
    MinSpareThreads         25
    MaxSpareThreads        250
    MaxThreads            1000
    MaxRequestsPerChild      0
    MaxMemFree             100
</IfModule>

# OS/2 MPM
# StartServers: Number of server processes to maintain
# MinSpareThreads: Minimum number of idle threads per process, 
#                  to handle request spikes
# MaxSpareThreads: Maximum number of idle threads per process
# MaxRequestsPerChild: Maximum number of connections per server process
<IfModule mpm_mpmt_os2_module>
    StartServers           2
    MinSpareThreads        5
    MaxSpareThreads       10
    MaxRequestsPerChild    0
</IfModule>

# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server process
# MaxRequestsPerChild: maximum  number of requests a server process serves
<IfModule mpm_winnt_module>
    ThreadsPerChild      150
    MaxRequestsPerChild    0
</IfModule>

Open in new window

0
giltjrCommented:
So that I am not assuming, please find the file apachctl, change to that directory and issue the command:

./apachectl -V

Make sure the V is in upper case.  Need to know if you are in prefork (should be) or worker.
0
aliyevAuthor Commented:
ok. here is the result. also i issued ./apachectl -M. might be helpful

bash-3.00# ./apachectl -V
Server version: Apache/2.2.20 (Unix)
Server built:   Sep  2 2011 12:08:56
Server's Module Magic Number: 20051115:28
Server loaded:  APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/apache"
 -D SUEXEC_BIN="/usr/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"


bash-3.00# ./apachectl -M
Loaded Modules:
 core_module (static)
 authn_file_module (static)
 authn_default_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_default_module (static)
 auth_basic_module (static)
 include_module (static)
 filter_module (static)
 log_config_module (static)
 env_module (static)
 setenvif_module (static)
 version_module (static)
 mpm_prefork_module (static)
 http_module (static)
 mime_module (static)
 status_module (static)

Open in new window

0
giltjrCommented:
O.K.  You need to change the prefork section in httpd-mpm.conf

<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          350
    MaxRequestsPerChild   0
</IfModule>


Although you have MaxClients at 350, you not going to get more than 256.  This is because the default max for ServerLimit is 256 and is the true limit.

So I would start off by adding ServerLimit 350 to the above section.

However, since you stated that the max concurrent users you expect is 120, this number is well above that so I don't think changing the ServerLimit to 350 will really do anything, but it is worth a try.

You still need to look at tomcat and see what its max's are and what it is hitting.  I really don't know tomcat, so not sure exactly where to look for logs and for configuration.

You may also want to read this and set it up so you can get the Apache Stats.

http://httpd.apache.org/docs/2.2/mod/mod_status.html
0
aliyevAuthor Commented:
Hi,

Just edited the file. Http server doesnt respond to http requests at all. insterted new line just after start servers and put value 500. both to serverlimit and maxclients.
0
aliyevAuthor Commented:
rebooted the server. its working now. lets see what happens. normally in peak hours it would   go down.
0
giltjrCommented:
If it still has problems. I would start focusing on tomcat.
0
aliyevAuthor Commented:
:( unfortunately, still has the problem. Would it be helpful to restart the tomcat and see if it helps?
0
giltjrCommented:
Assuming that you are still seeing reaching MaxClients probably not.  As Apache is just acting as a "front" end to tomcat, my guess is there is a problem in tomcat.

You need to start looking at those logs and have the application developers get involved so they can see what errors are occurring.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Apache Web Server

From novice to tech pro — start learning today.