Solved

Problems with IIS, isapi_redirect.dll and Tomcat under high load

Posted on 2007-11-28
10
1,385 Views
Last Modified: 2013-12-02
I have a Java application running on Tomcat 5.5 on a Win2k box. The box runs IIS 5 connected to Tomcat using JK 1.2. During periods of our highest load (around 50 page requests per minute) we have a failure. User's get a browser error (I think it's page unavailable) looking like the application has gone down.

In the IIS logs I can see requests still coming in as expected, and IIS still serves up images ok. Also if I log into the server and connect to Tomcat directly on port 8080 I can load the application (albeit without images) and confirm it is still running correctly.

The only way to rectify the fault is to restart Tomcat, even though this seems to still be running ok

I have turned on logging to debug for the JK connector but can't see anything useful in the logs. I have also tried upgrading JK to the latest version of the DLL (7th aug '07).

If anyone has any recommendations for what I can try I'd be very grateful. Thanks
0
Comment
Question by:lloyd142
  • 5
  • 5
10 Comments
 
LVL 27

Expert Comment

by:mrcoffee365
ID: 20370972
We have run that configuration, and 50 requests per minute doesn't sound like too much for the IIS-Tomcat connector.

However, the isapi jk connector behaves oddly, in my opinion.  There are some ways to increase the connections between IIS and Tomcat which might help (didn't make a lot of difference for us, but we can hope).
Try setting cachsize=100 in your workers.properties file:
worker.ajp13w.cachesize=100

It has been recommended by Mladen Turk for Windows/IIS jk connector configurations (it has to be one of the Windows servers, not XP or other non-server OS).

General doc on connectors:
http://tomcat.apache.org/faq/connectors.html
http://tomcat.apache.org/connectors-doc/


Mladen Turk's article on fronting Tomcat with various Web servers, incl. IIS, is a good one for understanding what the Tomcat and connector developers intended:
http://people.apache.org/~mturk/docs/article/ftwai.html

People in Microsoft land would recommend upgrading to Win2003 server and IIS6, and they have improved the thread and caching model in IIS6.

Have you checked the Tomcat logs to see if you can find anything happening there that seems odd?
0
 

Author Comment

by:lloyd142
ID: 20375183
Thanks very much for your comments and the links. I agree the number of requests is not high at all.

I'll try the cache size change see if that has any effect. If that doesn't, I guess the IIS/Windows upgrade is the next step.

There's nothing unusual in the Tomcat logs. All we see is page requests which suddenly stop with no error. Then after that the only debug is from a scheduled task which is part of the application which keeps going proving Tomcat on its own is fine.

I'll have to wait for a recurrence (or hopefully not) of this issue to see whether any of these solutions has worked

0
 
LVL 27

Expert Comment

by:mrcoffee365
ID: 20376379
Your error actually sounds as if IIS can't make any more connections to Tomcat.  So it might be the IIS-to-isapi connector connection that is the problem.  In which case, you might look at connection times and your keepalive values for the IIS-to-connector configuration.

For example, if all your IIS connections via ISAPI are used up (I'm not sure this is possible, but it might be), then the next request for a connection from IIS to Tomcat will get a 500 or 503 -- internally to IIS or the connector -- until the existing connections have been freed by IIS (or the connector -- again, I'm not sure which is holding the connection here).

If you haven't researched this, David Wang's articles on IIS and ISAPI are very informative.  This is one on running JSP from IIS:
http://blogs.msdn.com/david.wang/archive/2005/10/11/How_does_JSP_work_on_IIS.aspx
0
 

Author Comment

by:lloyd142
ID: 20410198
Hi - thanks again for your reply. The application has been relatively stable recently but failed again yesterday so first I will try and alter the cache size.

How do I select suitable values for connection timeout and keepalive? Do you think I should set all three properties at once (cachesize, cache_timeout, socket_keepalive) and try that or just the cache size for now?

Also I checked yesterday and can confirm it is a 500 error. It's as if Tomcat is unavailable (which it isn't) so agree it looks very much as if IIS can't make another connection to Tomcat
0
 
LVL 27

Expert Comment

by:mrcoffee365
ID: 20412792
Hard to say, but I'd try just changing cachesize first.
0
Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

 

Author Comment

by:lloyd142
ID: 20483869
As an update to this, I've now tried changing the cachesize to 100 with no improvement. I have not yet tried to alter timeout or keepalive, I guess this might be the next step.

 I've also been able to extract some debug from the asapi log file which I hadn't previously been able to see:
[Sat Nov 24 12:52:05.093 2007] [3324:5424] [info] jk_connect.c (473): connect to 127.0.0.1:8009 failed (errno=61)

[Sat Nov 24 12:52:05.093 2007] [3324:5424] [info] jk_ajp_common.c (891): Failed opening socket to (127.0.0.1:8009) (errno=61)

[Sat Nov 24 12:52:05.093 2007] [3324:5424] [info] jk_ajp_common.c (1311): (ajp13) error connecting to the backend server (errno=61)

[Sat Nov 24 12:52:05.093 2007] [3324:5424] [info] jk_ajp_common.c (2085): (ajp13) sending request to tomcat failed,  recoverable operation attempt=1

[Sat Nov 24 12:52:05.093 2007] [3324:5424] [debug] jk_connect.c (357): socket TCP_NODELAY set to On

[Sat Nov 24 12:52:05.093 2007] [3324:5424] [debug] jk_connect.c (455): trying to connect socket 11252 to 127.0.0.1:8009

[Sat Nov 24 12:52:05.406 2007] [3324:5284] [debug] jk_ajp_common.c (1058): received from ajp13 pos=0 len=8188 max=8192

Open in new window

0
 
LVL 27

Expert Comment

by:mrcoffee365
ID: 20494319
These are hard to debug, but we don't get the errors you're getting, with the same W2k/IIS5/Tomcat5.5 architecture.  We have much higher loads at times than you're describing.

Since you're seeing the ISAPI logs, then IIS is making the request to ISAPI, but Tomcat is refusing them (occasionally).

errno 61 seems to mean, usually, that Tomcat isn't started.  

Maybe if you post your workers.properties and uiworkermap.properties files, we can make more suggestions.
0
 

Author Comment

by:lloyd142
ID: 20508429
Tomcat is definitely started up as I can access it directly over port 8080

Here are my workers.properties, server.xml(part of) and uiworkermap.properties. If you can spot any obvious differences between mine and your setup that would be very useful. Thanks
# workers.properties -

workers.tomcat_home=E:\Apache Group\Tomcat 5.5

logfiles_home=E:\logs\tomcat

workers.java_home=D:\Program Files\j2sdk1.4.2_02
 

ps=\

worker.list=ajp13

worker.ajp13.port=8009

worker.ajp13.host=localhost

worker.ajp13.type=ajp13

worker.ajp13.cachesize=100
 
 
 
 
 
 
 

Part of server.xml:
 

<!--

    <Connector port="8443" maxHttpHeaderSize="8192"

               maxThreads="150" minSpareThreads="25" maxSpareThreads="75"

               enableLookups="false" disableUploadTimeout="true"

               acceptCount="100" scheme="https" secure="true"

               clientAuth="false" sslProtocol="TLS" />

    -->
 

    <!-- Define an AJP 1.3 Connector on port 8009 -->

    <Connector port="8009" 

               minProcessors="5" maxProcessors="75" enableLookups="true" 

               redirectPort="8443" acceptCount="20" useURIValidationHack="false" 

               debug="0" connectionTimeout="60000" protocol="AJP/1.3" />
 
 
 

uiworkermap.properties:
 

/broker=ajp13

/broker/*=ajp13

Open in new window

0
 
LVL 27

Expert Comment

by:mrcoffee365
ID: 20509840
We have set some slightly different parameters on the config we have that is closest to yours -- especially the timeout on port 8009.  Since IIS and Tomcat are on the same server on our machine, we don't expect long times for connections between the two.

server.xml:
<Connector className="org.apache.coyote.tomcat5.CoyoteConnector"
               port="8009" minProcessors="5" maxProcessors="75"
               enableLookups="true" redirectPort="8443"
               acceptCount="10" debug="0" connectionTimeout="0"
               useURIValidationHack="false"
               protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler"/>

In your workers.properties file, we don't use the ps parameter.  It's not in the doc:
http://tomcat.apache.org/connectors-doc/reference/workers.html
so I assume that it's ignored.

We do specify load-balancing, but there's only a single Tomcat running in that config, so I don't think it makes a difference.

We set the following values in workers.properties:
worker.ajp13w.connection_pool_size=10
worker.ajp13w.connection_pool_minsize=5
worker.ajp13w.connection_pool_timeout=30

and they don't break, but, like cachesize, I'm not sure we have the configuration to take advantage of these.

It's a little odd that you haven't upgraded your release of Java to 1.5 or 1.6, but it seems likely that that is not causing your problem here.
0
 

Accepted Solution

by:
lloyd142 earned 0 total points
ID: 21323621
Having tinkered with all the settings suggested the solution came from somewhere else. I updated the database driver from an old non-production one to an up to date SQL Server driver. This seemed to stop the IIS server from crashing. It has been suggested that the previous database driver was crashing which in turn caused IIS or Tomcat to crash. Thanks for all the comments that have been made on this issue
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
split53 challenge 7 77
firstswap challenge 20 65
ASP.Net Session State alternatives 3 57
javap bin 2 25
Prologue It is often required to host multiple websites on a single instance of IIS, mostly in development environments instead of on production servers. I am sure it is not much a preferred solution on production servers but this is at least a pos…
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Viewers learn how to read error messages and identify possible mistakes that could cause hours of frustration. Coding is as much about debugging your code as it is about writing it. Define Error Message: Line Numbers: Type of Error: Break Down…
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:

914 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now