Start Free Trial

asked on

com.sun.net.httpserver.HttpServer leaking HTTP connections

Hello,
I have a basic HTTP server inside my java application.
It's supposed to handle a big load of HTTP request from many IPs, like a server would.
I'm using com.sun.net.httpserver.HttpServer, and starting it as follows.

In perfmon, the [TCPv4/Connections Established] counter keeps growing.

In netmon, there is a HUGE amount of TIME_WAIT connections from local:80 to remote:variable, a small but worth mention amount of FIN_WAIT_2 connections local:80 to remote:variable, some ESTABLISHED local:80 to remote:variable
and finally, LOTS of "TCP 127.0.0.1:7225 127.0.0.1:7313 ESTABLISHED" (local:variable to local:variable)

In the task manager, I see the Handle Count of the process that handles HTTP connections is growing. It does not reset when I shut down the web server class, but only when I close the process.

Any suggestions?
Thanks!

InetSocketAddress addr = new InetSocketAddress(WEB_SERVER_PORT);
server = HttpServer.create(addr, 1000);
MyHandler oHttpHandler = new MyHandler();
server.createContext("/", oHttpHandler);
ThreadPoolExecutor executor = new ThreadPoolExecutor(50, 200, 60,
	TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(1000));
server.setExecutor(executor);
server.start();
 
 
class MyHandler implements HttpHandler {
 
	public MyHandler() {
	}
 
	public void handle(HttpExchange exchange) throws IOException {
 
		String requestMethod = exchange.getRequestMethod();
		if (requestMethod.equalsIgnoreCase("GET")) {
 
			// reply 200 OK
			Headers responseHeaders = exchange.getResponseHeaders();
			responseHeaders.set("Content-Type", "text/plain");
			exchange.sendResponseHeaders(200, 0);
 
			OutputStream osResponseBody = exchange.getResponseBody();
 
			// <do some processing, irrelevant as issue occurs if nothing here>
 
			osResponseBody.close();
		}
		exchange.close();
	}

Open in new window

That's a problem often seen in WIndows:

http://kamranzaidi.com/2006/11/02/setting-the-tcp-time_wait-time/

ASKER

But aren't the HTTP connections supposed to be closed and cleared as soon as the server sends it's response?

Its nothing to do with http, its a TCP/IP issue

actually that may not be entirely correct.
you could try disabling keepalive, nnot sure its ultimately what you want to be doing though

ASKER

Well I only get requests from clients, process them, and have no intention of keeping a connection alive or saving a session.
How can I disable KeepAlive from the server-side in com.sun.net.httpserver.HttpServer?

ASKER

Trying "Connection: close" header
from http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.10

You could have your own SocketFactory and set the keep alive parameters. The TCP/IP parameters are an OS-level issue

You can also set keep alive and max connections thru properties:

http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html

ASKER

CEHJ:

Re SocketFactory - do you have a code example?
Re properties - can I override it just on the level of this server, and not globally in Java?

you can set properties using the command line when you start up the vm (so they will be vm wide)

example here:

http://helpdesk.objects.com.au/java/how-to-tell-a-java-application-to-use-a-proxy-server

>>Re properties - can I override it just on the level of this server, and not globally in Java?

No it will be global

You don't actually need to use a ScoketFactory if you don't want. Just call the Socket methods to alter keepalive/timeout etc.

Is this actually causing you a problem or is it just not what you expected to see?

ASKER

I can't change things globally for TCP, as I have other (non-HTTP) connection handling in the application, which I don't want to change.

This is causing a huge problem, when the process crashes with an ugly out of memory exception due to lack of ability to allocate more connections or threads.

whats linking the OOM exception to TIME_WAIT connections?
Have you profiled it?

ASKER

I have profiled the TCPv4 Connections Established counter (after checking, this is the one that grows), and when it reached the peak value of 28,739, the application, still trying to accept and process HTTP requests, crashed with the following exception.

Exception in thread "pool-37-thread-36" java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)
        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34
)
        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)
        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)
        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid
er.java:26)
        at java.nio.channels.Selector.open(Selector.java:209)
        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)
        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)
        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
        at java.lang.Thread.run(Thread.java:619)
Exception in thread "pool-37-thread-24" java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)
        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34
)
        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)
        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)
        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid
er.java:26)
        at java.nio.channels.Selector.open(Selector.java:209)
        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)
        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)
        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
        at java.lang.Thread.run(Thread.java:619)
Exception in thread "pool-37-thread-31" java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)
        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34
)
        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)
        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)
        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid
er.java:26)
        at java.nio.channels.Selector.open(Selector.java:209)
        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)
        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)
        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
        at java.lang.Thread.run(Thread.java:619)
Exception in thread "pool-37-thread-1" java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)
        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34
)
        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)
        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)
        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid
er.java:26)
        at java.nio.channels.Selector.open(Selector.java:209)
        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)
        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)
        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
        at java.lang.Thread.run(Thread.java:619)
Exception in thread "pool-37-thread-39" java.lang.OutOfMemoryError
        at sun.misc.Unsafe.allocateMemory(Native Method)
        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)
        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34
)
        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)
        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)
        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid
er.java:26)
        at java.nio.channels.Selector.open(Selector.java:209)
        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)
        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)
        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
        at java.lang.Thread.run(Thread.java:619)
#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 24848 bytes for Handle in C:\BUILD_AREA\
jdk6_11\hotspot\src\share\vm\prims\jvmtiGetLoadedClasses.cpp. Out of swap space?
 
#
#  Internal Error (allocation.inline.hpp:42), pid=14728, tid=8136
#  Error: Handle in C:\BUILD_AREA\jdk6_11\hotspot\src\share\vm\prims\jvmtiGetLoa
dedClasses.cpp
#
# Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode windows-x86)
# An error report file with more information is saved as:
# hs_err_pid14728.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

Open in new window

ASKER

CEHJ:
Where can I call the Socket methods to affect the sun http server bahaviour?

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

Yeah I'm using a Thread Pool. Wrote it in the first message!

ThreadPoolExecutor executor = new ThreadPoolExecutor(50, 200, 60,
        TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(1000));

Open in new window

that doesn't look like standard behaviour to me. how much traffic are you dealing with?
have you tried running it on a different box?

ASKER

I'm dealing with huge traffic, counting impressions, at few tens of requests per second.
This is already a dedicated box running just it, to separate from any other possible issues on a box doing other stuff as well.

(I have to ask) why aren't you using apache?

ASKER

(I know you do) Because I need to write something directly to my application's memory, and converting has an overload. First tried with IIS, but it was too heavy.
I hope to solve this now as it is. If I can't, I'll look what's need to move this into Apache, or Webster.

how long is an average request? 20K open connections could well be expected under those sort of load.
Have you tried throwing more memory at it?
And the effect of tuning your connection pool

ASKER

A single request is very short. It parses the URL, writes the params in memory, and returns. Nothing to wait for, no content is even returned to the user.

I don't think memory is the issue, as with whatever load of requests, the amount of open connections should stabilize at some point, and not grow out of control and crash the server.

What tuning of the connection pool? If you mean the suggested global changes in registry or java settings, I can't do that there. If you mean other changes, please direct me.

Try lowering the thread pool keepAliveTime

Actually - ignore that last comment

> I don't think memory is the issue

the vm is running out of memory :)

> What tuning of the connection pool?

eg. its size

ASKER

It's running out of memory because it is not releasing connections ever, not because any certain amount of memory or pool size would suffice...

it has released the connections, its tcp/ip that is holding onto them by the look
If you stop hitting the server do the connections eventually go away?
Whats the TIMED_WAIT period set to on that box?

ASKER

I cannot stop hitting the server, it gets connections from the internet all the time.

Where do I check the TIMED_WAIT period currently set?

I meant in testing.

Try blocking connections by decreasing the queue size and let us know if it's still running out of memory

ASKER CERTIFIED SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

as I mentioned above it isn't the client thats keeping it open :)

SOLUTION

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ASKER

Yeah so good I moved to Jetty, if Sun's server is buggy, was reported two years ago and not handled.

However, even in Jetty, without flushing and closing the connection, it was the client still holding it open after the server didn't send anything more.

Its not a bug (at least according to Sun), as explained in the bug report

good to hear jetty is working for you, better product anyway.