Solved

com.sun.net.httpserver.HttpServer leaking HTTP connections

Posted on 2009-05-18
38
4,074 Views
Last Modified: 2013-11-21
Hello,
I have a basic HTTP server inside my java application.
It's supposed to handle a big load of HTTP request from many IPs, like a server would.
I'm using com.sun.net.httpserver.HttpServer, and starting it as follows.

In perfmon, the [TCPv4/Connections Established] counter keeps growing.

In netmon, there is a HUGE amount of TIME_WAIT connections from local:80 to remote:variable, a small but worth mention amount of FIN_WAIT_2 connections local:80 to remote:variable, some ESTABLISHED local:80 to remote:variable
and finally, LOTS of   "TCP 127.0.0.1:7225  127.0.0.1:7313  ESTABLISHED" (local:variable to local:variable)

In the task manager, I see the Handle Count of the process that handles HTTP connections is growing. It does not reset when I shut down the web server class, but only when I close the process.

Any suggestions?
Thanks!
InetSocketAddress addr = new InetSocketAddress(WEB_SERVER_PORT);

server = HttpServer.create(addr, 1000);

MyHandler oHttpHandler = new MyHandler();

server.createContext("/", oHttpHandler);

ThreadPoolExecutor executor = new ThreadPoolExecutor(50, 200, 60,

	TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(1000));

server.setExecutor(executor);

server.start();
 
 

class MyHandler implements HttpHandler {
 

	public MyHandler() {

	}
 

	public void handle(HttpExchange exchange) throws IOException {
 

		String requestMethod = exchange.getRequestMethod();

		if (requestMethod.equalsIgnoreCase("GET")) {
 

			// reply 200 OK

			Headers responseHeaders = exchange.getResponseHeaders();

			responseHeaders.set("Content-Type", "text/plain");

			exchange.sendResponseHeaders(200, 0);
 

			OutputStream osResponseBody = exchange.getResponseBody();
 

			// <do some processing, irrelevant as issue occurs if nothing here>
 

			osResponseBody.close();

		}

		exchange.close();

	}

Open in new window

0
Comment
Question by:Romi Kuntsman
  • 15
  • 14
  • 9
38 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 24409979
0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410040
But aren't the HTTP connections supposed to be closed and cleared as soon as the server sends it's response?
0
 
LVL 92

Expert Comment

by:objects
ID: 24410063
Its nothing to do with http, its a TCP/IP issue

0
 
LVL 92

Expert Comment

by:objects
ID: 24410074
actually that may not be entirely correct.
you could try disabling keepalive, nnot sure its ultimately what you want to be doing though

0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410100
Well I only get requests from clients, process them, and have no intention of keeping a connection alive or saving a session.
How can I disable KeepAlive from the server-side in com.sun.net.httpserver.HttpServer?
0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410163
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24410172
You could have your own SocketFactory and set the keep alive parameters. The TCP/IP parameters are an OS-level issue
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24410190
You can also set keep alive and max connections thru properties:

http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410225
CEHJ:

Re SocketFactory - do you have a code example?
Re properties - can I override it just on the level of this server, and not globally in Java?
0
 
LVL 92

Expert Comment

by:objects
ID: 24410241
you can set properties using the command line when you start up the vm (so they will be vm wide)

example here:

http://helpdesk.objects.com.au/java/how-to-tell-a-java-application-to-use-a-proxy-server
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24410288
>>Re properties - can I override it just on the level of this server, and not globally in Java?

No it will be global
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24410316
You don't actually need to use a ScoketFactory if you don't want. Just call the Socket methods to alter keepalive/timeout etc.
0
 
LVL 92

Expert Comment

by:objects
ID: 24410324
Is this actually causing you a problem or is it just not what you expected to see?

0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410344
I can't change things globally for TCP, as I have other (non-HTTP) connection handling in the application, which I don't want to change.

This is causing a huge problem, when the process crashes with an ugly out of memory exception due to lack of ability to allocate more connections or threads.
0
 
LVL 92

Expert Comment

by:objects
ID: 24410351
whats linking the OOM exception to TIME_WAIT connections?
Have you profiled it?

0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410778
I have profiled the TCPv4 Connections Established counter (after checking, this is the one that grows), and when it reached the peak value of 28,739, the application, still trying to accept and process HTTP requests, crashed with the following exception.

Exception in thread "pool-37-thread-36" java.lang.OutOfMemoryError

        at sun.misc.Unsafe.allocateMemory(Native Method)

        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)

        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34

)

        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)

        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)

        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid

er.java:26)

        at java.nio.channels.Selector.open(Selector.java:209)

        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)

        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)

        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

Exception in thread "pool-37-thread-24" java.lang.OutOfMemoryError

        at sun.misc.Unsafe.allocateMemory(Native Method)

        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)

        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34

)

        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)

        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)

        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid

er.java:26)

        at java.nio.channels.Selector.open(Selector.java:209)

        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)

        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)

        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

Exception in thread "pool-37-thread-31" java.lang.OutOfMemoryError

        at sun.misc.Unsafe.allocateMemory(Native Method)

        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)

        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34

)

        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)

        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)

        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid

er.java:26)

        at java.nio.channels.Selector.open(Selector.java:209)

        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)

        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)

        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

Exception in thread "pool-37-thread-1" java.lang.OutOfMemoryError

        at sun.misc.Unsafe.allocateMemory(Native Method)

        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)

        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34

)

        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)

        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)

        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid

er.java:26)

        at java.nio.channels.Selector.open(Selector.java:209)

        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)

        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)

        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

Exception in thread "pool-37-thread-39" java.lang.OutOfMemoryError

        at sun.misc.Unsafe.allocateMemory(Native Method)

        at sun.nio.ch.NativeObject.<init>(NativeObject.java:60)

        at sun.nio.ch.AllocatedNativeObject.<init>(AllocatedNativeObject.java:34

)

        at sun.nio.ch.PollArrayWrapper.<init>(PollArrayWrapper.java:49)

        at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:104)

        at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvid

er.java:26)

        at java.nio.channels.Selector.open(Selector.java:209)

        at sun.net.httpserver.SelectorCache.getSelector(SelectorCache.java:79)

        at sun.net.httpserver.Request$ReadStream.<init>(Request.java:205)

        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:423)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

#

# An unexpected error has been detected by Java Runtime Environment:

#

# java.lang.OutOfMemoryError: requested 24848 bytes for Handle in C:\BUILD_AREA\

jdk6_11\hotspot\src\share\vm\prims\jvmtiGetLoadedClasses.cpp. Out of swap space?
 

#

#  Internal Error (allocation.inline.hpp:42), pid=14728, tid=8136

#  Error: Handle in C:\BUILD_AREA\jdk6_11\hotspot\src\share\vm\prims\jvmtiGetLoa

dedClasses.cpp

#

# Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode windows-x86)

# An error report file with more information is saved as:

# hs_err_pid14728.log

#

# If you would like to submit a bug report, please visit:

#   http://java.sun.com/webapps/bugreport/crash.jsp

#

Open in new window

0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410790
CEHJ:
Where can I call the Socket methods to affect the sun http server bahaviour?
0
 
LVL 86

Assisted Solution

by:CEHJ
CEHJ earned 150 total points
ID: 24410834
It seems that your app might be using a thread pool? Forget about the TCP/IP for the moment. This is more likely to be app-level
0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410866
Yeah I'm using a Thread Pool. Wrote it in the first message!

ThreadPoolExecutor executor = new ThreadPoolExecutor(50, 200, 60,

        TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(1000));

Open in new window

0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 92

Expert Comment

by:objects
ID: 24410867
that doesn't look like standard behaviour to me. how much traffic are you dealing with?
have you tried running it on a different box?

0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410901
I'm dealing with huge traffic, counting impressions, at few tens of requests per second.
This is already a dedicated box running just it, to separate from any other possible issues on a box doing other stuff as well.
0
 
LVL 92

Expert Comment

by:objects
ID: 24410934
(I have to ask) why aren't you using apache?

0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24410986
(I know you do) Because I need to write something directly to my application's memory, and converting has an overload. First tried with IIS, but it was too heavy.
I hope to solve this now as it is. If I can't, I'll look what's need to move this into Apache, or Webster.
0
 
LVL 92

Expert Comment

by:objects
ID: 24411018
how long is an average request? 20K open connections could well be expected under those sort of load.
Have you tried throwing more memory at it?
And the effect of tuning your connection pool


0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24411116
A single request is very short. It parses the URL, writes the params in memory, and returns. Nothing to wait for, no content is even returned to the user.

I don't think memory is the issue, as with whatever load of requests, the amount of open connections should stabilize at some point, and not grow out of control and crash the server.

What tuning of the connection pool? If you mean the suggested global changes in registry or java settings, I can't do that there. If you mean other changes, please direct me.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24411612
Try lowering the thread pool keepAliveTime
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24411684
Actually - ignore that last comment
0
 
LVL 92

Expert Comment

by:objects
ID: 24417050
> I don't think memory is the issue

the vm is running out of memory :)

> What tuning of the connection pool?

eg. its size

0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24419121
It's running out of memory because it is not releasing connections ever, not because any certain amount of memory or pool size would suffice...
0
 
LVL 92

Expert Comment

by:objects
ID: 24419158
it has released the connections, its tcp/ip that is holding onto them by the look
If you stop hitting the server do the connections eventually go away?
Whats the TIMED_WAIT period set to on that box?
0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24420605
I cannot stop hitting the server, it gets connections from the internet all the time.

Where do I check the TIMED_WAIT period currently set?
0
 
LVL 92

Expert Comment

by:objects
ID: 24420623
I meant in testing.

0
 
LVL 86

Expert Comment

by:CEHJ
ID: 24420644
Try blocking connections by decreasing the queue size and let us know if it's still running out of memory
0
 
LVL 4

Accepted Solution

by:
Romi Kuntsman earned 0 total points
ID: 24422698
Moved to Jetty v6 web server and it works like in their code example.

To close HTTP connection, I added a call to flush() and close() on the output stream.

Thanks everyone for trying to help! The issue was the client keeping an open connection and waiting for more output from the page.
It is now stable at about 2,800 open handles, which is a lot, but not growing as before!

0
 
LVL 92

Expert Comment

by:objects
ID: 24427624
as I mentioned above it isn't the client thats keeping it open :)

0
 
LVL 92

Assisted Solution

by:objects
objects earned 150 total points
ID: 24427769
0
 
LVL 4

Author Comment

by:Romi Kuntsman
ID: 24429058
Yeah so good I moved to Jetty, if Sun's server is buggy, was reported two years ago and not handled.

However, even in Jetty, without flushing and closing the connection, it was the client still holding it open after the server didn't send anything more.
0
 
LVL 92

Expert Comment

by:objects
ID: 24429068
Its not a bug (at least according to Sun), as explained in the bug report

good to hear jetty is working for you, better product anyway.


0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

If you don't have the right permissions set for your WordPress location in IIS, you won't be able to perform automatic updates. Here's how to fix the problem.
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now