Solved

Tomcat JVM : Too many open files

Posted on 2009-04-14
4
2,390 Views
Last Modified: 2013-12-02
i have an application running on tomcat application server.  After sometime of running it goes down with "Too many open files" error.

Here is the error.

Apr 14, 2009 6:22:53 PM org.apache.catalina.core.StandardServer await
SEVERE: StandardServer.await: accept:
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.catalina.core.StandardServer.await(StandardServer.java:388)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:615)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:575)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:275)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Apr 14, 2009 6:22:53 PM org.apache.coyote.http11.Http11BaseProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8082
Apr 14, 2009 6:22:53 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8082] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
java.lang.NullPointerException
*****************************************************
 
When I checked the server, I found more than 400 connection made to other server. which are in CLOSE_WAIT state.


java      31550 tomcat2  157u     IPv6 1472334157                 TCP cihcispapp099.corporate.ge.com:34609->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  158u     IPv6 1472335993                 TCP cihcispapp099.corporate.ge.com:34665->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  159u     IPv6 1472338064                 TCP cihcispapp099.corporate.ge.com:34717->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  161u     IPv6 1472351704                 TCP cihcispapp099.corporate.ge.com:35036->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  168u     IPv6 1472365912                 TCP cihcispapp099.corporate.ge.com:35420->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)

****************************************************************

I want to drill down the issue as which specific class or method is making this connection ? How can I start this troubleshooting ? Can someone please guide how to proceed ?
0
Comment
Question by:Prajal_Sutaria
  • 2
4 Comments
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 24144591
Whats the OS here? In case of Unix, you can tighten the DOS attach
parameters. I have a feeling that the system is under DOS attack.
0
 
LVL 1

Author Comment

by:Prajal_Sutaria
ID: 24144598
Its a Linux server. I do not understand where you are trying to point to ?
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 24144814
DOS, (Denial of Service) more here,
http://en.wikipedia.org/wiki/Denial-of-service_attack


is a type of attack, where the user opens a lot of sockets in very short
time and flood the sockets. The legitimate users, will not be able to
connect to server.
0
 
LVL 3

Accepted Solution

by:
CptGiggles earned 500 total points
ID: 24148231
"I found more than 400 connection made to other server"

If both servers that are being used belong to you (Directly or Indirectly) I would tend to think that you could have some typos in your application/configuration that opens files but doesn't fully close them....


We had a problem with a JBoss server at my workplace that was doing something like this (JBoss runs an embedded Tomcat, so I would think they could be the same issue)
If however you think this is not an issue with the application, you can always increase the maximum open files configuration :

LINK: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

I also realize that most people would disagree with me but, sometimes a good old fashioned reboot solves a lot of lingering processes problems (Defunt Javas and such) which can do some crazy stuff to your machines.


0

Featured Post

ScreenConnect 6.0 Free Trial

At ScreenConnect, partner feedback doesn't fall on deaf ears. We collected partner suggestions off of their virtual wish list and transformed them into one game-changing release: ScreenConnect 6.0. Explore all of the extras and enhancements for yourself!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This was posted to the Netbeans forum a Feb, 2010 and I also sent it to Verisign. Who didn't help much in my struggles to get my application signed. ------------------------- Start The idea here is to target your cell phones with the correct…
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
This tutorial covers a practical example of lazy loading technique and early loading technique in a Singleton Design Pattern.

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question