Solved

Tomcat JVM : Too many open files

Posted on 2009-04-14
4
2,377 Views
Last Modified: 2013-12-02
i have an application running on tomcat application server.  After sometime of running it goes down with "Too many open files" error.

Here is the error.

Apr 14, 2009 6:22:53 PM org.apache.catalina.core.StandardServer await
SEVERE: StandardServer.await: accept:
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.catalina.core.StandardServer.await(StandardServer.java:388)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:615)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:575)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:275)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Apr 14, 2009 6:22:53 PM org.apache.coyote.http11.Http11BaseProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8082
Apr 14, 2009 6:22:53 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8082] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
java.lang.NullPointerException
*****************************************************
 
When I checked the server, I found more than 400 connection made to other server. which are in CLOSE_WAIT state.


java      31550 tomcat2  157u     IPv6 1472334157                 TCP cihcispapp099.corporate.ge.com:34609->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  158u     IPv6 1472335993                 TCP cihcispapp099.corporate.ge.com:34665->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  159u     IPv6 1472338064                 TCP cihcispapp099.corporate.ge.com:34717->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  161u     IPv6 1472351704                 TCP cihcispapp099.corporate.ge.com:35036->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  168u     IPv6 1472365912                 TCP cihcispapp099.corporate.ge.com:35420->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)

****************************************************************

I want to drill down the issue as which specific class or method is making this connection ? How can I start this troubleshooting ? Can someone please guide how to proceed ?
0
Comment
Question by:Prajal_Sutaria
  • 2
4 Comments
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 24144591
Whats the OS here? In case of Unix, you can tighten the DOS attach
parameters. I have a feeling that the system is under DOS attack.
0
 
LVL 1

Author Comment

by:Prajal_Sutaria
ID: 24144598
Its a Linux server. I do not understand where you are trying to point to ?
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 24144814
DOS, (Denial of Service) more here,
http://en.wikipedia.org/wiki/Denial-of-service_attack


is a type of attack, where the user opens a lot of sockets in very short
time and flood the sockets. The legitimate users, will not be able to
connect to server.
0
 
LVL 3

Accepted Solution

by:
CptGiggles earned 500 total points
ID: 24148231
"I found more than 400 connection made to other server"

If both servers that are being used belong to you (Directly or Indirectly) I would tend to think that you could have some typos in your application/configuration that opens files but doesn't fully close them....


We had a problem with a JBoss server at my workplace that was doing something like this (JBoss runs an embedded Tomcat, so I would think they could be the same issue)
If however you think this is not an issue with the application, you can always increase the maximum open files configuration :

LINK: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

I also realize that most people would disagree with me but, sometimes a good old fashioned reboot solves a lot of lingering processes problems (Defunt Javas and such) which can do some crazy stuff to your machines.


0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Most of the developers using Tomcat find it easy to configure the datasource in Server.xml and use the JNDI name in the code to get the connection.  So the default connection pool using DBCP (or any other framework) is made available and the life go…
Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
Viewers learn about the “while” loop and how to utilize it correctly in Java. Additionally, viewers begin exploring how to include conditional statements within a while loop and avoid an endless loop. Define While Loop: Basic Example: Explanatio…
The viewer will learn how to implement Singleton Design Pattern in Java.

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now