?
Solved

Tomcat JVM : Too many open files

Posted on 2009-04-14
4
Medium Priority
?
2,431 Views
Last Modified: 2013-12-02
i have an application running on tomcat application server.  After sometime of running it goes down with "Too many open files" error.

Here is the error.

Apr 14, 2009 6:22:53 PM org.apache.catalina.core.StandardServer await
SEVERE: StandardServer.await: accept:
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.catalina.core.StandardServer.await(StandardServer.java:388)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:615)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:575)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:275)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Apr 14, 2009 6:22:53 PM org.apache.coyote.http11.Http11BaseProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8082
Apr 14, 2009 6:22:53 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8082] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
        at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
        at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)
java.lang.NullPointerException
*****************************************************
 
When I checked the server, I found more than 400 connection made to other server. which are in CLOSE_WAIT state.


java      31550 tomcat2  157u     IPv6 1472334157                 TCP cihcispapp099.corporate.ge.com:34609->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  158u     IPv6 1472335993                 TCP cihcispapp099.corporate.ge.com:34665->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  159u     IPv6 1472338064                 TCP cihcispapp099.corporate.ge.com:34717->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  161u     IPv6 1472351704                 TCP cihcispapp099.corporate.ge.com:35036->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)
java      31550 tomcat2  168u     IPv6 1472365912                 TCP cihcispapp099.corporate.ge.com:35420->corpp602-V04.corporate.ge.com:http (CLOSE_WAIT)

****************************************************************

I want to drill down the issue as which specific class or method is making this connection ? How can I start this troubleshooting ? Can someone please guide how to proceed ?
0
Comment
Question by:Prajal_Sutaria
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 24144591
Whats the OS here? In case of Unix, you can tighten the DOS attach
parameters. I have a feeling that the system is under DOS attack.
0
 
LVL 1

Author Comment

by:Prajal_Sutaria
ID: 24144598
Its a Linux server. I do not understand where you are trying to point to ?
0
 
LVL 23

Expert Comment

by:Ajay-Singh
ID: 24144814
DOS, (Denial of Service) more here,
http://en.wikipedia.org/wiki/Denial-of-service_attack


is a type of attack, where the user opens a lot of sockets in very short
time and flood the sockets. The legitimate users, will not be able to
connect to server.
0
 
LVL 3

Accepted Solution

by:
CptGiggles earned 1000 total points
ID: 24148231
"I found more than 400 connection made to other server"

If both servers that are being used belong to you (Directly or Indirectly) I would tend to think that you could have some typos in your application/configuration that opens files but doesn't fully close them....


We had a problem with a JBoss server at my workplace that was doing something like this (JBoss runs an embedded Tomcat, so I would think they could be the same issue)
If however you think this is not an issue with the application, you can always increase the maximum open files configuration :

LINK: http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

I also realize that most people would disagree with me but, sometimes a good old fashioned reboot solves a lot of lingering processes problems (Defunt Javas and such) which can do some crazy stuff to your machines.


0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This exercise is about for the following scenario: Dmgr and One node with 2 application server. Each application server contains it owns application. Application server name as follows server1 contains app1 server2 contains app1 Prereq…
Upgrading Tomcat – There are a couple of methods to upgrade Tomcat is to use The Apache Installer is to download and unzip and run the services.bat remove|install Tomcat6 Because of the App that we are working with, we can only use Tomcat 6.…
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …
Suggested Courses
Course of the Month13 days, 6 hours left to enroll

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question