Solved

Tomcat JVM Inaccessible  - Logs Clean

Posted on 2009-05-07
14
650 Views
Last Modified: 2013-12-02
I have been running into this situation every 1-2 days on my production site where my Tomcat 5.5.7 JVM will become unresponsive. My Tomcat logs are clean. I have included a dump of my memory usage below. I notice that the Committed & Max are the same for the Eden Space, Survivor Space, & Old Gen, yet the used is considerably lower. Can someone please explain and offer suggestions:

MEMORY TYPE: Code Cache  
 Used:        22937280  
 Committed:   23035904
 Max:         50331648  
   
 MEMORY TYPE: PS Eden Space  
 Used:        28793728  
 Committed:   51183616
 Max:         51183616  
   
 MEMORY TYPE: PS Survivor Space  
 Used:        1935936  
 Committed:   4325376
 Max:         4325376  
   
 MEMORY TYPE: PS Old Gen  
 Used:        179478224  
 Committed:   477233152
 Max:         477233152  
   
 MEMORY TYPE: PS Perm Gen  
 Used:        66484456  
 Committed:   71172096
 Max:         268435456    
 
 Used Memory:  210207888
 Free Memory:  322534256
 Total Memory: 532742144
0
Comment
Question by:scott_m_ruby
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 6
  • 5
14 Comments
 
LVL 92

Expert Comment

by:objects
ID: 24400901
sounds like you are running of request threads.
Do a thread dump when it is unresponsive to check whats going on
may be that you just need to increase the thread pool size

0
 

Author Comment

by:scott_m_ruby
ID: 24400931
Can you give me the syntax for doing a thread dump?
0
 
LVL 92

Expert Comment

by:objects
ID: 24400938
send the tomcat process a QUIT signal

what OS?

0
Turn Insights into Action

Communication across every corner of your business is essential to increase the velocity of your application delivery and support pipeline. Automate, standardize, and contextualize your communication processes with xMatters.

 

Author Comment

by:scott_m_ruby
ID: 24400947
Linux
0
 
LVL 92

Expert Comment

by:objects
ID: 24400957
kill -quit

the thread dump will get dumped to stdout, which usually ends up going to catalina.out

0
 

Author Comment

by:scott_m_ruby
ID: 24416844
Tomcat became unresponsive again this afternoon and I executed the thread dump as you suggested. It looks like I am using my maxThreads of 150. I will plan to increase but was hoping that you could look at my dump file to help diagnose what the "problem" may be. I have attached the portion of my catalina.out containing the thread dump. Also, as a note when I execute the kill -quit <tomcat pid> my process did not get killed. Should it have, normally?

Thanks,
Scott
catalina.log
0
 
LVL 92

Accepted Solution

by:
objects earned 500 total points
ID: 24417342
appears to be lots of waiting on your connection pool.
check the size of your pool and the amount of database connections available
and make sure you close connections when you no longer need them

>  Should it have, normally?

no it doesn't kill the process just sends it a signal

0
 

Author Comment

by:scott_m_ruby
ID: 24437439
I did double my maxThreads from 150 to 300 and increase my max connections for my database connection pool from 25 to 50. The problem did not occur on Tuesday or Wednesday. In my monitoring via the Tomcat Manager web app I saw the thread count at 25 all day both days. I did notice today that I had 37 database connections gobbled up in about 10 seconds and I don't even have that many active users. Any ideas on how to get a handle on? Is there any java profiling tool or Tomcat monitoring tool that can help?
0
 
LVL 92

Expert Comment

by:objects
ID: 24437457
sounds better.

> . I did notice today that I had 37 database connections gobbled up in about 10 seconds and I don't even have that many active users.

if you are checking at the db then that could be ok as the pool will keep them open for a period (in case they can be reused)
0
 

Author Comment

by:scott_m_ruby
ID: 24530763
I wanted to check back with you as I believe I need a Tomcat monitoring tool that will allow me to see what is going on with my database connection pool as I continue to have periods throughout the day where the app tries to gobble up large number of connections and that seems to be causing our problems.
0
 
LVL 92

Expert Comment

by:objects
ID: 24532300
you can check the db to see how many connections are currently open

0

Featured Post

What Is Transaction Monitoring and who needs it?

Synthetic Transaction Monitoring that you need for the day to day, which ensures your business website keeps running optimally, and that there is no downtime to impact your customer experience.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article was originally published on Monitis Blog, you can check it here . Today it’s fairly well known that high-performing websites and applications bring in more visitors, higher SEO, and ultimately more sales. By the same token, downtime…
Although a lot of people devote their energy toward marketing for specific industries, there are some basic principles that can be applied to any sector imaginable. We’ll look at four steps to take and examine how those steps were put into action fo…
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

717 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question