Solved

Tomcat Hangs twice or three times a day

Posted on 2013-05-27
4
1,061 Views
Last Modified: 2013-07-03
Hi,

I have a Tomcat Server with some applications who asks a Mysql Database, and twice or three times a day It hangs. I need to stop and start it to bring it up online again. I've seen these errors:

ADVERTENCIA: Connection has been abandoned PooledConnection[com.mysql.jdbc.JDBC4Connection@9168b4]:java.lang.Exception
      at org.apache.tomcat.jdbc.pool.ConnectionPool.getThreadDump

...

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
...
java.sql.SQLException: java.lang.OutOfMemoryError: Java heap space
...

Here is the configuration of the Two pools in our Application:

  <Resource name="jdbc/PconDB"
  auth="Container"
  type="javax.sql.DataSource"
  factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
  testWhileIdle="true"
            testOnBorrow="true"
            testOnReturn="false"
            validationQuery="SELECT 1"
            validationInterval="30000"
            timeBetweenEvictionRunsMillis="30000"
            maxActive="300"
            minIdle="30"
            maxWait="10000"
            initialSize="30"
            removeAbandonedTimeout="60"
            removeAbandoned="true"
            logAbandoned="true"
            minEvictableIdleTimeMillis="30000"
            jmxEnabled="true"
            username="system"
            password="----"
            driverClassName="com.mysql.jdbc.Driver"
            url="jdbc:mysql://172.16.2.179:3306/pcondb?autoReconnect=true"/>              


  <Resource name="jdbc/EBS"
  auth="Container"
  type="javax.sql.DataSource"
  factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
  testWhileIdle="true"
            testOnBorrow="true"
            testOnReturn="false"
            validationQuery="SELECT 1 FROM DUAL"
            validationInterval="30000"
            timeBetweenEvictionRunsMillis="30000"
            maxActive="100"
            minIdle="20"
            maxWait="10000"
            initialSize="20"
            removeAbandonedTimeout="60"
            removeAbandoned="true"
            logAbandoned="true"
            minEvictableIdleTimeMillis="30000"
            jmxEnabled="true"
            username="apps"
            password="---"
            driverClassName="oracle.jdbc.driver.OracleDriver"
            url="jdbc:oracle:thin:@//172.16.2.249:1521/prod"/>  



We are usint a VM CEntOS 5.8 x64 on Esxi, and Tomcat 7.0.35 with jre1_6_0_27.

Can you give me an advice about what I need to do to get rid of these errors, and avoid to restart the Tomcat Server every 4-5 hours  because it hangs?

Thanks in Advance.

Josep
catalina.out.txt.zip
0
Comment
Question by:josepvallejo
  • 3
4 Comments
 
LVL 35

Accepted Solution

by:
mccarl earned 500 total points
ID: 39200143
You would more likely need to look more at the actual applications rather than anything Tomcat specific. Did you develop these applications yourself? Or are they third party apps?
0
 

Author Comment

by:josepvallejo
ID: 39200565
Hi mccarl,

Yes, These applications are internal and developped by a programmer. He is trying to solve some problems and errors related to queries, but there still aren't any improvements, so we were thinking about checking some parameteres to stabilize the system while the Programmer is solving Application errors.

The mainly thing is that this Web Server ins in production and is getting worse.

I don't know if we could do something to make it run a little bit better and not dying twice o three times a day.

Thanks in Advance for your answer.
Josep.
0
 

Author Comment

by:josepvallejo
ID: 39221620
I have stabilized the problem putting the parameter "RemoveAbandonedTimeout="6000" in both pools and modifying the script to reboot Tomcat every day at 3:00AM (it wasn't working before).

I've put this parameter in Tomcat options: -XX:-UseGCOverheadLimit

Please, close the request .

Josep.
0
 

Author Closing Comment

by:josepvallejo
ID: 39296219
Finally as mccarl said, it was a leaking problem due to a bad application design. It was nearly impossible to stabilize the system adjusting parameters and restarting it periodically. Finally the programmer  saw some problems about not closing sessions with mysql.close and not  catching exceptions properly with try-catch.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

As a database administrator, you may need to audit your table(s) to determine whether the data types are optimal for your real-world data needs.  This Article is intended to be a resource for such a task. Preface The other day, I was involved …
Popularity Can Be Measured Sometimes we deal with questions of popularity, and we need a way to collect opinions from our clients.  This article shows a simple teaching example of how we might elect a favorite color by letting our clients vote for …
Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now