Solved

Tomcat near 100% use on Windows 2008

Posted on 2013-01-28
13
1,646 Views
Last Modified: 2013-01-31
We have a Windows 2008 R2 virtual machine on an ESXi server.  It only runs one application that requires Apache, Tomcat, Java and PostgreSQL.  We have the latest versions of these programs. The machine has 1TB of disk space and 8GB of RAM.  When I start the Tomcat 7 service, it starts using around 95% of CPU in a span of a minute, after which the total utilization of CPU by the machine is 100%.  Process Explorer shows that the number of threads for this process goes up to 91 and one of them shows over 15,000,000 Cycles Delta and uses all the CPU.  While the machine is stalled by Tomcat, the commit charge for the machine is at 12% and physical memory usage at 25%, only the CPU is maxed out.

I added CATALINA_OPTS=-Xms4096m -Xmx512m to the Environment Variables, but it doesn't seem to help.  I believe there is a memory leak here, but not sure how to pinpoint it using free tools and patch this leak.
0
Comment
Question by:lolekbolek
13 Comments
 

Author Comment

by:lolekbolek
Comment Utility
I am attaching everything from the Tomcat's logs directory, including the output of the Probe application that I added to Tomcat.  The last file shows what kind of information Process Explorer lists for the 91 threads that are part of the tomcat7.exe process, including the first one that is eating up all the CPU.
catalina.2013-01-28.log
commons-daemon.2013-01-28.log
localhost.2013-01-28.log
manager.2013-01-28.log
probe.log
tomcat7-stderr.2013-01-28.log
tomcat7-stdout.2013-01-28.log
Tomcat-7-threads-in-Process-Expl.txt
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
I haven't time to pore through these logs now, so just one tip: you can forget about HD/RAM considerations almost certainly. What you should be doing is looking for code that's cpu-intensive for whatever reason.
0
 

Author Comment

by:lolekbolek
Comment Utility
I would love to do that, but, as I said, I am not sure how to.
0
 
LVL 82

Expert Comment

by:leakim971
Comment Utility
0
 
LVL 36

Expert Comment

by:ArneLovius
Comment Utility
There are also free java profiling tools available.

Considering the rest if the software stack, I'd suggest trying it on Linux rather than Windows...
0
Complete Microsoft Windows PC® & Mac Backup

Backup and recovery solutions to protect all your PCs & Mac– on-premises or in remote locations. Acronis backs up entire PC or Mac with patented reliable disk imaging technology and you will be able to restore workstations to a new, dissimilar hardware in minutes.

 
LVL 26

Accepted Solution

by:
mrcoffee365 earned 500 total points
Comment Utility
This is too hard a problem for us to help with remotely.  Hire someone who knows java and tomcat and have them look  at the problem on your machine.

There's nothing wrong with running tomcat on a Windows box.

Hopefully you mistyped when you said what you set memory to in catalina opts.
CATALINA_OPTS=-Xms(min heap)m -Xmx(max heap)m
In your post you said you made the minimum 4 times your max.  If you actually did that, then maybe that's your problem?

You might also have to increase PermGen memory space, with an option like:
-XX:MaxPermSize=128m (or whatever would be more than you have now)

You are running a ton of software with tomcat and it will take quite a bit of time for someone to track down your runaway app (assuming there is one).

If the problem really is the memory allocation for tomcat, then setting catalina opts is ... it might work, but why aren't you using the Configure Tomcat app?  If tomcat is running as a service, which I'd expect is how you have it set up, I'm not sure that catalina opts will definitely get used.
0
 

Author Comment

by:lolekbolek
Comment Utility
We fixed it by applying the suggestion from Apache Tomcat forum, adding the following to the command line options to the Tomcat Window service tomcat7/bin/tomcat7w.exe:

-Xmx4096m
-XX:MaxPermSize=512m
-XX:+UseTLAB
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled

Now, Tomcat uses only a few percents of CPU.
1
 

Author Comment

by:lolekbolek
Comment Utility
I've requested that this question be closed as follows:

Accepted answer: 0 points for lolekbolek's comment #a38841177

for the following reason:

It was the option that worked, as opposed to general comments or links to tools that we should try to use on our own.
1
 
LVL 26

Expert Comment

by:mrcoffee365
Comment Utility
Glad you solved it.  I was telling you some of that already, so it's too bad that you decided not to award any points for the expert help you got.
0
 

Author Comment

by:lolekbolek
Comment Utility
mrcoffee365 provided a good answer and deserves the points, while I still give myself a pat on the back for finding a stanza that I could use as is (for dummy solution that I needed).  Could you please cancel the closing for now and allow me to award mrcoffee365 the points?
0
 

Author Closing Comment

by:lolekbolek
Comment Utility
Mostly right on the money.
0

Featured Post

Do email signature updates give you a headache?

Constantly trying to correctly format email signatures? Spending all of your time at every user’s desk to make updates? Want high-quality HTML signatures on all devices, including on mobiles and Macs? Then, let Exclaimer solve all your email signature problems today!

Join & Write a Comment

Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Sometimes drives fill up and we don't know why.  If you don't understand the best way to use the tools available, you may end up being stumped as to why your drive says it's not full when you have no space left!  Here's how you can find out...
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now