Solved

Tomcat near 100% use on Windows 2008

Posted on 2013-01-28
13
1,773 Views
Last Modified: 2013-01-31
We have a Windows 2008 R2 virtual machine on an ESXi server.  It only runs one application that requires Apache, Tomcat, Java and PostgreSQL.  We have the latest versions of these programs. The machine has 1TB of disk space and 8GB of RAM.  When I start the Tomcat 7 service, it starts using around 95% of CPU in a span of a minute, after which the total utilization of CPU by the machine is 100%.  Process Explorer shows that the number of threads for this process goes up to 91 and one of them shows over 15,000,000 Cycles Delta and uses all the CPU.  While the machine is stalled by Tomcat, the commit charge for the machine is at 12% and physical memory usage at 25%, only the CPU is maxed out.

I added CATALINA_OPTS=-Xms4096m -Xmx512m to the Environment Variables, but it doesn't seem to help.  I believe there is a memory leak here, but not sure how to pinpoint it using free tools and patch this leak.
0
Comment
Question by:lolekbolek
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
13 Comments
 

Author Comment

by:lolekbolek
ID: 38827861
I am attaching everything from the Tomcat's logs directory, including the output of the Probe application that I added to Tomcat.  The last file shows what kind of information Process Explorer lists for the 91 threads that are part of the tomcat7.exe process, including the first one that is eating up all the CPU.
catalina.2013-01-28.log
commons-daemon.2013-01-28.log
localhost.2013-01-28.log
manager.2013-01-28.log
probe.log
tomcat7-stderr.2013-01-28.log
tomcat7-stdout.2013-01-28.log
Tomcat-7-threads-in-Process-Expl.txt
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 38828020
I haven't time to pore through these logs now, so just one tip: you can forget about HD/RAM considerations almost certainly. What you should be doing is looking for code that's cpu-intensive for whatever reason.
0
 

Author Comment

by:lolekbolek
ID: 38830558
I would love to do that, but, as I said, I am not sure how to.
0
Salesforce Has Never Been Easier

Improve and reinforce salesforce training & adoption using WalkMe's digital adoption platform. Start saving on costly employee training by creating fast intuitive Walk-Thrus for Salesforce. Claim your Free Account Now

 
LVL 82

Expert Comment

by:leakim971
ID: 38836984
0
 
LVL 37

Expert Comment

by:ArneLovius
ID: 38837160
There are also free java profiling tools available.

Considering the rest if the software stack, I'd suggest trying it on Linux rather than Windows...
0
 
LVL 27

Accepted Solution

by:
mrcoffee365 earned 500 total points
ID: 38838463
This is too hard a problem for us to help with remotely.  Hire someone who knows java and tomcat and have them look  at the problem on your machine.

There's nothing wrong with running tomcat on a Windows box.

Hopefully you mistyped when you said what you set memory to in catalina opts.
CATALINA_OPTS=-Xms(min heap)m -Xmx(max heap)m
In your post you said you made the minimum 4 times your max.  If you actually did that, then maybe that's your problem?

You might also have to increase PermGen memory space, with an option like:
-XX:MaxPermSize=128m (or whatever would be more than you have now)

You are running a ton of software with tomcat and it will take quite a bit of time for someone to track down your runaway app (assuming there is one).

If the problem really is the memory allocation for tomcat, then setting catalina opts is ... it might work, but why aren't you using the Configure Tomcat app?  If tomcat is running as a service, which I'd expect is how you have it set up, I'm not sure that catalina opts will definitely get used.
0
 

Author Comment

by:lolekbolek
ID: 38841177
We fixed it by applying the suggestion from Apache Tomcat forum, adding the following to the command line options to the Tomcat Window service tomcat7/bin/tomcat7w.exe:

-Xmx4096m
-XX:MaxPermSize=512m
-XX:+UseTLAB
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled

Now, Tomcat uses only a few percents of CPU.
1
 

Author Comment

by:lolekbolek
ID: 38842140
I've requested that this question be closed as follows:

Accepted answer: 0 points for lolekbolek's comment #a38841177

for the following reason:

It was the option that worked, as opposed to general comments or links to tools that we should try to use on our own.
1
 
LVL 27

Expert Comment

by:mrcoffee365
ID: 38842091
Glad you solved it.  I was telling you some of that already, so it's too bad that you decided not to award any points for the expert help you got.
0
 

Author Comment

by:lolekbolek
ID: 38842138
mrcoffee365 provided a good answer and deserves the points, while I still give myself a pat on the back for finding a stanza that I could use as is (for dummy solution that I needed).  Could you please cancel the closing for now and allow me to award mrcoffee365 the points?
0
 

Author Closing Comment

by:lolekbolek
ID: 38842141
Mostly right on the money.
0

Featured Post

Free NetCrunch network monitor licenses!

Only on Experts-Exchange: Sign-up for a free-trial and we'll send you your permanent license!

Here is what you get: 30 Nodes | Unlimited Sensors | No Time Restrictions | Absolutely FREE!

Act now. This offer ends July 14, 2017.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
OfficeMate Freezes on login or does not load after login credentials are input.
The viewer will learn how to implement Singleton Design Pattern in Java.
This video teaches viewers about errors in exception handling.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question