Solved

Tomcat near 100% use on Windows 2008

Posted on 2013-01-28
13
1,748 Views
Last Modified: 2013-01-31
We have a Windows 2008 R2 virtual machine on an ESXi server.  It only runs one application that requires Apache, Tomcat, Java and PostgreSQL.  We have the latest versions of these programs. The machine has 1TB of disk space and 8GB of RAM.  When I start the Tomcat 7 service, it starts using around 95% of CPU in a span of a minute, after which the total utilization of CPU by the machine is 100%.  Process Explorer shows that the number of threads for this process goes up to 91 and one of them shows over 15,000,000 Cycles Delta and uses all the CPU.  While the machine is stalled by Tomcat, the commit charge for the machine is at 12% and physical memory usage at 25%, only the CPU is maxed out.

I added CATALINA_OPTS=-Xms4096m -Xmx512m to the Environment Variables, but it doesn't seem to help.  I believe there is a memory leak here, but not sure how to pinpoint it using free tools and patch this leak.
0
Comment
Question by:lolekbolek
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
13 Comments
 

Author Comment

by:lolekbolek
ID: 38827861
I am attaching everything from the Tomcat's logs directory, including the output of the Probe application that I added to Tomcat.  The last file shows what kind of information Process Explorer lists for the 91 threads that are part of the tomcat7.exe process, including the first one that is eating up all the CPU.
catalina.2013-01-28.log
commons-daemon.2013-01-28.log
localhost.2013-01-28.log
manager.2013-01-28.log
probe.log
tomcat7-stderr.2013-01-28.log
tomcat7-stdout.2013-01-28.log
Tomcat-7-threads-in-Process-Expl.txt
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 38828020
I haven't time to pore through these logs now, so just one tip: you can forget about HD/RAM considerations almost certainly. What you should be doing is looking for code that's cpu-intensive for whatever reason.
0
 

Author Comment

by:lolekbolek
ID: 38830558
I would love to do that, but, as I said, I am not sure how to.
0
Has Powershell sent you back into the Stone Age?

If managing Active Directory using Windows Powershell® is making you feel like you stepped back in time, you are not alone.  For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why.

 
LVL 82

Expert Comment

by:leakim971
ID: 38836984
0
 
LVL 37

Expert Comment

by:ArneLovius
ID: 38837160
There are also free java profiling tools available.

Considering the rest if the software stack, I'd suggest trying it on Linux rather than Windows...
0
 
LVL 27

Accepted Solution

by:
mrcoffee365 earned 500 total points
ID: 38838463
This is too hard a problem for us to help with remotely.  Hire someone who knows java and tomcat and have them look  at the problem on your machine.

There's nothing wrong with running tomcat on a Windows box.

Hopefully you mistyped when you said what you set memory to in catalina opts.
CATALINA_OPTS=-Xms(min heap)m -Xmx(max heap)m
In your post you said you made the minimum 4 times your max.  If you actually did that, then maybe that's your problem?

You might also have to increase PermGen memory space, with an option like:
-XX:MaxPermSize=128m (or whatever would be more than you have now)

You are running a ton of software with tomcat and it will take quite a bit of time for someone to track down your runaway app (assuming there is one).

If the problem really is the memory allocation for tomcat, then setting catalina opts is ... it might work, but why aren't you using the Configure Tomcat app?  If tomcat is running as a service, which I'd expect is how you have it set up, I'm not sure that catalina opts will definitely get used.
0
 

Author Comment

by:lolekbolek
ID: 38841177
We fixed it by applying the suggestion from Apache Tomcat forum, adding the following to the command line options to the Tomcat Window service tomcat7/bin/tomcat7w.exe:

-Xmx4096m
-XX:MaxPermSize=512m
-XX:+UseTLAB
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled

Now, Tomcat uses only a few percents of CPU.
1
 

Author Comment

by:lolekbolek
ID: 38842140
I've requested that this question be closed as follows:

Accepted answer: 0 points for lolekbolek's comment #a38841177

for the following reason:

It was the option that worked, as opposed to general comments or links to tools that we should try to use on our own.
1
 
LVL 27

Expert Comment

by:mrcoffee365
ID: 38842091
Glad you solved it.  I was telling you some of that already, so it's too bad that you decided not to award any points for the expert help you got.
0
 

Author Comment

by:lolekbolek
ID: 38842138
mrcoffee365 provided a good answer and deserves the points, while I still give myself a pat on the back for finding a stanza that I could use as is (for dummy solution that I needed).  Could you please cancel the closing for now and allow me to award mrcoffee365 the points?
0
 

Author Closing Comment

by:lolekbolek
ID: 38842141
Mostly right on the money.
0

Featured Post

Online Training Solution

Drastically shorten your training time with WalkMe's advanced online training solution that Guides your trainees to action. Forget about retraining and skyrocket knowledge retention rates.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you are a web developer, you would be aware of the <iframe> tag in HTML. The <iframe> stands for inline frame and is used to embed another document within the current HTML document. The embedded document could be even another website.
OfficeMate Freezes on login or does not load after login credentials are input.
To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention. Multiple USB devices need t…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question