Java garbage collection tuning with 18GB heap

We have a Java application deployed on a Tomcat Server with 18GB of RAM running the IBM JVM version 5 for AIX.  The -Xms and -Xmx parms are both set to the full 18GB.  The server itself has 20GB total, with no other applications running.

Our application has a data cache that uses around 4 GB of the memory for persistent objects.  We kick of processing that creates large hashmaps.  We are seeing the entire JVM hang for long periods of time (several minutes) doing garbage collection.  I have never tuned a JVM like this.  Below is an example of the garbage collection stats.

I know there are many switches for tuning the garbage collection, but I need some guidance.  Thanks in advance.





<af type="tenured" id="69" timestamp="Wed Feb 06 21:19:08 2008" intervalms="158235.592">
  <minimum requested_bytes="320" />
  <time exclusiveaccessms="15.588" />
  <tenured freebytes="193272832" totalbytes="19327352832" percent="0" >
    <soa freebytes="0" totalbytes="19134080000" percent="0" />
    <loa freebytes="193272832" totalbytes="193272832" percent="100" />
  </tenured>
  <gc type="global" id="69" totalid="69" intervalms="158236.021">
    <refs_cleared soft="19" threshold="32" weak="0" phantom="0" />
    <finalization objectsqueued="0" />
    <timesms mark="119937.384" sweep="2439.074" compact="0.000" total="122376.815" />
    <tenured freebytes="9602494072" totalbytes="19327352832" percent="49" >
      <soa freebytes="9428548216" totalbytes="19153406976" percent="49" />
      <loa freebytes="173945856" totalbytes="173945856" percent="100" />
    </tenured>
  </gc>
  <tenured freebytes="9602493056" totalbytes="19327352832" percent="49" >
    <soa freebytes="9428547200" totalbytes="19153406976" percent="49" />
    <loa freebytes="173945856" totalbytes="173945856" percent="100" />
  </tenured>
  <time totalms="123865.122" />
</af>
ggetz66Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ajay-SinghCommented:
What are the JVM options related to GC apart from -Xmx and -Xms
0
ggetz66Author Commented:
None at this point.  I have limited access to the server, so I can't just try stuff.  I am seeking ideas to try.  Thanks
0
cmalakarCommented:
I suggest you, to read the below article, which explains how the memory in heap is organized..

http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html#1.1.%20Generations%7Coutline

Use jconsole to see how your memory is varying..

http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jconsole.html

Also.. you can look at this blog entry.. which explains how to size various memory pools of heap..

http://blogs.sun.com/partnertech/entry/a_short_primer_to_java

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

Ajay-SinghCommented:
I would suggest to use new GC algorithms, they are vey efficient and
drastically reduce the GC pause time.
 

http://developers.sun.com/mobility/midp/articles/garbagecollection2/
0
ggetz66Author Commented:
I have tried adding the parameter -Xgcpolicy:optavgpause and am testing.  I have read a number of these articles.  How similiar is gc between the Sun JVM and IBM JVM?

I don't know how to determine what my eden and tenured pool is, or what it should be.  I don't know how to interpret the garbage collection messages in the console:

<con event="kickoff" timestamp="Thu Feb 07 08:01:13 2008">
  <stats tenurefreebytes="1752956464" tracetarget="6250465792" kickoff="786616640" tracerate="8.00" />
</con>

<con event="collection" id="1" timestamp="Thu Feb 07 08:01:22 2008" intervalms="0.000">
  <time exclusiveaccessms="0.068" />
  <tenured freebytes="1592633648" totalbytes="19327352832" percent="8" >
    <soa freebytes="626266416" totalbytes="18360985600" percent="3" />
    <loa freebytes="966367232" totalbytes="966367232" percent="100" />
  </tenured>
  <stats tracetarget="6250465792">
    <traced total="1363963416" mutators="1096036128" helpers="267927288" percent="21" />
    <cards cleaned="148" kickoff="262205552" />
  </stats>
  <con event="final card cleaning">
    <stats cardscleaned="6" traced="0" durationms="13.881" />
  </con>
  <gc type="global" id="1" totalid="1" intervalms="0.000">
    <warning details="completed sweep to facilitate LOA resize" />
    <classloadersunloaded count="22" timetakenms="3109.477" />
    <refs_cleared soft="99" threshold="32" weak="365" phantom="0" />
    <finalization objectsqueued="470" />
    <timesms mark="78.964" sweep="6404.577" compact="0.000" total="9593.163" />
    <tenured freebytes="16780302504" totalbytes="19327352832" percent="86" >
      <soa freebytes="15813935272" totalbytes="18360985600" percent="86" />
      <loa freebytes="966367232" totalbytes="966367232" percent="100" />
    </tenured>
  </gc>
  <tenured freebytes="16780302504" totalbytes="19327352832" percent="86" >
    <soa freebytes="15813935272" totalbytes="18360985600" percent="86" />
    <loa freebytes="966367232" totalbytes="966367232" percent="100" />
  </tenured>
  <time totalms="9613.538" />
</con>
0
Ajay-SinghCommented:
I recommend to use parallel and concurrent GC algos,

try setting these parameters
-XX:+ UseConcMarkSweepGC -XX:+UseParNewGC

I believe they are applicable for IBM JRE as well.
0
johng75370Commented:
In reading your question I thought very fast hash table algorithms might also be of interest.  I work with Cliff Click who presented this at JavaOne last year, and (I believe) also has posted this on sourceforge.  Here is a link to Cliff's presentation on this:  www.azulsystems.com/events/javaone_2007/2007_LockFreeHash.pdf

Azul deals with very large heap sizes (up to 670GB) with no GC pauses, so there may be other info of interest on the website.  
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java App Servers

From novice to tech pro — start learning today.