I'm using Sun's 1.4.2_03-b02 JDK.  I tried using JProfiler.  However, I could not set the -Xms and -Xmx variables.  (I used to set them from the Tomcat Configuration utility.  I start Tomcat as a service, not by using startup.bat)  The server ended up starting with the 64MB of RAM which wasn't enough.  I had to turn off JProfiler immediately.  I could try using some other profiler that you would recommend.

Maybe my problem is related to a fix in 5.0.19.  Here is the change log for 5.0.19 (remember, I'm using 5.0.18): http://jakarta.apache.org/tomcat/tomcat-5.0-doc/changelog.html.  Please take a look at the 3rd item on Cayote.  It reads "Add request unregistration, to avoid the memory leak which was  affecting HTTP (billbarker)"

When I tried installing 5.0.19 on our testing server, I'm getting an internal NullPointerException whenever I try to use crosscontext c:imports.

I doubt that I'm using the beta version of the JDK.  This is the latest downloadable version from java.sun.com.  Here is where I have downloaded the file from: http://java.sun.com/j2se/1.4.2/download.html (first SDK download in the Download J2SE v 1.4.2_03 section.  In the release notes (http://java.sun.com/j2se/1.4.2/ReleaseNotes.html) it reads "The official version number for update release 1.4.2_03 is 1.4.2_03-b02".  If you feel that this is the beta version, I could try installing some other version, not a problem at all.

I have started Tomcat using JProfiler with the -Xms128m -Xmx256m options.  I saved a couple of snapshots.  Should I send them to your email address provided in your user profile (liran@2...4.com)?  Is there anything more spesific you need?

Here is what I observed after playing around JProfiler:

When I load a DB intensive page with lots of data, my objects do use reasonable amounts of memory.  However, after running garbage collection manually, everything gets recycled 100%.  So I assume that there is no memory leakage on my application.  The weird thing is that, even though JProfiler shows that all the memory is recycled, java.exe's size stays the same.

I have tuned my JVM so that the GCs don't take so long.  In fact, I noticed that it was a redundant info because it was taken care of before I posted my question.  Prior to posting my question, I have allocated 512MB max memory to Tomcat which was way too much.  Most of the memory was on the page file...

The structure of my application is actually very straight forward.  All it does is that it reads data from a MySQL database, and then represents the data in a nice format.

Last night, I think I found the reason as to why my java.exe's memory doubles very fast.  Here is what happens:

When I first start tomcat, my java.exe is about 65MB.  When I surf one of our websites, it doesn't really increase much.  However, we've got a rather unusual page on one of our website that needs to display approximately 20,000 rows of MySQL data.  When I click on that page, java.exe goes from 65 MB to ~120 MB in a matter of seconds.  Then, when I refresh the page, it goes to upto ~155 MB.  However, on the third refresh, it stays at ~155 MB, it stops increasing.  When I take a look at the data from JProfiler, I see that most of this data is stored as java.lang.String.  However, after GC, it gets recycled.

Last night, when I stopped testing at 4 AM, my java.exe was ~155MB.  Today, as of 10 AM, it is 246 MB (and we don't get much hits until 8 AM in the morning).  I'm guessing that my memory keeps increasing on every SQL query, however I don't notice it much simply because the size of the result set is usually very very small.  However the 20,000-row case gives me a hint.  What do you think?

Further more, JProfiler clearly shows that my data gets recycled.  You asked me what happens in an hour...  Tomcat status manager shows that it gets recycled.  In fact right now, my total Tomcat heap size is 210 MB and free memory is 150 MB.  About 15 minutes ago, the total heap size was the same, but the free memory was about 30 MB.  Somehow, this doesn't get reflected on java.exe.  I guess the task manager doesn't notice that the most of the heap is empty.

Does this mean that java.exe includes the total heap size of Tomcat?  I used to think that task manager didn't show the heap size in the processes view.  If so, maybe the task manager doesn't know what's really going on in the heap.  If this is the case java.exe is only getting larger depending on the total heap size.  When my total heap size reaches 256MB (-Xmx256m), my java.exe might get stabilized at around 350 MB.  As long as I can prove that it is stabilized (regardless of how big it gets), I would consider the problem solved.  That would mean that there is no memory leakage which is my main concern.

Thank you very much for the explanation.  Just to be on the same page and clear my doubts, I have one last question:

Since our max heap size is 256 MB, it is not actually technically possible for java.exe to grow indefinetely, because Tomcat would throw "Out of Memory" error once the heap is full.  So, as a conclusion: Unless I get an Out of Memory error on normal conditions (reasonable amount of requests), there is no memory leak.

Are we on the same page?  If so, I'm satisfied that everything is running fine (I made some optimizations to our source code so that our application doesn't require more than 256 MB of memory.  It's been working fine since friday).

Thank you,

Asim

Thank you very much for your patience and help.  I learned a lot about the heap size and the JVM memory relationship.  Everything seems to be working fine on our servers.