[Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Garbage Collection - multiple sequential full gcs produces OutOfMemoryError

Posted on 2005-05-06
12
Medium Priority
?
367 Views
Last Modified: 2012-08-14
I am recieving an OutOfMemoryError in my tomcat logs.  I've done a bit of research, tuned parameters, logged gc etc. What I have found is that the OutOfMemoryErrors occur when the garbage collector does several back-to-back full garbage collections.  Below is the output from turning on the -Xloggc flag.

        ...
        45058.932: [GC 241734K->226138K(257164K), 0.0406180 secs]
        45069.840: [Full GC 242134K->64082K(257164K), 1.1170640 secs]
        45075.677: [GC 79503K->65708K(246232K), 0.0513280 secs]
        ...
        46517.803: [GC 189661K->189569K(246232K), 0.0840740 secs]
        46518.033: [GC 219496K->219434K(246232K), 0.0876900 secs]
        46518.287: [Full GC 245984K->168803K(246232K), 1.5587820 secs]
        46519.978: [GC 185245K->184869K(260160K), 0.1363050 secs]
        46520.281: [GC 204247K->203053K(260160K), 0.1215160 secs]
        46520.524: [GC 219385K->218914K(260160K), 0.1191330 secs]
 -->  46520.902: [Full GC 245053K->225597K(260160K), 1.8996470 secs]
 -->  46523.039: [Full GC 257946K->257356K(260160K), 0.6038950 secs]
 -->  46523.673: [Full GC 260144K->259881K(260160K), 1.3549970 secs]
 -->  46525.028: [Full GC 259881K->259881K(260160K), 0.4633640 secs]
 -->  46525.555: [Full GC 260159K->35411K(260160K), 0.5106320 secs]
        46531.403: [GC 51602K->37156K(260160K), 0.2099900 secs]
        46533.591: [GC 53348K->37075K(260160K), 0.0421670 secs]

Somewhere during the highlighted [Full GC...] lines, I get at least one OutOfMemoryError.  Because the garbage collector often runs a single time and brings the memory usage down to around 64M (see second line of output above), I believe that my real required memory is in that range, I currently have Xmx set to 256m (Could having this set so high actually be causing the problem?).   There are 4G of RAM on the machine.  

Java version information:
        # java -version
        java version "1.4.1_03"
        Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_03-b02)
        Java HotSpot(TM) Client VM (build 1.4.1_03-b02, mixed mode)

JAVA_OPTS
       -XX:PermSize=256m -XX:MaxPermSize=256m -Xmx256m -Xloggc:gc.log
       (PermSize and MaxPermSize are being set because at one point, we believed
        the source of the OutOfMemoryError might actually be Perm space, however,
        the addition of these settings has not changed the behavior.  Is there a flag
        similar to -Xloggc which will give me information about the Perm space?
        All of the tools that I have seen thus far are for java 5 and above).

Any insight appreciated.
0
Comment
Question by:sharonfrench
  • 2
  • 2
  • 2
  • +2
9 Comments
 
LVL 24

Expert Comment

by:sciuriware
ID: 13945666
I don't know if you do, but better not call the GC explicitly.
Besides that, there's now 1.4.2_08 and NOT WITHOUT REASON!
;JOOP!
0
 
LVL 92

Expert Comment

by:objects
ID: 13948513
If you are getting an OOM error it is because you don't have enough spare memory available (after running gc and freeing any unreferenced objects) to meet a request for memory.
I suggest running a profiler on your code to determine where all your memory is being used.
0
 
LVL 15

Expert Comment

by:aozarov
ID: 13949659
>> OutOfMemoryErrors occur when the garbage collector does several back-to-back full garbage collections.
This normally happens when you don't have enough memory as the G.C is trying to find what ever it can (until it gives up and throw
OutOfMemoryError)

>>  -XX:PermSize=256m -XX:MaxPermSize=256m -Xmx256m -Xloggc:gc.log
PermSize is the space for stroing the classes. So normally it is much smaller and after certain point
becomes pretty static. The default size is sufficient for most uses. Are you using hot-deployment ?
Personally I think that 256m for it is way to much.
See: http://java.sun.com/docs/hotspot/gc1.4.2/faq.html (section 7 and 8) [section 8 shows how to get verbose G.C for it]

>> There are 4G of RAM on the machine.  
Then try to increase your heap size (e.g. -Xmx1024m [-Xms1024m is also recommended]).
Your program might really need it (assuming you don't have a memory leak, and if you have one then eventually even with
increasing the size you will get OOME)

>> Java HotSpot(TM) Client VM (build 1.4.1_03-b02, mixed mode)
For serverside applications you will be better of with the Server VM (using the flag -server or -Xoptimze)
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 15

Expert Comment

by:JakobA
ID: 13949978
There is a program that leaks memory.

Ther garbeage collecting is not the cause of your error, it is caused by it. as the amount of free memory decrease the garbage colletor get called more and more often.
0
 

Author Comment

by:sharonfrench
ID: 13959352
I have tried running with -Xmx1024m, and the same behavior occurs.

Perhaps someone can explain to me why I get the following behavior:
    - memory slowly grows
    - incremental garbage collection happens along the way
    - reach Xmx value (we've tried everything from 256 ->1024M and get the same result)
    - full gc reclaims some space
    - additional incrementals happen
    - reach the Xmx value
    - full gc reclaims a small portion of the space
    - repeated back to back full gcs reclaim additional space back down to around 64M

Given that the same thing occurs regardless of the Xmx value, I get the impression that there is something in the full gc logic that doesn't actually reclaim all available space, such that repeated back to back full gcs are necessary and do reclaim the space.

Because we repeatedly return to around the 64M mark after the full gcs happen, I was under the impression that this is the amount of space we actually require.  If this is the case, why the repeated back to back full-gcs.

Could there be something about tomcat in particular that is interfering with the gc?
Perhaps there is something that is happening that actually requires a large amount of memory for a small amount of time (but if so, why the same behavior regarless of the Xmx value?).

I appreciate the suggestion that we run a memory profiler, but can anyone suggest such a tool which runs against the version of the VM I am using?
     java version "1.4.1_03"
     Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_03-b02)
     Java HotSpot(TM) Client VM (build 1.4.1_03-b02, mixed mode)
0
 

Author Comment

by:sharonfrench
ID: 13959415
An additional piece of information:

After the repeated full-gcs the amount of memory used drops back down to 64M, and the program continues on happily.  So although the initial full does not reclaim enough memory, subsequent full gcs (approx 1/2 second later) do.

Could it be that the the multiple full gcs are coming from distinct threads within tomcat, and it isn't until one of the threads dies (with an OOME), that enough memory is available?

Perhaps what we are really seeing is that tomcat worker threads are actually dieing off and therefore releasing memory for the others to continue on?

Again, any suggestions for memory profiling tools that will work with the 1.4.1 JVM would be appreciated.
0
 
LVL 15

Accepted Solution

by:
aozarov earned 1000 total points
ID: 13960128
>Could there be something about tomcat in particular that is interfering with the gc?
Don't think so, it is probably something in your application.

>Perhaps there is something that is happening that actually requires a large amount of memory for a small amount of time (but if so, why the same behavior regarless of the Xmx value?).
Yes, because maybe the needed "llarge amount of memory for a small amount of time" is sometime bigger then -Xmx (you can still increase it as you have 4G)

> memory slowly grows
> repeated back to back full gcs reclaim additional space back down to around 64M
Though the first suggest some kind of memory leak the second statement seems to contradict it.

> I appreciate the suggestion that we run a memory profiler, but can anyone suggest such a tool which runs against the version of the VM I am using?
Most of the profilers (Optimize it,  Eclipse profiler plugin, ...) or you can use the standard java profile options java -Xprof

>Could it be that the the multiple full gcs are coming from distinct threads within tomcat
Are you calling System.gc manually after dereferencing some objects?
0
 
LVL 92

Assisted Solution

by:objects
objects earned 1000 total points
ID: 13963895
As I mentioned above if you're gettinmg an OOM exception it is because your app does not have enough memory available to meet a request for more memory and there is no memory that can be freed because all objects using memory are being referenced by your application.
A profiler will help you identify where in your application all the memory is veing held.
0
 
LVL 24

Expert Comment

by:sciuriware
ID: 14595671
Please split between aozarov and objects.
;JOOP!
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This was posted to the Netbeans forum a Feb, 2010 and I also sent it to Verisign. Who didn't help much in my struggles to get my application signed. ------------------------- Start The idea here is to target your cell phones with the correct…
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Viewers will learn about the different types of variables in Java and how to declare them. Decide the type of variable desired: Put the keyword corresponding to the type of variable in front of the variable name: Use the equal sign to assign a v…
The viewer will learn how to implement Singleton Design Pattern in Java.
Suggested Courses
Course of the Month20 days, 12 hours left to enroll

864 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question