Solved

OutOfMemoryError even though 650 megabytes still available in heap

Posted on 2007-11-30
23
748 Views
Last Modified: 2013-12-29
We have a mature client/server Java product, which recently threw a mysterious OutOfMemoryError.
The mysterious part is that there was about 650 megabytes available when the OutOfMemoryError was thrown.  

We don't have proof, because we didn't have all of the necessary diagnostics on, but we're sure,
based on our experience with this program, that it didn't attempt to allocate that much memory all of a sudden.  

What would cause this?

Details:

Hardware Platform:
    64-Bit 2xCore 2 Duo 2.4 GHz
Java Platform:
    Java 1.6.0, running in 64-bit
OS Platform:
    SUSE Linux

Conditions from shortly before the error occurred:

Max Java Heap Size:
    1,908,932,608 (correlates to -Xmx setting)
Current  Heap Size:
    1,450,311,680 (still has about 450meg available)
Free in  Heap:
      123,412,328 (here's another 120meg, too)

Error Message:
java.lang.OutOfMemoryError: Java heap space
No stack trace was included.
0
Comment
Question by:ChrisVoilaweb
  • 6
  • 6
  • 4
  • +3
23 Comments
 
LVL 9

Expert Comment

by:ysnky
Comment Utility
you must increase memory settings;
java -Xms<initial heap size> -Xmx<maximum heap size> MyApplication

Defaults are:
java -Xms32m -Xmx128m MyApplication
0
 

Author Comment

by:ChrisVoilaweb
Comment Utility
No, as I tried to make clear in my problem statement, we're completely familiar with the memory requirements of this application -- it wouldn't have been trying to acquire hundreds of megabytes at that point in its life.  Also, after the OutOfMemoryError was thrown, the current heap size had only grown to 1,459,093,504 bytes -- there were still at least 450 megabytes available for heap growth, which the JVM hadn't attempted to use.

0
 
LVL 26

Expert Comment

by:mrcoffee365
Comment Utility
Have you tried dumping the heap at the exception?  There's a new option for the JVM called HeapDumpOnOutOfMemoryError which will dump the heap at the time of the exception.  In your case, that would be a very large heap, but if you can make the space available, then it might help.  See this page for instructions:
http://blogs.ittoolbox.com/eai/software/archives/help-on-the-way-for-dreaded-javalangoutofmemoryerror-java-heap-space-10743
0
 

Author Comment

by:ChrisVoilaweb
Comment Utility
Rats.  That's a very good suggestion, but I should have mentioned that we'd already found that flag this morning, documented by Sun at...
    http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf

We have the program running with that flag enabled, but we haven't yet been able to reproduce the bug.  We've only ever seen the bug just the one time ever.

I should also mention other things that we know about or have thought of already:

    (a) We know about the GC verbosity flags, namely:
        -verbose:gc
        -XX:+PrintGCTimeStamps
        -XX:+PrintGCDetails

    (b) We know about the GCPortal:
http://java.sun.com/developer/technicalArticles/Programming/GCPortal/

    (c) The machine has 16 Gb native memory.  We don't have a process dump from when the problem happened, but we believe that ample memory was available to the application.  Also, we are definitely in 64-bit mode.  We were able to restart the application with -Xmx6500m without any trouble, for example.

    (d) We've thought of it potentially being a ulimit issue, but the application was running as root.

    (e) We've thought of the workaround of setting the -Xms value to be the same as the -Xmx value, so that all of the potential heap is available immediately.  Shouldn't have to do that, given all of the above, absent a JVM bug, of course.

What else, guys?
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
If it is thowing an OOME then you app is trying to request a more memory than is currently available.
you need to either fix the bug that is causing the request, or if it is valid then assign more memory
0
 
LVL 13

Expert Comment

by:Bart_Cr
Comment Utility
Can you give a list of all VM options set?

Probably there was need for creating a large object and there was no space available. VM options especially the -X and -XX options tell how the heap is divided and how it can change during the lifetime of the program.

I would suggest using VisualGC to monitor the different spaces. You might see it behaving badly. Further it's interresting to know what resides on a heap at several points in time. You can use jmap together with jhat to analyze your heap.
0
 

Author Comment

by:ChrisVoilaweb
Comment Utility
We're familiar with jmap and jhat.  I've never used VisualGC, though, I'll give that a look.  This is a failure of the primary heap, though (for example, it didn't run out of perm space; the error message would have been different).

Guys, I've got 10 years continuous experience in Java.  I've debugged lots of OutOfMemory issues in the past. I can't overemphasize how sure I am that this program didn't suddenly attempt to request 450 megabytes, after running with a stable heap for 12 hours.  Stop suggesting that that's the problem, because it just isn't.  

What I'm looking for are reasons why the JVM would throw an OutOfMemoryError without first growing a heap to the maximum allocated size, in 64-bit mode, on Linux.  I know that it's a disturbing and peculiar idea, but it's obviously possible, e.g. in the event of a bug in the JVM.  But why else?
0
 
LVL 13

Accepted Solution

by:
Bart_Cr earned 168 total points
Comment Utility
Ok, I'm just guessing, because that's the only option I (we) have got without having access to the system. So, here it goes:

Another possibility might be that the heap is very fragmented so there is not enough continuous space anywhere in the heap to create a specific object.

It could also relate to the system's memory. If there's a JNI call somewhere trying to allocate memory while there's none available it might result in an OOEM. This relates to the systems memory outside the JVM.

The statement that it could be a bug also might be a valid statement.

I still would like to get a complete list of JVM options passed to the virtual machine. Understanding the type of garbage collector, the heap settings, ... would give us some pointers of where to look. I'm assuming you're using Sun's JVM.

VisualGC will give you an insight in how the heap behaves over time. It will not give you any pointers to problematic objects. For that you'll need jhat and jmap, or commercial tools like JProbe Memory Debugger, ...
0
 
LVL 26

Expert Comment

by:mrcoffee365
Comment Utility
In my experience, when the JVM gives an OutOfMemory exception, it's out of memory.  It's certainly possible that you've found a bug in the JVM for 64bit processing, in which case you're going to have to start tracking it down.

Your problem here is that it hasn't happened again, right?  So the best you can do is what you've done, set up ample debugging, checked to see if anyone knows of  random OOM exceptions in 64bit JVMs.

You never posted your VM options, so we're not sure what you've set.  I found this discussion of the new XX parms here:
http://forum.java.sun.com/thread.jspa?threadID=5236840

You might also try using a test program like this one (if you haven't already) to provoke the behavior you're seeing.

You could also check your memory hardware to see if you had a fault in the hardware.  If it can be the JVM, it can also be SUSE Linux -- you might try their support people.
0
 
LVL 92

Assisted Solution

by:objects
objects earned 166 total points
Comment Utility
the *only* reason a an OOM gets thrown is if a request for a chunk of memory is made that cannot be fulfilled after gc has occurred.
If its happening in a situation other than that then it is a bug in the jvm and there is nothing you can except report it.
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 

Author Comment

by:ChrisVoilaweb
Comment Utility
I've asked the tester to send me the VM options that he was running under.  Coming soon!

0
 

Author Comment

by:ChrisVoilaweb
Comment Utility
The VM options used for the failed test were:

-Djavax.net.ssl.trustStore=<filename>
-Xms64m
-Xmx2048m
-XX:PermSize=100m
-XX:MaxPermSize=150m
-Djava.security.policy=<filename>
-Dsun.java2d.noddraw=true
-Dsun.rmi.dgc.client.gcInterval=600000
-Dsun.rmi.dgc.server.gcInterval=600000
0
 
LVL 26

Assisted Solution

by:mrcoffee365
mrcoffee365 earned 166 total points
Comment Utility
There's a good discussion of real-world experience with OOM exceptions and the effects of PermSize and MaxPermSize here:
http://www.sumoc.com/blog/index.cfm?mode=entry&entry=CDCDBF8B-5004-2066-B7460CDEAB79328F
If you have a lot of classes loaded in your JVM (or it has recently gone up quite a bit) then that might be why you suddenly need more memory -- the MaxPermSize is being hit.

It's another variable to consider in your debugging.

Also -- this isn't related to the OOM error, but if you think your application grows as much as you've reported, your Xms of 64m seems oddly small.  Why not allocate more at the beginning?
0
 

Author Comment

by:ChrisVoilaweb
Comment Utility
Valid point about upping the -Xms -- that's just inertia from the default install settings.  The product can run in 384m in smaller configurations.  But sure, when running here with an mx of 2g, then an ms of 1g is certainly reasonable.

As for the perm max -- yes, that was an oversight by the tester, the PermSize and MaxPermSize were meant to have been raised.  However, I wouldn't think that the Perm Max was hit, because the error message was "Java heap space", and I'm pretty sure that when you hit the perm limit, Java 5 and beyond mention "perm space" in the error message.

Still, easy to imagine that they overlooked updating the error message in one place (because the older JVM's didn't use to give you that clue), and that's a bit easier to imagine than a complete failure to ask for memory that was available.
0
 
LVL 2

Expert Comment

by:egtebert
Comment Utility
Hi,
I have a similar problem with this application we bought and they can`t seem to find it yet so I try to help them.
I found this one site that seems to have the sollution, I`m not sure yet if it helps:

http://www.mcse.ms/message2722033.html
0
 
LVL 26

Expert Comment

by:mrcoffee365
Comment Utility
There are several important differences between the application mentioned in egtebert's link and this question, but the most important one is that this question is about a Java application, and the link above is about a Visual Studio application -- C# or VB, probably.

It's somewhat moot, since the link above offers no information or solutions to the  OutOfMemory message.
0
 
LVL 2

Expert Comment

by:egtebert
Comment Utility
but it all has to do with IIS right? There might be a commen relation...?
0
 
LVL 2

Expert Comment

by:egtebert
Comment Utility
At the end of the comments from the link it says to add bit by bit the code to find where it goes wrong, severeal other people where helped by this. and the link is more helping then just complaining...
0
 
LVL 2

Expert Comment

by:egtebert
Comment Utility
visual studio might be different but the COMMON problem is """ASP .NET"""
0
 
LVL 26

Expert Comment

by:mrcoffee365
Comment Utility
I think you can see from the excellent configuration info given in the question that we're talking about SUSE Linux and a Java application.  No Windows, no ASP.Net.
0
 
LVL 26

Expert Comment

by:mrcoffee365
Comment Utility
It seems reasonable to me to include 3 of us for the points on this one:
mrcoffee365, objects, Bart_Cr
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Introduction This article is the second of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers the basic installation and configuration of the test automation tools used by…
Viewers will learn about basic arrays, how to declare them, and how to use them. Introduction and definition: Declare an array and cover the syntax of declaring them: Initialize every index in the created array: Example/Features of a basic arr…
This video teaches viewers about errors in exception handling.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now