Solved

outofmemory exceptions, please help

Posted on 2004-03-30
28
582 Views
Last Modified: 2008-02-01
hey we are running wls70 in a farm environment on redhat linux enterprise and keep getting out of memory exceptions for large data sets from xml (we generate the webpages from the xml file).

The wls group already increased our memory/heap size to 700mb which should be more than enough. It only happens to 1 particular instance when there is a very large dataset.  We were not able to use any of the memory debugger tools such as jprobe since we access our linux server via ssh and there is no graphic ui attached to it.

Here's the latest log after 1 day of use since reboot. If anyone can shed some light on this it will be greatly appreciated. thanks.

[ 8453][memory ] 672.883: GC 356933K->154974K (524288K) in 425.388 ms
[ 8453][memory ] 673.389: GC 165891K->151619K (524288K) in 342.962 ms
[ 8453][memory ] 673.739: GC 151619K->151083K (524288K) in 502.063 ms
[ 8453][memory ] 674.249: GC 151083K->150134K (524288K) in 437.784 ms
[ 8453][memory ] 674.697: GC 150134K->150082K (524288K) in 383.152 ms
[ 8453][memory ] 675.098: GC 150082K->149416K (524288K) in 655.017 ms
[ 8453][memory ] 675.762: GC 149416K->144247K (524288K) in 639.520 ms
[ 8453][memory ] 676.407: GC 144247K->135180K (524288K) in 515.898 ms
[ 8453][memory ] 676.930: GC 135180K->127807K (524288K) in 476.655 ms
[ 8453][memory ] 677.412: GC 127807K->117806K (524288K) in 693.205 ms
[ 8453][memory ] 21373.560: GC 487584K->146378K (524288K) in 484.512 ms


[ 8454][memory ] 21483.893: GC 524288K->117880K (524288K) in 386.723 ms
<Mar 25, 2004 4:36:02 AM EST> <Info> <HTTP> <101047> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] /*: init>
<Mar 25, 2004 4:36:02 AM EST> <Info> <HTTP> <101047> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] /*: Using standard I/O>
mon : 3
mon : 3
[ 8454][memory ] 43206.047: GC 499684K->155802K (524288K) in 383.893 ms
mon : 3
mon : 3
mon : 3
mon : 3
mon : 3
[ 8453][memory ] 55175.603: GC 293354K->147537K (524288K) in 287.490 ms
[ 8453][memory ] 55175.899: GC 147537K->141312K (524288K) in 237.305 ms
[ 8453][memory ] 55176.140: GC 141312K->141234K (524288K) in 266.500 ms
[ 8453][memory ] 55176.412: GC 141234K->139276K (524288K) in 313.818 ms
[ 8453][memory ] 55176.730: GC 139276K->136913K (524288K) in 390.718 ms
[ 8453][memory ] 55177.128: GC 136913K->135383K (524288K) in 481.528 ms
[ 8453][memory ] 55177.654: GC 152115K->137978K (524288K) in 242.307 ms
[ 8453][memory ] 55177.901: GC 137978K->137956K (524288K) in 229.091 ms
[ 8453][memory ] 55178.134: GC 137956K->137724K (524288K) in 278.285 ms
[ 8453][memory ] 55178.417: GC 137724K->137298K (524288K) in 282.001 ms
[ 8453][memory ] 55178.703: GC 137298K->136364K (524288K) in 345.572 ms
[ 8453][memory ] 55179.053: GC 136364K->136296K (524288K) in 448.977 ms
[ 8453][memory ] 55179.512: GC 136296K->135054K (524288K) in 666.687 ms
[ 8453][memory ] 55180.269: GC 143692K->138923K (524288K) in 230.474 ms
[ 8453][memory ] 55180.503: GC 138923K->138875K (524288K) in 223.139 ms
[ 8453][memory ] 55180.730: GC 138875K->138649K (524288K) in 298.802 ms
[ 8453][memory ] 55181.032: GC 138649K->138631K (524288K) in 273.352 ms
[ 8453][memory ] 55181.308: GC 138631K->138597K (524288K) in 246.089 ms
[ 8453][memory ] 55181.557: GC 138597K->138534K (524288K) in 252.921 ms
[ 8453][memory ] 55181.813: GC 138534K->138393K (524288K) in 301.996 ms
[ 8453][memory ] 55182.119: GC 138393K->123204K (524288K) in 1554.888 ms
<Mar 25, 2004 9:46:30 AM EST> <Error> <HTTP> <101017> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] Root cause of ServletException
java.lang.OutOfMemoryError
0
Comment
Question by:gagaliya
  • 8
  • 8
  • 6
  • +2
28 Comments
 
LVL 92

Expert Comment

by:objects
Comment Utility
do you know which code is actually causing the OutOfMemoryError
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
Is this a DOM or a SAX operation? The former is always relatively much more memory intensive than the latter. You should SAX process wherever possible
0
 
LVL 1

Author Comment

by:gagaliya
Comment Utility
Hey I did not design this app so i aplogize for being a bit fuzzy on the details. But it is a relatively standard web app hooked to a db and use xml to generate html. I am not sure if it is dom or sax(can you tell from the log i provided below?).  The only suspicious thing(besides outofmemory) from the log are those exceptions below, i am not sure if they have anything at all to do with outofmemory errors. Of course those exception messages stop at the package class so we dont even know which user class called and they are completely random as far as we can tell.  

Out of memory exception happens at various pages/code, the only thing we can determine is it happens when there is a very large xml data set. So for most clients it runs fine indefinitly, then for one particular client with a lot of data it causes outofmemory exceptions after a day or 2 of usage. Again the memory heap is at 700 mb for our wls server, it still should be more than enough to handle the large datasets.  Thanks for your help!!

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)
        at org.apache.xalan.serialize.SerializerToHTML.endElement(Ljava.lang.Str
ing;Ljava.lang.String;Ljava.lang.String;)V(SerializerToHTML.java:656)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.
String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:280)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xala
n.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(E
lemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lo
rg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.
utils.QName;Z)V(TransformerImpl.java:2251)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Sourc
e)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToXML.accum(C)V(SerializerToXML.
java:1321)
        at org.apache.xalan.serialize.SerializerToXML.outputLineSep()V(Serialize
rToXML.java:195)
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)
        at org.apache.xalan.serialize.SerializerToHTML.endElement(Ljava.lang.Str
ing;Ljava.lang.String;Ljava.lang.String;)V(SerializerToHTML.java:656)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.
String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:280)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToHTML.startElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;Lorg.xml.sax.Attributes;)V(Optimized Method)
        at org.apache.xalan.transformer.QueuedStartElement.flush()V(QueuedStartElement.java:357)
        at org.apache.xalan.transformer.ResultTreeHandler.flushPending(I)V(ResultTreeHandler.java:770)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:279)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xalan.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(ElemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;Z)V(TransformerImpl.java:2251)
        at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)Z(TransformerImpl.java:2134)
        at org.apache.xalan.transformer.TransformerImpl.transformNode(Lorg.w3c.dom.Node;)V(TransformerImpl.java:1246)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
mon : 3
javax.xml.transform.TransformerException
        at org.apache.xalan.transformer.TransformerImpl.transformNode(Lorg.w3c.dom.Node;)V(TransformerImpl.java:1269)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToHTML.startElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;Lorg.xml.sax.Attributes;)V(Optimized Method)
        at org.apache.xalan.transformer.QueuedStartElement.flush()V(QueuedStartElement.java:357)
        at org.apache.xalan.transformer.ResultTreeHandler.flushPending(I)V(ResultTreeHandler.java:770)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:279)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xalan.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(ElemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;Z)V(TransformerImpl.java:2251)
        at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)Z(TransformerImpl.java:2134)
Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace

End  server side stack trace

End  server side stack trace

>
mon : 3
mon : 3
mon : 3
mon : 3
mon : 3
0
 
LVL 1

Author Comment

by:gagaliya
Comment Utility
sorry ignore this part, i pasted it wrong from vim. they are not related at all

Start server side stack trace:
java.lang.OutOfMemoryError:  (not related to the line below)
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)  (not related to the out of memory exception above)
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
Is it just the one XML file you are having problems with?
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
if you restart the server and then immediately try to load one of the large data sets do you get the exception?
0
 
LVL 2

Expert Comment

by:Moroni24
Comment Utility
When you say " a very large dataset" how many bytes are you talking? Like CEHJ mentioned, DOM is ver memory intensive because it loads the ENTIRE xml document before doing anything with it.

Howver, since you didn't write the app, I'm assuming you don't access to the source, which will prevent you from switching to sax. Maybe it would be best to par down the dataset if at all possible.

 
0
 
LVL 1

Author Comment

by:gagaliya
Comment Utility
-Is it just the one XML file you are having problems with?

No, it has nothing to do with the XML file format. The xml file contains different data and format for each load, the outofmemory exception happens when the XML file contains a very large set of data.


-if you restart the server and then immediately try to load one of the large data sets do you get the exception?

No, it loads ok. Outofmemory only happens on large data sets after a period of time. Usually one or two days. This led me to believe it is a memory leak issue not lack of physical memory

-When you say " a very large dataset" how many bytes are you talking? Like CEHJ mentioned, DOM is ver memory intensive because it loads the ENTIRE xml document before doing anything with it.
Howver, since you didn't write the app, I'm assuming you don't access to the source, which will prevent you from switching to sax. Maybe it would be best to par down the dataset if at all possible.

The page(which caused outofmemory) containing both the html code and the entire xml sheet is 3.55 MB.  How do i tell if it is using DOM or SAX?
I do have full access to the source, just not too familiar with some of the details.  Is the switch to sax easy? or does it require a complete rewrite.  Cutting up the data is really not an option at the moment.

thanks again for your time!

gaga




0
 
LVL 92

Assisted Solution

by:objects
objects earned 200 total points
Comment Utility
> Outofmemory only happens on large data sets after a period of time.

Then it sounds like you have a memory leak somewhere. So switching to SAX won't fix the problem, what you need to do is determine where the leak is occurring.
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 200 total points
Comment Utility
>>So switching to SAX won't fix the problem

Unless it happens to be the libraries doing the DOM stuff (quite likely)
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
So according to CEHJ never use DOM :-D
0
 
LVL 9

Expert Comment

by:mmuruganandam
Comment Utility
Which JDK version are you using.

In JDK 1.4.2 (not sure), there is a memory leak in the StringBuffer.  That might cause this problem.  Use the latest version of JDK, that might help you in this case.


Regards,
Muruga
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
>>So according to CEHJ never use DOM :-D

Quite right - if it's avoidable

>>In JDK 1.4.2 (not sure), there is a memory leak in the StringBuffer

? Please expand...
0
 
LVL 9

Expert Comment

by:mmuruganandam
Comment Utility
In JDK itself there is a memory leak in the StringBuffer code.  So they came up with a patch.

I am not quite sure about the JDK version.  But pretty much sure on 1.4.*

Since SAX parsers are using StringBuffer for all the manipulation, there is a high chance of OutOfMemoryError.

0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
>>.  So they came up with a patch.

I haven't been able to find anything on this.
0
 
LVL 9

Assisted Solution

by:mmuruganandam
mmuruganandam earned 100 total points
Comment Utility
0
 
LVL 9

Expert Comment

by:mmuruganandam
Comment Utility
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
Hmm. That's still seems to be in the realms of 'hearsay' unfortunately
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
(My last was to your last-before-last)
0
 
LVL 9

Expert Comment

by:mmuruganandam
Comment Utility
it is 1.4.1_x

0
 
LVL 9

Expert Comment

by:mmuruganandam
Comment Utility
check it out... I am not too sure about it.
0
 
LVL 1

Author Comment

by:gagaliya
Comment Utility
hey, i know there is a leak the problem is i don't know how to go about fixing it. since the codes are all on server side and we can only access it via a console. I guess my real questions:

1) how do i know which parse i am using dom / sax
2) are there any good memory debugger tools that will work with just telnet, since i dont have any gui to linux.

thanks

gaga
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
Many profilers allow you to connect to the running process via socket, would this help you?
0
 
LVL 92

Expert Comment

by:objects
Comment Utility
> That's still seems to be in the realms of 'hearsay' unfortunately

As does your advice to never use DOM.
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
>>As does your advice to never use DOM.

Your comments suggest you haven't read the thread carefully enough
0
 
LVL 1

Author Comment

by:gagaliya
Comment Utility
thanks, i managed to get jprobe working with exceed. will see what happens
0
 
LVL 86

Expert Comment

by:CEHJ
Comment Utility
:-)

Well done for finding that link mmuruganandam, that's a nasty little bug, although i haven't
looked for it in any source i'm using. It must be said that StringBuffers don't actually need to
be buggy in order to cause memory problems - quite a few people don't know how to use
them in an efficient way so they can 'leak' memory anyway.

That's just character buffering though. The real leakiness i would guess (i have never looked at the source
of a DOM implementation) would tend to arise from the 'gotchas' that can occur through reference
management and the garbage collector.

0
 
LVL 92

Expert Comment

by:objects
Comment Utility
>  i managed to get jprobe working with exceed.

yes jprobe is one that supports remote debugging.
0

Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Suggested Solutions

Java Flight Recorder and Java Mission Control together create a complete tool chain to continuously collect low level and detailed runtime information enabling after-the-fact incident analysis. Java Flight Recorder is a profiling and event collectio…
Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
Viewers learn about the third conditional statement “else if” and use it in an example program. Then additional information about conditional statements is provided, covering the topic thoroughly. Viewers learn about the third conditional statement …
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now