Link to home
Start Free TrialLog in
Avatar of gagaliya
gagaliya

asked on

outofmemory exceptions, please help

hey we are running wls70 in a farm environment on redhat linux enterprise and keep getting out of memory exceptions for large data sets from xml (we generate the webpages from the xml file).

The wls group already increased our memory/heap size to 700mb which should be more than enough. It only happens to 1 particular instance when there is a very large dataset.  We were not able to use any of the memory debugger tools such as jprobe since we access our linux server via ssh and there is no graphic ui attached to it.

Here's the latest log after 1 day of use since reboot. If anyone can shed some light on this it will be greatly appreciated. thanks.

[ 8453][memory ] 672.883: GC 356933K->154974K (524288K) in 425.388 ms
[ 8453][memory ] 673.389: GC 165891K->151619K (524288K) in 342.962 ms
[ 8453][memory ] 673.739: GC 151619K->151083K (524288K) in 502.063 ms
[ 8453][memory ] 674.249: GC 151083K->150134K (524288K) in 437.784 ms
[ 8453][memory ] 674.697: GC 150134K->150082K (524288K) in 383.152 ms
[ 8453][memory ] 675.098: GC 150082K->149416K (524288K) in 655.017 ms
[ 8453][memory ] 675.762: GC 149416K->144247K (524288K) in 639.520 ms
[ 8453][memory ] 676.407: GC 144247K->135180K (524288K) in 515.898 ms
[ 8453][memory ] 676.930: GC 135180K->127807K (524288K) in 476.655 ms
[ 8453][memory ] 677.412: GC 127807K->117806K (524288K) in 693.205 ms
[ 8453][memory ] 21373.560: GC 487584K->146378K (524288K) in 484.512 ms


[ 8454][memory ] 21483.893: GC 524288K->117880K (524288K) in 386.723 ms
<Mar 25, 2004 4:36:02 AM EST> <Info> <HTTP> <101047> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] /*: init>
<Mar 25, 2004 4:36:02 AM EST> <Info> <HTTP> <101047> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] /*: Using standard I/O>
mon : 3
mon : 3
[ 8454][memory ] 43206.047: GC 499684K->155802K (524288K) in 383.893 ms
mon : 3
mon : 3
mon : 3
mon : 3
mon : 3
[ 8453][memory ] 55175.603: GC 293354K->147537K (524288K) in 287.490 ms
[ 8453][memory ] 55175.899: GC 147537K->141312K (524288K) in 237.305 ms
[ 8453][memory ] 55176.140: GC 141312K->141234K (524288K) in 266.500 ms
[ 8453][memory ] 55176.412: GC 141234K->139276K (524288K) in 313.818 ms
[ 8453][memory ] 55176.730: GC 139276K->136913K (524288K) in 390.718 ms
[ 8453][memory ] 55177.128: GC 136913K->135383K (524288K) in 481.528 ms
[ 8453][memory ] 55177.654: GC 152115K->137978K (524288K) in 242.307 ms
[ 8453][memory ] 55177.901: GC 137978K->137956K (524288K) in 229.091 ms
[ 8453][memory ] 55178.134: GC 137956K->137724K (524288K) in 278.285 ms
[ 8453][memory ] 55178.417: GC 137724K->137298K (524288K) in 282.001 ms
[ 8453][memory ] 55178.703: GC 137298K->136364K (524288K) in 345.572 ms
[ 8453][memory ] 55179.053: GC 136364K->136296K (524288K) in 448.977 ms
[ 8453][memory ] 55179.512: GC 136296K->135054K (524288K) in 666.687 ms
[ 8453][memory ] 55180.269: GC 143692K->138923K (524288K) in 230.474 ms
[ 8453][memory ] 55180.503: GC 138923K->138875K (524288K) in 223.139 ms
[ 8453][memory ] 55180.730: GC 138875K->138649K (524288K) in 298.802 ms
[ 8453][memory ] 55181.032: GC 138649K->138631K (524288K) in 273.352 ms
[ 8453][memory ] 55181.308: GC 138631K->138597K (524288K) in 246.089 ms
[ 8453][memory ] 55181.557: GC 138597K->138534K (524288K) in 252.921 ms
[ 8453][memory ] 55181.813: GC 138534K->138393K (524288K) in 301.996 ms
[ 8453][memory ] 55182.119: GC 138393K->123204K (524288K) in 1554.888 ms
<Mar 25, 2004 9:46:30 AM EST> <Error> <HTTP> <101017> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] Root cause of ServletException
java.lang.OutOfMemoryError
Avatar of Mick Barry
Mick Barry
Flag of Australia image
do you know which code is actually causing the OutOfMemoryError
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
Is this a DOM or a SAX operation? The former is always relatively much more memory intensive than the latter. You should SAX process wherever possible
Avatar of gagaliya
gagaliya

ASKER

Hey I did not design this app so i aplogize for being a bit fuzzy on the details. But it is a relatively standard web app hooked to a db and use xml to generate html. I am not sure if it is dom or sax(can you tell from the log i provided below?).  The only suspicious thing(besides outofmemory) from the log are those exceptions below, i am not sure if they have anything at all to do with outofmemory errors. Of course those exception messages stop at the package class so we dont even know which user class called and they are completely random as far as we can tell.  

Out of memory exception happens at various pages/code, the only thing we can determine is it happens when there is a very large xml data set. So for most clients it runs fine indefinitly, then for one particular client with a lot of data it causes outofmemory exceptions after a day or 2 of usage. Again the memory heap is at 700 mb for our wls server, it still should be more than enough to handle the large datasets.  Thanks for your help!!

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)
        at org.apache.xalan.serialize.SerializerToHTML.endElement(Ljava.lang.Str
ing;Ljava.lang.String;Ljava.lang.String;)V(SerializerToHTML.java:656)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.
String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:280)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xala
n.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(E
lemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lo
rg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.
utils.QName;Z)V(TransformerImpl.java:2251)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Sourc
e)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToXML.accum(C)V(SerializerToXML.
java:1321)
        at org.apache.xalan.serialize.SerializerToXML.outputLineSep()V(Serialize
rToXML.java:195)
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)
        at org.apache.xalan.serialize.SerializerToHTML.endElement(Ljava.lang.Str
ing;Ljava.lang.String;Ljava.lang.String;)V(SerializerToHTML.java:656)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.
String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:280)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToHTML.startElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;Lorg.xml.sax.Attributes;)V(Optimized Method)
        at org.apache.xalan.transformer.QueuedStartElement.flush()V(QueuedStartElement.java:357)
        at org.apache.xalan.transformer.ResultTreeHandler.flushPending(I)V(ResultTreeHandler.java:770)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:279)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xalan.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(ElemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;Z)V(TransformerImpl.java:2251)
        at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)Z(TransformerImpl.java:2134)
        at org.apache.xalan.transformer.TransformerImpl.transformNode(Lorg.w3c.dom.Node;)V(TransformerImpl.java:1246)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
mon : 3
javax.xml.transform.TransformerException
        at org.apache.xalan.transformer.TransformerImpl.transformNode(Lorg.w3c.dom.Node;)V(TransformerImpl.java:1269)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToHTML.startElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;Lorg.xml.sax.Attributes;)V(Optimized Method)
        at org.apache.xalan.transformer.QueuedStartElement.flush()V(QueuedStartElement.java:357)
        at org.apache.xalan.transformer.ResultTreeHandler.flushPending(I)V(ResultTreeHandler.java:770)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:279)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xalan.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(ElemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;Z)V(TransformerImpl.java:2251)
        at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)Z(TransformerImpl.java:2134)
Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace

End  server side stack trace

End  server side stack trace

>
mon : 3
mon : 3
mon : 3
mon : 3
mon : 3
Avatar of gagaliya
gagaliya

ASKER

sorry ignore this part, i pasted it wrong from vim. they are not related at all

Start server side stack trace:
java.lang.OutOfMemoryError:  (not related to the line below)
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)  (not related to the out of memory exception above)
Avatar of Mick Barry
Mick Barry
Flag of Australia image
Is it just the one XML file you are having problems with?
Avatar of Mick Barry
Mick Barry
Flag of Australia image
if you restart the server and then immediately try to load one of the large data sets do you get the exception?
Avatar of Moroni24
Moroni24
When you say " a very large dataset" how many bytes are you talking? Like CEHJ mentioned, DOM is ver memory intensive because it loads the ENTIRE xml document before doing anything with it.

Howver, since you didn't write the app, I'm assuming you don't access to the source, which will prevent you from switching to sax. Maybe it would be best to par down the dataset if at all possible.

 
Avatar of gagaliya
gagaliya

ASKER

-Is it just the one XML file you are having problems with?

No, it has nothing to do with the XML file format. The xml file contains different data and format for each load, the outofmemory exception happens when the XML file contains a very large set of data.


-if you restart the server and then immediately try to load one of the large data sets do you get the exception?

No, it loads ok. Outofmemory only happens on large data sets after a period of time. Usually one or two days. This led me to believe it is a memory leak issue not lack of physical memory

-When you say " a very large dataset" how many bytes are you talking? Like CEHJ mentioned, DOM is ver memory intensive because it loads the ENTIRE xml document before doing anything with it.
Howver, since you didn't write the app, I'm assuming you don't access to the source, which will prevent you from switching to sax. Maybe it would be best to par down the dataset if at all possible.

The page(which caused outofmemory) containing both the html code and the entire xml sheet is 3.55 MB.  How do i tell if it is using DOM or SAX?
I do have full access to the source, just not too familiar with some of the details.  Is the switch to sax easy? or does it require a complete rewrite.  Cutting up the data is really not an option at the moment.

thanks again for your time!

gaga




SOLUTION
Avatar of Mick Barry
Mick Barry
Flag of Australia image
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
ASKER CERTIFIED SOLUTION
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Mick Barry
Mick Barry
Flag of Australia image
So according to CEHJ never use DOM :-D
Avatar of mmuruganandam
mmuruganandam
Flag of United States of America image
Which JDK version are you using.

In JDK 1.4.2 (not sure), there is a memory leak in the StringBuffer.  That might cause this problem.  Use the latest version of JDK, that might help you in this case.


Regards,
Muruga
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
>>So according to CEHJ never use DOM :-D

Quite right - if it's avoidable

>>In JDK 1.4.2 (not sure), there is a memory leak in the StringBuffer

? Please expand...
Avatar of mmuruganandam
mmuruganandam
Flag of United States of America image
In JDK itself there is a memory leak in the StringBuffer code.  So they came up with a patch.

I am not quite sure about the JDK version.  But pretty much sure on 1.4.*

Since SAX parsers are using StringBuffer for all the manipulation, there is a high chance of OutOfMemoryError.

Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
>>.  So they came up with a patch.

I haven't been able to find anything on this.
SOLUTION
Avatar of mmuruganandam
mmuruganandam
Flag of United States of America image
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
Hmm. That's still seems to be in the realms of 'hearsay' unfortunately
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
(My last was to your last-before-last)
Avatar of mmuruganandam
mmuruganandam
Flag of United States of America image
it is 1.4.1_x

Avatar of mmuruganandam
mmuruganandam
Flag of United States of America image
check it out... I am not too sure about it.
Avatar of gagaliya
gagaliya

ASKER

hey, i know there is a leak the problem is i don't know how to go about fixing it. since the codes are all on server side and we can only access it via a console. I guess my real questions:

1) how do i know which parse i am using dom / sax
2) are there any good memory debugger tools that will work with just telnet, since i dont have any gui to linux.

thanks

gaga
Avatar of Mick Barry
Mick Barry
Flag of Australia image
Many profilers allow you to connect to the running process via socket, would this help you?
Avatar of Mick Barry
Mick Barry
Flag of Australia image
> That's still seems to be in the realms of 'hearsay' unfortunately

As does your advice to never use DOM.
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
>>As does your advice to never use DOM.

Your comments suggest you haven't read the thread carefully enough
Avatar of gagaliya
gagaliya

ASKER

thanks, i managed to get jprobe working with exceed. will see what happens
Avatar of CEHJ
CEHJ
Flag of United Kingdom of Great Britain and Northern Ireland image
:-)

Well done for finding that link mmuruganandam, that's a nasty little bug, although i haven't
looked for it in any source i'm using. It must be said that StringBuffers don't actually need to
be buggy in order to cause memory problems - quite a few people don't know how to use
them in an efficient way so they can 'leak' memory anyway.

That's just character buffering though. The real leakiness i would guess (i have never looked at the source
of a DOM implementation) would tend to arise from the 'gotchas' that can occur through reference
management and the garbage collector.

Avatar of Mick Barry
Mick Barry
Flag of Australia image
>  i managed to get jprobe working with exceed.

yes jprobe is one that supports remote debugging.