Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 597
  • Last Modified:

outofmemory exceptions, please help

hey we are running wls70 in a farm environment on redhat linux enterprise and keep getting out of memory exceptions for large data sets from xml (we generate the webpages from the xml file).

The wls group already increased our memory/heap size to 700mb which should be more than enough. It only happens to 1 particular instance when there is a very large dataset.  We were not able to use any of the memory debugger tools such as jprobe since we access our linux server via ssh and there is no graphic ui attached to it.

Here's the latest log after 1 day of use since reboot. If anyone can shed some light on this it will be greatly appreciated. thanks.

[ 8453][memory ] 672.883: GC 356933K->154974K (524288K) in 425.388 ms
[ 8453][memory ] 673.389: GC 165891K->151619K (524288K) in 342.962 ms
[ 8453][memory ] 673.739: GC 151619K->151083K (524288K) in 502.063 ms
[ 8453][memory ] 674.249: GC 151083K->150134K (524288K) in 437.784 ms
[ 8453][memory ] 674.697: GC 150134K->150082K (524288K) in 383.152 ms
[ 8453][memory ] 675.098: GC 150082K->149416K (524288K) in 655.017 ms
[ 8453][memory ] 675.762: GC 149416K->144247K (524288K) in 639.520 ms
[ 8453][memory ] 676.407: GC 144247K->135180K (524288K) in 515.898 ms
[ 8453][memory ] 676.930: GC 135180K->127807K (524288K) in 476.655 ms
[ 8453][memory ] 677.412: GC 127807K->117806K (524288K) in 693.205 ms
[ 8453][memory ] 21373.560: GC 487584K->146378K (524288K) in 484.512 ms


[ 8454][memory ] 21483.893: GC 524288K->117880K (524288K) in 386.723 ms
<Mar 25, 2004 4:36:02 AM EST> <Info> <HTTP> <101047> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] /*: init>
<Mar 25, 2004 4:36:02 AM EST> <Info> <HTTP> <101047> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] /*: Using standard I/O>
mon : 3
mon : 3
[ 8454][memory ] 43206.047: GC 499684K->155802K (524288K) in 383.893 ms
mon : 3
mon : 3
mon : 3
mon : 3
mon : 3
[ 8453][memory ] 55175.603: GC 293354K->147537K (524288K) in 287.490 ms
[ 8453][memory ] 55175.899: GC 147537K->141312K (524288K) in 237.305 ms
[ 8453][memory ] 55176.140: GC 141312K->141234K (524288K) in 266.500 ms
[ 8453][memory ] 55176.412: GC 141234K->139276K (524288K) in 313.818 ms
[ 8453][memory ] 55176.730: GC 139276K->136913K (524288K) in 390.718 ms
[ 8453][memory ] 55177.128: GC 136913K->135383K (524288K) in 481.528 ms
[ 8453][memory ] 55177.654: GC 152115K->137978K (524288K) in 242.307 ms
[ 8453][memory ] 55177.901: GC 137978K->137956K (524288K) in 229.091 ms
[ 8453][memory ] 55178.134: GC 137956K->137724K (524288K) in 278.285 ms
[ 8453][memory ] 55178.417: GC 137724K->137298K (524288K) in 282.001 ms
[ 8453][memory ] 55178.703: GC 137298K->136364K (524288K) in 345.572 ms
[ 8453][memory ] 55179.053: GC 136364K->136296K (524288K) in 448.977 ms
[ 8453][memory ] 55179.512: GC 136296K->135054K (524288K) in 666.687 ms
[ 8453][memory ] 55180.269: GC 143692K->138923K (524288K) in 230.474 ms
[ 8453][memory ] 55180.503: GC 138923K->138875K (524288K) in 223.139 ms
[ 8453][memory ] 55180.730: GC 138875K->138649K (524288K) in 298.802 ms
[ 8453][memory ] 55181.032: GC 138649K->138631K (524288K) in 273.352 ms
[ 8453][memory ] 55181.308: GC 138631K->138597K (524288K) in 246.089 ms
[ 8453][memory ] 55181.557: GC 138597K->138534K (524288K) in 252.921 ms
[ 8453][memory ] 55181.813: GC 138534K->138393K (524288K) in 301.996 ms
[ 8453][memory ] 55182.119: GC 138393K->123204K (524288K) in 1554.888 ms
<Mar 25, 2004 9:46:30 AM EST> <Error> <HTTP> <101017> <[ServletContext(id=154491569,name=ABC,context-path=/ABC)] Root cause of ServletException
java.lang.OutOfMemoryError
0
gagaliya
Asked:
gagaliya
  • 8
  • 8
  • 6
  • +2
3 Solutions
 
objectsCommented:
do you know which code is actually causing the OutOfMemoryError
0
 
CEHJCommented:
Is this a DOM or a SAX operation? The former is always relatively much more memory intensive than the latter. You should SAX process wherever possible
0
 
gagaliyaAuthor Commented:
Hey I did not design this app so i aplogize for being a bit fuzzy on the details. But it is a relatively standard web app hooked to a db and use xml to generate html. I am not sure if it is dom or sax(can you tell from the log i provided below?).  The only suspicious thing(besides outofmemory) from the log are those exceptions below, i am not sure if they have anything at all to do with outofmemory errors. Of course those exception messages stop at the package class so we dont even know which user class called and they are completely random as far as we can tell.  

Out of memory exception happens at various pages/code, the only thing we can determine is it happens when there is a very large xml data set. So for most clients it runs fine indefinitly, then for one particular client with a lot of data it causes outofmemory exceptions after a day or 2 of usage. Again the memory heap is at 700 mb for our wls server, it still should be more than enough to handle the large datasets.  Thanks for your help!!

Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)
        at org.apache.xalan.serialize.SerializerToHTML.endElement(Ljava.lang.Str
ing;Ljava.lang.String;Ljava.lang.String;)V(SerializerToHTML.java:656)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.
String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:280)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xala
n.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(E
lemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lo
rg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.
utils.QName;Z)V(TransformerImpl.java:2251)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Sourc
e)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToXML.accum(C)V(SerializerToXML.
java:1321)
        at org.apache.xalan.serialize.SerializerToXML.outputLineSep()V(Serialize
rToXML.java:195)
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)
        at org.apache.xalan.serialize.SerializerToHTML.endElement(Ljava.lang.Str
ing;Ljava.lang.String;Ljava.lang.String;)V(SerializerToHTML.java:656)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.
String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:280)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToHTML.startElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;Lorg.xml.sax.Attributes;)V(Optimized Method)
        at org.apache.xalan.transformer.QueuedStartElement.flush()V(QueuedStartElement.java:357)
        at org.apache.xalan.transformer.ResultTreeHandler.flushPending(I)V(ResultTreeHandler.java:770)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:279)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xalan.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(ElemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;Z)V(TransformerImpl.java:2251)
        at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)Z(TransformerImpl.java:2134)
        at org.apache.xalan.transformer.TransformerImpl.transformNode(Lorg.w3c.dom.Node;)V(TransformerImpl.java:1246)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
mon : 3
javax.xml.transform.TransformerException
        at org.apache.xalan.transformer.TransformerImpl.transformNode(Lorg.w3c.dom.Node;)V(TransformerImpl.java:1269)
        at org.apache.xalan.transformer.TransformerImpl.run()V(TransformerImpl.java:3070)
        at java.lang.Thread.run()V(Unknown Source)
        at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
---------
java.lang.ArrayIndexOutOfBoundsException
        at org.apache.xalan.serialize.SerializerToHTML.startElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;Lorg.xml.sax.Attributes;)V(Optimized Method)
        at org.apache.xalan.transformer.QueuedStartElement.flush()V(QueuedStartElement.java:357)
        at org.apache.xalan.transformer.ResultTreeHandler.flushPending(I)V(ResultTreeHandler.java:770)
        at org.apache.xalan.transformer.ResultTreeHandler.endElement(Ljava.lang.String;Ljava.lang.String;Ljava.lang.String;)V(ResultTreeHandler.java:279)
        at org.apache.xalan.templates.ElemLiteralResult.execute(Lorg.apache.xalan.transformer.TransformerImpl;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)V(ElemLiteralResult.java:643)
        at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;Z)V(TransformerImpl.java:2251)
        at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.apache.xalan.templates.ElemTemplateElement;Lorg.w3c.dom.Node;Lorg.apache.xml.utils.QName;)Z(TransformerImpl.java:2134)
Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace


Start server side stack trace:
java.lang.OutOfMemoryError:
Start server side stack trace:
java.lang.OutOfMemoryError
End  server side stack trace

End  server side stack trace

End  server side stack trace

End  server side stack trace

>
mon : 3
mon : 3
mon : 3
mon : 3
mon : 3
0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
gagaliyaAuthor Commented:
sorry ignore this part, i pasted it wrong from vim. they are not related at all

Start server side stack trace:
java.lang.OutOfMemoryError:  (not related to the line below)
        at org.apache.xalan.serialize.SerializerToXML.indent(I)V(SerializerToXML
.java:2241)  (not related to the out of memory exception above)
0
 
objectsCommented:
Is it just the one XML file you are having problems with?
0
 
objectsCommented:
if you restart the server and then immediately try to load one of the large data sets do you get the exception?
0
 
Moroni24Commented:
When you say " a very large dataset" how many bytes are you talking? Like CEHJ mentioned, DOM is ver memory intensive because it loads the ENTIRE xml document before doing anything with it.

Howver, since you didn't write the app, I'm assuming you don't access to the source, which will prevent you from switching to sax. Maybe it would be best to par down the dataset if at all possible.

 
0
 
gagaliyaAuthor Commented:
-Is it just the one XML file you are having problems with?

No, it has nothing to do with the XML file format. The xml file contains different data and format for each load, the outofmemory exception happens when the XML file contains a very large set of data.


-if you restart the server and then immediately try to load one of the large data sets do you get the exception?

No, it loads ok. Outofmemory only happens on large data sets after a period of time. Usually one or two days. This led me to believe it is a memory leak issue not lack of physical memory

-When you say " a very large dataset" how many bytes are you talking? Like CEHJ mentioned, DOM is ver memory intensive because it loads the ENTIRE xml document before doing anything with it.
Howver, since you didn't write the app, I'm assuming you don't access to the source, which will prevent you from switching to sax. Maybe it would be best to par down the dataset if at all possible.

The page(which caused outofmemory) containing both the html code and the entire xml sheet is 3.55 MB.  How do i tell if it is using DOM or SAX?
I do have full access to the source, just not too familiar with some of the details.  Is the switch to sax easy? or does it require a complete rewrite.  Cutting up the data is really not an option at the moment.

thanks again for your time!

gaga




0
 
objectsCommented:
> Outofmemory only happens on large data sets after a period of time.

Then it sounds like you have a memory leak somewhere. So switching to SAX won't fix the problem, what you need to do is determine where the leak is occurring.
0
 
CEHJCommented:
>>So switching to SAX won't fix the problem

Unless it happens to be the libraries doing the DOM stuff (quite likely)
0
 
objectsCommented:
So according to CEHJ never use DOM :-D
0
 
mmuruganandamCommented:
Which JDK version are you using.

In JDK 1.4.2 (not sure), there is a memory leak in the StringBuffer.  That might cause this problem.  Use the latest version of JDK, that might help you in this case.


Regards,
Muruga
0
 
CEHJCommented:
>>So according to CEHJ never use DOM :-D

Quite right - if it's avoidable

>>In JDK 1.4.2 (not sure), there is a memory leak in the StringBuffer

? Please expand...
0
 
mmuruganandamCommented:
In JDK itself there is a memory leak in the StringBuffer code.  So they came up with a patch.

I am not quite sure about the JDK version.  But pretty much sure on 1.4.*

Since SAX parsers are using StringBuffer for all the manipulation, there is a high chance of OutOfMemoryError.

0
 
CEHJCommented:
>>.  So they came up with a patch.

I haven't been able to find anything on this.
0
 
CEHJCommented:
Hmm. That's still seems to be in the realms of 'hearsay' unfortunately
0
 
CEHJCommented:
(My last was to your last-before-last)
0
 
mmuruganandamCommented:
it is 1.4.1_x

0
 
mmuruganandamCommented:
check it out... I am not too sure about it.
0
 
gagaliyaAuthor Commented:
hey, i know there is a leak the problem is i don't know how to go about fixing it. since the codes are all on server side and we can only access it via a console. I guess my real questions:

1) how do i know which parse i am using dom / sax
2) are there any good memory debugger tools that will work with just telnet, since i dont have any gui to linux.

thanks

gaga
0
 
objectsCommented:
Many profilers allow you to connect to the running process via socket, would this help you?
0
 
objectsCommented:
> That's still seems to be in the realms of 'hearsay' unfortunately

As does your advice to never use DOM.
0
 
CEHJCommented:
>>As does your advice to never use DOM.

Your comments suggest you haven't read the thread carefully enough
0
 
gagaliyaAuthor Commented:
thanks, i managed to get jprobe working with exceed. will see what happens
0
 
CEHJCommented:
:-)

Well done for finding that link mmuruganandam, that's a nasty little bug, although i haven't
looked for it in any source i'm using. It must be said that StringBuffers don't actually need to
be buggy in order to cause memory problems - quite a few people don't know how to use
them in an efficient way so they can 'leak' memory anyway.

That's just character buffering though. The real leakiness i would guess (i have never looked at the source
of a DOM implementation) would tend to arise from the 'gotchas' that can occur through reference
management and the garbage collector.

0
 
objectsCommented:
>  i managed to get jprobe working with exceed.

yes jprobe is one that supports remote debugging.
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

  • 8
  • 8
  • 6
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now