?
Solved

Weblogic - Out of Memory Error

Posted on 2012-09-06
10
Medium Priority
?
3,741 Views
Last Modified: 2012-09-14
Hi,

We have our application deployed on Weblogic 10.3.4.0. When we do a load run on this server, after 5-6 hrs the server crashes and the error we see in the log is as below:

# A fatal error has been detected by the Java Runtime Environment:

#

# java.lang.OutOfMemoryError: requested 86532 bytes for Handle in /export0/BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jvmtiGetLoadedClasses.cpp

. Out of swap space?

#

#  Internal Error (allocation.inline.hpp:39), pid=26427, tid=83

#  Error: Handle in /export0/BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jvmtiGetLoadedClasses.cpp

#

# JRE version: 6.0_21-b06

# Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode solaris-x86 )


The memory arguments setup on the server is as below:
MEM_ARGS='-Xms2560m -Xmx2560m -XX:MaxPermSize=512m -XX:ParallelGCThreads=6 -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent -XX:+UseParNewGC -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=8 -XX:NewSize=512m -XX:MaxNewSize=512m -XX:+DisableExplicitGC -XX:CMSInitiatingOccupancyFraction=80'

We cannot increase the memory further as we are using a 32Bit JVM and the max allowed is 2560m. What changes can be made to fix this problem? I would appreciate your help.

Thanks
0
Comment
Question by:Dishsales
  • 4
  • 4
  • 2
10 Comments
 
LVL 19

Accepted Solution

by:
ramazanyich earned 2000 total points
ID: 38375005
it seems that you have issue reported on Sun JVM http://bugs.sun.com/view_bug.do?bug_id=6973402 and related http://bugs.sun.com/view_bug.do?bug_id=6916062
Bug seems to be fixed in the JDK1.6.0_22 according to the last url.
So try to upgrade to JDK1.6.0_22 or higher.

Also if it does not help try workaround mentioned in https://confluence.atlassian.com/pages/viewpage.action?pageId=219023686
0
 
LVL 12

Expert Comment

by:Sharon Seth
ID: 38375770
Can you attach the complete JVM dump pls
0
 

Author Comment

by:Dishsales
ID: 38378711
Thanks for the reply.

We have upgraded to JDK1.6.0_35. The server crashed again and we have got the below error message in the logs. We are using the 32 bit JVM and we cannot increase it to 64bit. The error message has given us a list of possible reasons. What might be the actual cause and the best solution?

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 4092 bytes for char in /export0/BUILD_AREA/jdk6_35/hotspot/src/share/vm/utilities/stack.inline.hpp
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.inline.hpp:44), pid=29405, tid=13
#
# JRE version: 6.0_35-b10
# Java VM: Java HotSpot(TM) Server VM (20.10-b01 mixed mode solaris-x86 )

---------------  T H R E A D  ---------------

Current thread (0x08310400):  VMThread [stack: 0x39e7d000,0x39efd000] [id=13]

Stack: [0x39e7d000,0x39efd000],  sp=0x39efc1d0,  free space=508k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x89d627]
0
NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

 
LVL 19

Expert Comment

by:ramazanyich
ID: 38379160
it seems that in your application you use work with big amount of data which does not fit in memory. Or somewhere is a memory leak (you don't close some resources, like files, db connections, etc). Do profiling of your application first to understand where memory goes
0
 

Author Comment

by:Dishsales
ID: 38387476
We did the profiling on our application. After completing 19 hrs of load run we started seeing huge number of stalled transactions which lead to server going down.

Usually when the server crashes because of running out of memory, a log file by the name hs_err_pid.log gets generated. This time round, no such log file got generated. We were curious to know if an issue on database led to stalled transactions and this crash or is there some setting which needs to be done on the Weblogic server.

I have attached a screen shot of our DB & Heap from our profiler. Appreciate any help.
DBScreenShot.jpg
Heap.jpg
0
 
LVL 12

Expert Comment

by:Sharon Seth
ID: 38389728
Couple of questions:

1. What sort of transactions are these - the stalled ones ?
2. What's the max heap size set to ? (XmX) . Decreasing the heap size gives more space for native memory
3. How many threads do you see in the profiler results ?

I had the same situation , but figured out it was due to spawning of threads each of which create a new initial context . The initial contexts were not being closed and so , the threads that opened them never exited , thus filling up the native space and stopping any more thread creations.
0
 

Author Comment

by:Dishsales
ID: 38391483
Please find the answers below:
1. We are doing some CRUD operations and they are DB stalled transactions. We find the issue on the DB side and there is no performance problem now. Again we did the load run yesterday and the server crashed as the heap space got filled up.

2. XmX has been set 2048m. We are using the 32bit jvm.

3. At the begining of the load run we see 76 active threads. It reaches 88 after 21 hr of load run. As the heap  started getting filled it went upto 160-180 before it crashed. I have attached the screen shot from the profiler.

Also what was the fix you made when you were facing this situation?
Heap.jpg
ActiveThreads.jpg
0
 
LVL 12

Expert Comment

by:Sharon Seth
ID: 38393831
Heap space still getting filled up could be a prblm from your app too , not just DB . Also the thread count is increasing , which clearly shows the threads are not exiting as they are  holding on to some resource .

Suspect list :

1. Check what the threads are holding onto .When you get the heap space error , are you able to access WL console ? If yes ,then you can take a thread dump from the console . This will show kind of threads these are.

2. Check if DB connections are being closed properly . If DB connections were the prblm , you should first have seen a max count reached on connection pool , which is not happening .But check once to rule this out.

My case , we could clearly pinpoint the prblm , that the initial contexts are not being closed . So , the threads were holding on the initial contexts . Fix was to simply close the IC .
0
 

Author Comment

by:Dishsales
ID: 38400586
When the server crashes, it used to generate the hs_err_pid.log file.

From the last couple of load runs we are not seeing this log file getting generated after the server crashes. Is there some arguments which needs to be set to get this log file generated or does this mean the server crashing because of some other problem?
0
 
LVL 12

Expert Comment

by:Sharon Seth
ID: 38401108
1. In the first case , at which location was the hs_err file getting generated.

2 .In the later case , although the file is not getting generated, where do you see the error msgs - in stdout ?
Normally it is created in the dir frm which the app has been started ,if this dir is not writable , it gets created in /tmp . If it's not possible to write in /tmp,then it's only written to stdout .

But starting with 1.5 , Sun added one JVM arg -XX:ErrorFile . Set this to an editable dir , and you shld start seeing the hs_err_file there.

4. What about the heap dump ?  Do you have a dump that shows the state of the process at the time of crash?

I would suggest that you create a new thread on experts exhcange to track this , since you have already accepted an answer
0

Featured Post

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Article by: kevp75
Hey folks, 'bout time for me to come around with a little tip. Thanks to IIS 7.5 Extensions and Microsoft (well... really Windows 8, and IIS 8 I guess...), we can now prime our Application Pools, when IIS starts. Now, though it would be nice t…
One of the typical problems I have experienced is when you have to move a web server from one hosting site to another. You normally prepare all on the new host, transfer the site, change DNS and cross your fingers hoping all will be ok on new server…
Integration Management Part 2
With just a little bit of  SQL and VBA, many doors open to cool things like synchronize a list box to display data relevant to other information on a form.  If you have never written code or looked at an SQL statement before, no problem! ...  give i…

807 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question