Weblogic - Out of Memory Error

Posted on 2012-09-06
Last Modified: 2012-09-14

We have our application deployed on Weblogic When we do a load run on this server, after 5-6 hrs the server crashes and the error we see in the log is as below:

# A fatal error has been detected by the Java Runtime Environment:


# java.lang.OutOfMemoryError: requested 86532 bytes for Handle in /export0/BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jvmtiGetLoadedClasses.cpp

. Out of swap space?


#  Internal Error (allocation.inline.hpp:39), pid=26427, tid=83

#  Error: Handle in /export0/BUILD_AREA/jdk6_21/hotspot/src/share/vm/prims/jvmtiGetLoadedClasses.cpp


# JRE version: 6.0_21-b06

# Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode solaris-x86 )

The memory arguments setup on the server is as below:
MEM_ARGS='-Xms2560m -Xmx2560m -XX:MaxPermSize=512m -XX:ParallelGCThreads=6 -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent -XX:+UseParNewGC -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=4 -XX:SurvivorRatio=8 -XX:NewSize=512m -XX:MaxNewSize=512m -XX:+DisableExplicitGC -XX:CMSInitiatingOccupancyFraction=80'

We cannot increase the memory further as we are using a 32Bit JVM and the max allowed is 2560m. What changes can be made to fix this problem? I would appreciate your help.

Question by:Dishsales
    LVL 19

    Accepted Solution

    it seems that you have issue reported on Sun JVM and related
    Bug seems to be fixed in the JDK1.6.0_22 according to the last url.
    So try to upgrade to JDK1.6.0_22 or higher.

    Also if it does not help try workaround mentioned in
    LVL 12

    Expert Comment

    by:Sharon Seth
    Can you attach the complete JVM dump pls

    Author Comment

    Thanks for the reply.

    We have upgraded to JDK1.6.0_35. The server crashed again and we have got the below error message in the logs. We are using the 32 bit JVM and we cannot increase it to 64bit. The error message has given us a list of possible reasons. What might be the actual cause and the best solution?

    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (malloc) failed to allocate 4092 bytes for char in /export0/BUILD_AREA/jdk6_35/hotspot/src/share/vm/utilities/stack.inline.hpp
    # Possible reasons:
    #   The system is out of physical RAM or swap space
    #   In 32 bit mode, the process size limit was hit
    # Possible solutions:
    #   Reduce memory load on the system
    #   Increase physical memory or swap space
    #   Check if swap backing store is full
    #   Use 64 bit Java on a 64 bit OS
    #   Decrease Java heap size (-Xmx/-Xms)
    #   Decrease number of Java threads
    #   Decrease Java thread stack sizes (-Xss)
    #   Set larger code cache with -XX:ReservedCodeCacheSize=
    # This output file may be truncated or incomplete.
    #  Out of Memory Error (allocation.inline.hpp:44), pid=29405, tid=13
    # JRE version: 6.0_35-b10
    # Java VM: Java HotSpot(TM) Server VM (20.10-b01 mixed mode solaris-x86 )

    ---------------  T H R E A D  ---------------

    Current thread (0x08310400):  VMThread [stack: 0x39e7d000,0x39efd000] [id=13]

    Stack: [0x39e7d000,0x39efd000],  sp=0x39efc1d0,  free space=508k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V  []
    LVL 19

    Expert Comment

    it seems that in your application you use work with big amount of data which does not fit in memory. Or somewhere is a memory leak (you don't close some resources, like files, db connections, etc). Do profiling of your application first to understand where memory goes

    Author Comment

    We did the profiling on our application. After completing 19 hrs of load run we started seeing huge number of stalled transactions which lead to server going down.

    Usually when the server crashes because of running out of memory, a log file by the name hs_err_pid.log gets generated. This time round, no such log file got generated. We were curious to know if an issue on database led to stalled transactions and this crash or is there some setting which needs to be done on the Weblogic server.

    I have attached a screen shot of our DB & Heap from our profiler. Appreciate any help.
    LVL 12

    Expert Comment

    by:Sharon Seth
    Couple of questions:

    1. What sort of transactions are these - the stalled ones ?
    2. What's the max heap size set to ? (XmX) . Decreasing the heap size gives more space for native memory
    3. How many threads do you see in the profiler results ?

    I had the same situation , but figured out it was due to spawning of threads each of which create a new initial context . The initial contexts were not being closed and so , the threads that opened them never exited , thus filling up the native space and stopping any more thread creations.

    Author Comment

    Please find the answers below:
    1. We are doing some CRUD operations and they are DB stalled transactions. We find the issue on the DB side and there is no performance problem now. Again we did the load run yesterday and the server crashed as the heap space got filled up.

    2. XmX has been set 2048m. We are using the 32bit jvm.

    3. At the begining of the load run we see 76 active threads. It reaches 88 after 21 hr of load run. As the heap  started getting filled it went upto 160-180 before it crashed. I have attached the screen shot from the profiler.

    Also what was the fix you made when you were facing this situation?
    LVL 12

    Expert Comment

    by:Sharon Seth
    Heap space still getting filled up could be a prblm from your app too , not just DB . Also the thread count is increasing , which clearly shows the threads are not exiting as they are  holding on to some resource .

    Suspect list :

    1. Check what the threads are holding onto .When you get the heap space error , are you able to access WL console ? If yes ,then you can take a thread dump from the console . This will show kind of threads these are.

    2. Check if DB connections are being closed properly . If DB connections were the prblm , you should first have seen a max count reached on connection pool , which is not happening .But check once to rule this out.

    My case , we could clearly pinpoint the prblm , that the initial contexts are not being closed . So , the threads were holding on the initial contexts . Fix was to simply close the IC .

    Author Comment

    When the server crashes, it used to generate the hs_err_pid.log file.

    From the last couple of load runs we are not seeing this log file getting generated after the server crashes. Is there some arguments which needs to be set to get this log file generated or does this mean the server crashing because of some other problem?
    LVL 12

    Expert Comment

    by:Sharon Seth
    1. In the first case , at which location was the hs_err file getting generated.

    2 .In the later case , although the file is not getting generated, where do you see the error msgs - in stdout ?
    Normally it is created in the dir frm which the app has been started ,if this dir is not writable , it gets created in /tmp . If it's not possible to write in /tmp,then it's only written to stdout .

    But starting with 1.5 , Sun added one JVM arg -XX:ErrorFile . Set this to an editable dir , and you shld start seeing the hs_err_file there.

    4. What about the heap dump ?  Do you have a dump that shows the state of the process at the time of crash?

    I would suggest that you create a new thread on experts exhcange to track this , since you have already accepted an answer

    Featured Post

    Free Trending Threat Insights Every Day

    Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

    Join & Write a Comment

    Suggested Solutions

    In our day to day coding, how many times have we come across a necessity to check whether a URL is a broken link or not? For those of you that answered countless and are using ColdFusion like myself, then this article is for you.  It will show yo…
    Periodically we have to update or add SSL certificates for customers. Depending upon your hosting plan you may be responsible for the installation and/or key generation. In the wake of Heartbleed many sites were forced to re-key. We will concen…
    Sending a Secure fax is easy with eFax Corporate ( First, Just open a new email message.  In the To field, type your recipient's fax number You can even send a secure international fax — just include t…
    Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

    729 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    23 Experts available now in Live!

    Get 1:1 Help Now