Solved

WebLogic Server Core Dump on Solaris

Posted on 2006-07-06
17
5,318 Views
Last Modified: 2013-12-10
We have a web application deployed on WebLogic Server8.1 on 2 Solaris machine. The 1st Solaris machine contains the Admin server and the 1st Managed server. The 2nd Solaris machine contains the 2nd Managed server.

Every few hours, it seems that the WebLogic Server will crash, generating a core dump, and an error file named hs_err_pidXXXXX.log, with contents shown below. Our client thinks that it is memory problem. We have already set the min/max memory to the JVM to be 1GB. Our client thinks that it could be due to our application (allocating objects and not garbage collected).

Can anyone interprets the following file and tell us what the problem could be? Thanks!

--- Contents of hs_err_pid18110.log ---

Unexpected Signal : 11 occurred at PC=0xFEDCBCAC
Function=[Unknown. Nearest: JVM_FillInStackTrace+0x4CE4]
Library=/app/weblogic/jdk142_05/jre/lib/sparc/server/libjvm.so

Current Java thread:

Dynamic libraries:
0x10000       /app/weblogic/jdk142_05/bin/java
0xff370000       /usr/lib/libthread.so.1
0xff3fa000       /usr/lib/libdl.so.1
0xff280000       /usr/lib/libc.so.1
0xff3a0000       /usr/platform/SUNW,Sun-Fire-V440/lib/libc_psr.so.1
0xfec00000       /app/weblogic/jdk142_05/jre/lib/sparc/server/libjvm.so
0xff240000       /usr/lib/libCrun.so.1
0xff220000       /usr/lib/libsocket.so.1
0xfeb00000       /usr/lib/libnsl.so.1
0xfeab0000       /usr/lib/libm.so.1
0xff200000       /usr/lib/libsched.so.1
0xfebe0000       /usr/lib/libmp.so.2
0xfea80000       /app/weblogic/jdk142_05/jre/lib/sparc/native_threads/libhpi.so
0xfea60000       /usr/lib/nss_files.so.1
0xfea30000       /app/weblogic/jdk142_05/jre/lib/sparc/libverify.so
0xfe9f0000       /app/weblogic/jdk142_05/jre/lib/sparc/libjava.so
0xfe9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libzip.so
0xb07d0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnet.so
0xb07b0000       /app/weblogic/server/lib/solaris/libweblogicunix1.so
0xb0660000       /app/weblogic/server/lib/solaris/libstackdump.so
0xb01e0000       /app/weblogic/server/lib/solaris/libmuxer.so
0xb01c0000       /usr/ucblib/libucb.so.1
0xafda0000       /usr/lib/libresolv.so.2
0xb0190000       /usr/lib/libelf.so.1
0xaffe0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnio.so
0xadae0000       /usr/lib/librt.so.1
0xadac0000       /usr/lib/libaio.so.1
0xada90000       /usr/lib/libmd5.so.1
0xad9e0000       /usr/lib/libsendfile.so.1
0xad9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libioser12.so

Heap at VM Abort:
Heap
 def new generation   total 169728K, used 56817K [0xb5800000, 0xc02c0000, 0xcad50000)
  eden space 164608K,  31% used [0xb5800000, 0xb8a7c750, 0xbf8c0000)
  from space 5120K, 100% used [0xbfdc0000, 0xc02c0000, 0xc02c0000)
  to   space 5120K,   0% used [0xbf8c0000, 0xbf8c0000, 0xbfdc0000)
 tenured generation   total 349568K, used 289884K [0xcad50000, 0xe02b0000, 0xf5800000)
   the space 349568K,  82% used [0xcad50000, 0xdc8672d8, 0xdc867400, 0xe02b0000)
 compacting perm gen  total 37888K, used 37369K [0xf5800000, 0xf7d00000, 0xf9800000)
   the space 37888K,  98% used [0xf5800000, 0xf7c7e7b8, 0xf7c7e800, 0xf7d00000)

Local Time = Fri Jul  7 01:01:55 2006
Elapsed Time = 27753
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode)
#
0
Comment
Question by:yongsing
  • 9
  • 6
17 Comments
 
LVL 35

Expert Comment

by:girionis
Comment Utility
Hi yongsing

Have a llok here: http://forum.java.sun.com/thread.jspa?threadID=307252&start=60 some say it's hardware problem some say it's a problem with JDK. A suggested solution is to get rid of -server switch. Maybe if you try JRockit you will have better results.

Cheers
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
Thanks, girionis. That is quite a useful link.

I do not think that the application itself can cause the JVM to crash, as what our client thinks. I mean, if you run out of memory to create new objects, the JVM would have thrown OutOfMemoryError, right?

We'll try the suggested solutions to see if they solve our problem.
0
 
LVL 35

Expert Comment

by:girionis
Comment Utility
> the JVM would have thrown OutOfMemoryError, right?

Yes this is true, unless the JVM implementation is buggy.
0
 
LVL 3

Expert Comment

by:anumalas
Comment Utility
Hi

Try to increase the jvm heap size

for eg: java -Xms64m -Xmx512m


you can set this heap size in startWeblogic.sh for JAVA_OPTIONS environmental variable.

Hope this will help :)
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
>> Try to increase the jvm heap size

We have already increased to 512, then 1G, but still getting the crash.
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
Does JRocket comes with WebLogic Server 8.1? Where is it?
0
 
LVL 35

Expert Comment

by:girionis
Comment Utility
It does. It should be on the root folder of the BEA installation. On my computer it is in the D:\bea\jrockit81sp4_142_05
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
It doesn't seem to come with the BEA installation on our Solaris machine. I have it on my own computer though.
0
What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

 
LVL 9

Author Comment

by:yongsing
Comment Utility
The -server option seems to be there when you set PRODUCTION_MODE to true. I think if we want to remove this -server option, PRODUCTION_MODE would have to be set to false. But I was told that in production, PRODUCTION_MODE should be set to true.
0
 
LVL 35

Expert Comment

by:girionis
Comment Utility
Well, tbh there is little difference between production and development mode. The server just optimizes a few things. Just give it a go and see if the problem persists. If it does not then we know ehere the problem is and we can start working from there.
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
Thanks, we'll be trying that out tomorrow.
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
So far I've used the -Xmx and -Xms options. But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our client said that it must be set to true. All their production servers are set to true.
0
 
LVL 35

Accepted Solution

by:
girionis earned 100 total points
Comment Utility
> But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?

This is used for dynamic class generation purposes. Some applications need to dynamically generate and load classes, the MaxPermSize is used to set this permanent generation size.

Have a look here:

http://java.sun.com/docs/hotspot/gc/
http://forum.java.sun.com/thread.jspa?tstart=0&forumID=27&threadID=343789&trange=15

>We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our
>client said that it must be set to true. All their production servers are set to true.

Then I think we might have found the problem. If you can't set the PRODUCTION_MODE to false then you will need to find a workaround.  Does the crash happen to all servers? If not then try to find out the specifications the servers that do not crash and see if you can have the same to those who crash. Otherwise try to upgrade/downgrade/change the VM (for example jrockit instead of Sun)/OS/ and see if the problem persists.
0
 
LVL 9

Author Comment

by:yongsing
Comment Utility
Hi girionis,

(1) Is it -XX:MaxPermSize or -XmaxPermSize? I have encountered both on web sites I searched, but the former seems to be the correct one.

(2) Is there a MinPermSize as well?

(3) Do I set this to the Admin as well as the managed servers?
0
 
LVL 35

Expert Comment

by:girionis
Comment Utility
1) I think is the latter (-XMaxPermSize) but I am not 100% sure.

2) Not sure.

3) YOu need to set it to all server instances that will generate classes, so yes, both managed servers and admin (although I am not sure if admin needs it since it won't be running any applications)
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now