WebLogic Server Core Dump on Solaris

We have a web application deployed on WebLogic Server8.1 on 2 Solaris machine. The 1st Solaris machine contains the Admin server and the 1st Managed server. The 2nd Solaris machine contains the 2nd Managed server.

Every few hours, it seems that the WebLogic Server will crash, generating a core dump, and an error file named hs_err_pidXXXXX.log, with contents shown below. Our client thinks that it is memory problem. We have already set the min/max memory to the JVM to be 1GB. Our client thinks that it could be due to our application (allocating objects and not garbage collected).

Can anyone interprets the following file and tell us what the problem could be? Thanks!

--- Contents of hs_err_pid18110.log ---

Unexpected Signal : 11 occurred at PC=0xFEDCBCAC
Function=[Unknown. Nearest: JVM_FillInStackTrace+0x4CE4]

Current Java thread:

Dynamic libraries:
0x10000       /app/weblogic/jdk142_05/bin/java
0xff370000       /usr/lib/libthread.so.1
0xff3fa000       /usr/lib/libdl.so.1
0xff280000       /usr/lib/libc.so.1
0xff3a0000       /usr/platform/SUNW,Sun-Fire-V440/lib/libc_psr.so.1
0xfec00000       /app/weblogic/jdk142_05/jre/lib/sparc/server/libjvm.so
0xff240000       /usr/lib/libCrun.so.1
0xff220000       /usr/lib/libsocket.so.1
0xfeb00000       /usr/lib/libnsl.so.1
0xfeab0000       /usr/lib/libm.so.1
0xff200000       /usr/lib/libsched.so.1
0xfebe0000       /usr/lib/libmp.so.2
0xfea80000       /app/weblogic/jdk142_05/jre/lib/sparc/native_threads/libhpi.so
0xfea60000       /usr/lib/nss_files.so.1
0xfea30000       /app/weblogic/jdk142_05/jre/lib/sparc/libverify.so
0xfe9f0000       /app/weblogic/jdk142_05/jre/lib/sparc/libjava.so
0xfe9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libzip.so
0xb07d0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnet.so
0xb07b0000       /app/weblogic/server/lib/solaris/libweblogicunix1.so
0xb0660000       /app/weblogic/server/lib/solaris/libstackdump.so
0xb01e0000       /app/weblogic/server/lib/solaris/libmuxer.so
0xb01c0000       /usr/ucblib/libucb.so.1
0xafda0000       /usr/lib/libresolv.so.2
0xb0190000       /usr/lib/libelf.so.1
0xaffe0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnio.so
0xadae0000       /usr/lib/librt.so.1
0xadac0000       /usr/lib/libaio.so.1
0xada90000       /usr/lib/libmd5.so.1
0xad9e0000       /usr/lib/libsendfile.so.1
0xad9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libioser12.so

Heap at VM Abort:
 def new generation   total 169728K, used 56817K [0xb5800000, 0xc02c0000, 0xcad50000)
  eden space 164608K,  31% used [0xb5800000, 0xb8a7c750, 0xbf8c0000)
  from space 5120K, 100% used [0xbfdc0000, 0xc02c0000, 0xc02c0000)
  to   space 5120K,   0% used [0xbf8c0000, 0xbf8c0000, 0xbfdc0000)
 tenured generation   total 349568K, used 289884K [0xcad50000, 0xe02b0000, 0xf5800000)
   the space 349568K,  82% used [0xcad50000, 0xdc8672d8, 0xdc867400, 0xe02b0000)
 compacting perm gen  total 37888K, used 37369K [0xf5800000, 0xf7d00000, 0xf9800000)
   the space 37888K,  98% used [0xf5800000, 0xf7c7e7b8, 0xf7c7e800, 0xf7d00000)

Local Time = Fri Jul  7 01:01:55 2006
Elapsed Time = 27753
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
# Java VM: Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode)
Who is Participating?
girionisConnect With a Mentor Commented:
> But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?

This is used for dynamic class generation purposes. Some applications need to dynamically generate and load classes, the MaxPermSize is used to set this permanent generation size.

Have a look here:


>We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our
>client said that it must be set to true. All their production servers are set to true.

Then I think we might have found the problem. If you can't set the PRODUCTION_MODE to false then you will need to find a workaround.  Does the crash happen to all servers? If not then try to find out the specifications the servers that do not crash and see if you can have the same to those who crash. Otherwise try to upgrade/downgrade/change the VM (for example jrockit instead of Sun)/OS/ and see if the problem persists.
Hi yongsing

Have a llok here: http://forum.java.sun.com/thread.jspa?threadID=307252&start=60 some say it's hardware problem some say it's a problem with JDK. A suggested solution is to get rid of -server switch. Maybe if you try JRockit you will have better results.

yongsingAuthor Commented:
Thanks, girionis. That is quite a useful link.

I do not think that the application itself can cause the JVM to crash, as what our client thinks. I mean, if you run out of memory to create new objects, the JVM would have thrown OutOfMemoryError, right?

We'll try the suggested solutions to see if they solve our problem.
Get your problem seen by more experts

Be seen. Boost your question’s priority for more expert views and faster solutions

> the JVM would have thrown OutOfMemoryError, right?

Yes this is true, unless the JVM implementation is buggy.

Try to increase the jvm heap size

for eg: java -Xms64m -Xmx512m

you can set this heap size in startWeblogic.sh for JAVA_OPTIONS environmental variable.

Hope this will help :)
yongsingAuthor Commented:
>> Try to increase the jvm heap size

We have already increased to 512, then 1G, but still getting the crash.
yongsingAuthor Commented:
Does JRocket comes with WebLogic Server 8.1? Where is it?
It does. It should be on the root folder of the BEA installation. On my computer it is in the D:\bea\jrockit81sp4_142_05
yongsingAuthor Commented:
It doesn't seem to come with the BEA installation on our Solaris machine. I have it on my own computer though.
yongsingAuthor Commented:
The -server option seems to be there when you set PRODUCTION_MODE to true. I think if we want to remove this -server option, PRODUCTION_MODE would have to be set to false. But I was told that in production, PRODUCTION_MODE should be set to true.
Well, tbh there is little difference between production and development mode. The server just optimizes a few things. Just give it a go and see if the problem persists. If it does not then we know ehere the problem is and we can start working from there.
yongsingAuthor Commented:
Thanks, we'll be trying that out tomorrow.
yongsingAuthor Commented:
So far I've used the -Xmx and -Xms options. But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?
yongsingAuthor Commented:
We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our client said that it must be set to true. All their production servers are set to true.
yongsingAuthor Commented:
Hi girionis,

(1) Is it -XX:MaxPermSize or -XmaxPermSize? I have encountered both on web sites I searched, but the former seems to be the correct one.

(2) Is there a MinPermSize as well?

(3) Do I set this to the Admin as well as the managed servers?
1) I think is the latter (-XMaxPermSize) but I am not 100% sure.

2) Not sure.

3) YOu need to set it to all server instances that will generate classes, so yes, both managed servers and admin (although I am not sure if admin needs it since it won't be running any applications)
All Courses

From novice to tech pro — start learning today.