yongsing
asked on
WebLogic Server Core Dump on Solaris
We have a web application deployed on WebLogic Server8.1 on 2 Solaris machine. The 1st Solaris machine contains the Admin server and the 1st Managed server. The 2nd Solaris machine contains the 2nd Managed server.
Every few hours, it seems that the WebLogic Server will crash, generating a core dump, and an error file named hs_err_pidXXXXX.log, with contents shown below. Our client thinks that it is memory problem. We have already set the min/max memory to the JVM to be 1GB. Our client thinks that it could be due to our application (allocating objects and not garbage collected).
Can anyone interprets the following file and tell us what the problem could be? Thanks!
--- Contents of hs_err_pid18110.log ---
Unexpected Signal : 11 occurred at PC=0xFEDCBCAC
Function=[Unknown. Nearest: JVM_FillInStackTrace+0x4CE
Library=/app/weblogic/jdk1
Current Java thread:
Dynamic libraries:
0x10000 /app/weblogic/jdk142_05/bi
0xff370000 /usr/lib/libthread.so.1
0xff3fa000 /usr/lib/libdl.so.1
0xff280000 /usr/lib/libc.so.1
0xff3a0000 /usr/platform/SUNW,Sun-Fir
0xfec00000 /app/weblogic/jdk142_05/jr
0xff240000 /usr/lib/libCrun.so.1
0xff220000 /usr/lib/libsocket.so.1
0xfeb00000 /usr/lib/libnsl.so.1
0xfeab0000 /usr/lib/libm.so.1
0xff200000 /usr/lib/libsched.so.1
0xfebe0000 /usr/lib/libmp.so.2
0xfea80000 /app/weblogic/jdk142_05/jr
0xfea60000 /usr/lib/nss_files.so.1
0xfea30000 /app/weblogic/jdk142_05/jr
0xfe9f0000 /app/weblogic/jdk142_05/jr
0xfe9c0000 /app/weblogic/jdk142_05/jr
0xb07d0000 /app/weblogic/jdk142_05/jr
0xb07b0000 /app/weblogic/server/lib/s
0xb0660000 /app/weblogic/server/lib/s
0xb01e0000 /app/weblogic/server/lib/s
0xb01c0000 /usr/ucblib/libucb.so.1
0xafda0000 /usr/lib/libresolv.so.2
0xb0190000 /usr/lib/libelf.so.1
0xaffe0000 /app/weblogic/jdk142_05/jr
0xadae0000 /usr/lib/librt.so.1
0xadac0000 /usr/lib/libaio.so.1
0xada90000 /usr/lib/libmd5.so.1
0xad9e0000 /usr/lib/libsendfile.so.1
0xad9c0000 /app/weblogic/jdk142_05/jr
Heap at VM Abort:
Heap
def new generation total 169728K, used 56817K [0xb5800000, 0xc02c0000, 0xcad50000)
eden space 164608K, 31% used [0xb5800000, 0xb8a7c750, 0xbf8c0000)
from space 5120K, 100% used [0xbfdc0000, 0xc02c0000, 0xc02c0000)
to space 5120K, 0% used [0xbf8c0000, 0xbf8c0000, 0xbfdc0000)
tenured generation total 349568K, used 289884K [0xcad50000, 0xe02b0000, 0xf5800000)
the space 349568K, 82% used [0xcad50000, 0xdc8672d8, 0xdc867400, 0xe02b0000)
compacting perm gen total 37888K, used 37369K [0xf5800000, 0xf7d00000, 0xf9800000)
the space 37888K, 98% used [0xf5800000, 0xf7c7e7b8, 0xf7c7e800, 0xf7d00000)
Local Time = Fri Jul 7 01:01:55 2006
Elapsed Time = 27753
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode)
#
Every few hours, it seems that the WebLogic Server will crash, generating a core dump, and an error file named hs_err_pidXXXXX.log, with contents shown below. Our client thinks that it is memory problem. We have already set the min/max memory to the JVM to be 1GB. Our client thinks that it could be due to our application (allocating objects and not garbage collected).
Can anyone interprets the following file and tell us what the problem could be? Thanks!
--- Contents of hs_err_pid18110.log ---
Unexpected Signal : 11 occurred at PC=0xFEDCBCAC
Function=[Unknown. Nearest: JVM_FillInStackTrace+0x4CE
Library=/app/weblogic/jdk1
Current Java thread:
Dynamic libraries:
0x10000 /app/weblogic/jdk142_05/bi
0xff370000 /usr/lib/libthread.so.1
0xff3fa000 /usr/lib/libdl.so.1
0xff280000 /usr/lib/libc.so.1
0xff3a0000 /usr/platform/SUNW,Sun-Fir
0xfec00000 /app/weblogic/jdk142_05/jr
0xff240000 /usr/lib/libCrun.so.1
0xff220000 /usr/lib/libsocket.so.1
0xfeb00000 /usr/lib/libnsl.so.1
0xfeab0000 /usr/lib/libm.so.1
0xff200000 /usr/lib/libsched.so.1
0xfebe0000 /usr/lib/libmp.so.2
0xfea80000 /app/weblogic/jdk142_05/jr
0xfea60000 /usr/lib/nss_files.so.1
0xfea30000 /app/weblogic/jdk142_05/jr
0xfe9f0000 /app/weblogic/jdk142_05/jr
0xfe9c0000 /app/weblogic/jdk142_05/jr
0xb07d0000 /app/weblogic/jdk142_05/jr
0xb07b0000 /app/weblogic/server/lib/s
0xb0660000 /app/weblogic/server/lib/s
0xb01e0000 /app/weblogic/server/lib/s
0xb01c0000 /usr/ucblib/libucb.so.1
0xafda0000 /usr/lib/libresolv.so.2
0xb0190000 /usr/lib/libelf.so.1
0xaffe0000 /app/weblogic/jdk142_05/jr
0xadae0000 /usr/lib/librt.so.1
0xadac0000 /usr/lib/libaio.so.1
0xada90000 /usr/lib/libmd5.so.1
0xad9e0000 /usr/lib/libsendfile.so.1
0xad9c0000 /app/weblogic/jdk142_05/jr
Heap at VM Abort:
Heap
def new generation total 169728K, used 56817K [0xb5800000, 0xc02c0000, 0xcad50000)
eden space 164608K, 31% used [0xb5800000, 0xb8a7c750, 0xbf8c0000)
from space 5120K, 100% used [0xbfdc0000, 0xc02c0000, 0xc02c0000)
to space 5120K, 0% used [0xbf8c0000, 0xbf8c0000, 0xbfdc0000)
tenured generation total 349568K, used 289884K [0xcad50000, 0xe02b0000, 0xf5800000)
the space 349568K, 82% used [0xcad50000, 0xdc8672d8, 0xdc867400, 0xe02b0000)
compacting perm gen total 37888K, used 37369K [0xf5800000, 0xf7d00000, 0xf9800000)
the space 37888K, 98% used [0xf5800000, 0xf7c7e7b8, 0xf7c7e800, 0xf7d00000)
Local Time = Fri Jul 7 01:01:55 2006
Elapsed Time = 27753
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode)
#
ASKER
Thanks, girionis. That is quite a useful link.
I do not think that the application itself can cause the JVM to crash, as what our client thinks. I mean, if you run out of memory to create new objects, the JVM would have thrown OutOfMemoryError, right?
We'll try the suggested solutions to see if they solve our problem.
I do not think that the application itself can cause the JVM to crash, as what our client thinks. I mean, if you run out of memory to create new objects, the JVM would have thrown OutOfMemoryError, right?
We'll try the suggested solutions to see if they solve our problem.
> the JVM would have thrown OutOfMemoryError, right?
Yes this is true, unless the JVM implementation is buggy.
Yes this is true, unless the JVM implementation is buggy.
Hi
Try to increase the jvm heap size
for eg: java -Xms64m -Xmx512m
you can set this heap size in startWeblogic.sh for JAVA_OPTIONS environmental variable.
Hope this will help :)
Try to increase the jvm heap size
for eg: java -Xms64m -Xmx512m
you can set this heap size in startWeblogic.sh for JAVA_OPTIONS environmental variable.
Hope this will help :)
ASKER
>> Try to increase the jvm heap size
We have already increased to 512, then 1G, but still getting the crash.
We have already increased to 512, then 1G, but still getting the crash.
ASKER
Does JRocket comes with WebLogic Server 8.1? Where is it?
It does. It should be on the root folder of the BEA installation. On my computer it is in the D:\bea\jrockit81sp4_142_05
ASKER
It doesn't seem to come with the BEA installation on our Solaris machine. I have it on my own computer though.
ASKER
The -server option seems to be there when you set PRODUCTION_MODE to true. I think if we want to remove this -server option, PRODUCTION_MODE would have to be set to false. But I was told that in production, PRODUCTION_MODE should be set to true.
Well, tbh there is little difference between production and development mode. The server just optimizes a few things. Just give it a go and see if the problem persists. If it does not then we know ehere the problem is and we can start working from there.
ASKER
Thanks, we'll be trying that out tomorrow.
ASKER
So far I've used the -Xmx and -Xms options. But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?
ASKER
We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our client said that it must be set to true. All their production servers are set to true.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Hi girionis,
(1) Is it -XX:MaxPermSize or -XmaxPermSize? I have encountered both on web sites I searched, but the former seems to be the correct one.
(2) Is there a MinPermSize as well?
(3) Do I set this to the Admin as well as the managed servers?
(1) Is it -XX:MaxPermSize or -XmaxPermSize? I have encountered both on web sites I searched, but the former seems to be the correct one.
(2) Is there a MinPermSize as well?
(3) Do I set this to the Admin as well as the managed servers?
1) I think is the latter (-XMaxPermSize) but I am not 100% sure.
2) Not sure.
3) YOu need to set it to all server instances that will generate classes, so yes, both managed servers and admin (although I am not sure if admin needs it since it won't be running any applications)
2) Not sure.
3) YOu need to set it to all server instances that will generate classes, so yes, both managed servers and admin (although I am not sure if admin needs it since it won't be running any applications)
Have a llok here: http://forum.java.sun.com/thread.jspa?threadID=307252&start=60 some say it's hardware problem some say it's a problem with JDK. A suggested solution is to get rid of -server switch. Maybe if you try JRockit you will have better results.
Cheers