?
Solved

WebLogic Server Core Dump on Solaris

Posted on 2006-07-06
17
Medium Priority
?
5,458 Views
Last Modified: 2013-12-10
We have a web application deployed on WebLogic Server8.1 on 2 Solaris machine. The 1st Solaris machine contains the Admin server and the 1st Managed server. The 2nd Solaris machine contains the 2nd Managed server.

Every few hours, it seems that the WebLogic Server will crash, generating a core dump, and an error file named hs_err_pidXXXXX.log, with contents shown below. Our client thinks that it is memory problem. We have already set the min/max memory to the JVM to be 1GB. Our client thinks that it could be due to our application (allocating objects and not garbage collected).

Can anyone interprets the following file and tell us what the problem could be? Thanks!

--- Contents of hs_err_pid18110.log ---

Unexpected Signal : 11 occurred at PC=0xFEDCBCAC
Function=[Unknown. Nearest: JVM_FillInStackTrace+0x4CE4]
Library=/app/weblogic/jdk142_05/jre/lib/sparc/server/libjvm.so

Current Java thread:

Dynamic libraries:
0x10000       /app/weblogic/jdk142_05/bin/java
0xff370000       /usr/lib/libthread.so.1
0xff3fa000       /usr/lib/libdl.so.1
0xff280000       /usr/lib/libc.so.1
0xff3a0000       /usr/platform/SUNW,Sun-Fire-V440/lib/libc_psr.so.1
0xfec00000       /app/weblogic/jdk142_05/jre/lib/sparc/server/libjvm.so
0xff240000       /usr/lib/libCrun.so.1
0xff220000       /usr/lib/libsocket.so.1
0xfeb00000       /usr/lib/libnsl.so.1
0xfeab0000       /usr/lib/libm.so.1
0xff200000       /usr/lib/libsched.so.1
0xfebe0000       /usr/lib/libmp.so.2
0xfea80000       /app/weblogic/jdk142_05/jre/lib/sparc/native_threads/libhpi.so
0xfea60000       /usr/lib/nss_files.so.1
0xfea30000       /app/weblogic/jdk142_05/jre/lib/sparc/libverify.so
0xfe9f0000       /app/weblogic/jdk142_05/jre/lib/sparc/libjava.so
0xfe9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libzip.so
0xb07d0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnet.so
0xb07b0000       /app/weblogic/server/lib/solaris/libweblogicunix1.so
0xb0660000       /app/weblogic/server/lib/solaris/libstackdump.so
0xb01e0000       /app/weblogic/server/lib/solaris/libmuxer.so
0xb01c0000       /usr/ucblib/libucb.so.1
0xafda0000       /usr/lib/libresolv.so.2
0xb0190000       /usr/lib/libelf.so.1
0xaffe0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnio.so
0xadae0000       /usr/lib/librt.so.1
0xadac0000       /usr/lib/libaio.so.1
0xada90000       /usr/lib/libmd5.so.1
0xad9e0000       /usr/lib/libsendfile.so.1
0xad9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libioser12.so

Heap at VM Abort:
Heap
 def new generation   total 169728K, used 56817K [0xb5800000, 0xc02c0000, 0xcad50000)
  eden space 164608K,  31% used [0xb5800000, 0xb8a7c750, 0xbf8c0000)
  from space 5120K, 100% used [0xbfdc0000, 0xc02c0000, 0xc02c0000)
  to   space 5120K,   0% used [0xbf8c0000, 0xbf8c0000, 0xbfdc0000)
 tenured generation   total 349568K, used 289884K [0xcad50000, 0xe02b0000, 0xf5800000)
   the space 349568K,  82% used [0xcad50000, 0xdc8672d8, 0xdc867400, 0xe02b0000)
 compacting perm gen  total 37888K, used 37369K [0xf5800000, 0xf7d00000, 0xf9800000)
   the space 37888K,  98% used [0xf5800000, 0xf7c7e7b8, 0xf7c7e800, 0xf7d00000)

Local Time = Fri Jul  7 01:01:55 2006
Elapsed Time = 27753
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode)
#
0
Comment
Question by:yongsing
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 6
17 Comments
 
LVL 35

Expert Comment

by:girionis
ID: 17056941
Hi yongsing

Have a llok here: http://forum.java.sun.com/thread.jspa?threadID=307252&start=60 some say it's hardware problem some say it's a problem with JDK. A suggested solution is to get rid of -server switch. Maybe if you try JRockit you will have better results.

Cheers
0
 
LVL 9

Author Comment

by:yongsing
ID: 17057152
Thanks, girionis. That is quite a useful link.

I do not think that the application itself can cause the JVM to crash, as what our client thinks. I mean, if you run out of memory to create new objects, the JVM would have thrown OutOfMemoryError, right?

We'll try the suggested solutions to see if they solve our problem.
0
 
LVL 35

Expert Comment

by:girionis
ID: 17057254
> the JVM would have thrown OutOfMemoryError, right?

Yes this is true, unless the JVM implementation is buggy.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
LVL 3

Expert Comment

by:anumalas
ID: 17057667
Hi

Try to increase the jvm heap size

for eg: java -Xms64m -Xmx512m


you can set this heap size in startWeblogic.sh for JAVA_OPTIONS environmental variable.

Hope this will help :)
0
 
LVL 9

Author Comment

by:yongsing
ID: 17067135
>> Try to increase the jvm heap size

We have already increased to 512, then 1G, but still getting the crash.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17070209
Does JRocket comes with WebLogic Server 8.1? Where is it?
0
 
LVL 35

Expert Comment

by:girionis
ID: 17070864
It does. It should be on the root folder of the BEA installation. On my computer it is in the D:\bea\jrockit81sp4_142_05
0
 
LVL 9

Author Comment

by:yongsing
ID: 17071546
It doesn't seem to come with the BEA installation on our Solaris machine. I have it on my own computer though.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17071552
The -server option seems to be there when you set PRODUCTION_MODE to true. I think if we want to remove this -server option, PRODUCTION_MODE would have to be set to false. But I was told that in production, PRODUCTION_MODE should be set to true.
0
 
LVL 35

Expert Comment

by:girionis
ID: 17071573
Well, tbh there is little difference between production and development mode. The server just optimizes a few things. Just give it a go and see if the problem persists. If it does not then we know ehere the problem is and we can start working from there.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17071816
Thanks, we'll be trying that out tomorrow.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17113288
So far I've used the -Xmx and -Xms options. But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?
0
 
LVL 9

Author Comment

by:yongsing
ID: 17113293
We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our client said that it must be set to true. All their production servers are set to true.
0
 
LVL 35

Accepted Solution

by:
girionis earned 400 total points
ID: 17114115
> But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?

This is used for dynamic class generation purposes. Some applications need to dynamically generate and load classes, the MaxPermSize is used to set this permanent generation size.

Have a look here:

http://java.sun.com/docs/hotspot/gc/
http://forum.java.sun.com/thread.jspa?tstart=0&forumID=27&threadID=343789&trange=15

>We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our
>client said that it must be set to true. All their production servers are set to true.

Then I think we might have found the problem. If you can't set the PRODUCTION_MODE to false then you will need to find a workaround.  Does the crash happen to all servers? If not then try to find out the specifications the servers that do not crash and see if you can have the same to those who crash. Otherwise try to upgrade/downgrade/change the VM (for example jrockit instead of Sun)/OS/ and see if the problem persists.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17120780
Hi girionis,

(1) Is it -XX:MaxPermSize or -XmaxPermSize? I have encountered both on web sites I searched, but the former seems to be the correct one.

(2) Is there a MinPermSize as well?

(3) Do I set this to the Admin as well as the managed servers?
0
 
LVL 35

Expert Comment

by:girionis
ID: 17120797
1) I think is the latter (-XMaxPermSize) but I am not 100% sure.

2) Not sure.

3) YOu need to set it to all server instances that will generate classes, so yes, both managed servers and admin (although I am not sure if admin needs it since it won't be running any applications)
0

Featured Post

Supports up to 4K resolution!

The VS192 2-Port 4K DisplayPort Splitter is perfect for anyone who needs to send one source of DisplayPort high definition video to two or four DisplayPort displays. The VS192 can split and also expand DisplayPort audio/video signal on two or four DisplayPort monitors.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses
Course of the Month8 days, 22 hours left to enroll

765 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question