Solved

WebLogic Server Core Dump on Solaris

Posted on 2006-07-06
17
5,337 Views
Last Modified: 2013-12-10
We have a web application deployed on WebLogic Server8.1 on 2 Solaris machine. The 1st Solaris machine contains the Admin server and the 1st Managed server. The 2nd Solaris machine contains the 2nd Managed server.

Every few hours, it seems that the WebLogic Server will crash, generating a core dump, and an error file named hs_err_pidXXXXX.log, with contents shown below. Our client thinks that it is memory problem. We have already set the min/max memory to the JVM to be 1GB. Our client thinks that it could be due to our application (allocating objects and not garbage collected).

Can anyone interprets the following file and tell us what the problem could be? Thanks!

--- Contents of hs_err_pid18110.log ---

Unexpected Signal : 11 occurred at PC=0xFEDCBCAC
Function=[Unknown. Nearest: JVM_FillInStackTrace+0x4CE4]
Library=/app/weblogic/jdk142_05/jre/lib/sparc/server/libjvm.so

Current Java thread:

Dynamic libraries:
0x10000       /app/weblogic/jdk142_05/bin/java
0xff370000       /usr/lib/libthread.so.1
0xff3fa000       /usr/lib/libdl.so.1
0xff280000       /usr/lib/libc.so.1
0xff3a0000       /usr/platform/SUNW,Sun-Fire-V440/lib/libc_psr.so.1
0xfec00000       /app/weblogic/jdk142_05/jre/lib/sparc/server/libjvm.so
0xff240000       /usr/lib/libCrun.so.1
0xff220000       /usr/lib/libsocket.so.1
0xfeb00000       /usr/lib/libnsl.so.1
0xfeab0000       /usr/lib/libm.so.1
0xff200000       /usr/lib/libsched.so.1
0xfebe0000       /usr/lib/libmp.so.2
0xfea80000       /app/weblogic/jdk142_05/jre/lib/sparc/native_threads/libhpi.so
0xfea60000       /usr/lib/nss_files.so.1
0xfea30000       /app/weblogic/jdk142_05/jre/lib/sparc/libverify.so
0xfe9f0000       /app/weblogic/jdk142_05/jre/lib/sparc/libjava.so
0xfe9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libzip.so
0xb07d0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnet.so
0xb07b0000       /app/weblogic/server/lib/solaris/libweblogicunix1.so
0xb0660000       /app/weblogic/server/lib/solaris/libstackdump.so
0xb01e0000       /app/weblogic/server/lib/solaris/libmuxer.so
0xb01c0000       /usr/ucblib/libucb.so.1
0xafda0000       /usr/lib/libresolv.so.2
0xb0190000       /usr/lib/libelf.so.1
0xaffe0000       /app/weblogic/jdk142_05/jre/lib/sparc/libnio.so
0xadae0000       /usr/lib/librt.so.1
0xadac0000       /usr/lib/libaio.so.1
0xada90000       /usr/lib/libmd5.so.1
0xad9e0000       /usr/lib/libsendfile.so.1
0xad9c0000       /app/weblogic/jdk142_05/jre/lib/sparc/libioser12.so

Heap at VM Abort:
Heap
 def new generation   total 169728K, used 56817K [0xb5800000, 0xc02c0000, 0xcad50000)
  eden space 164608K,  31% used [0xb5800000, 0xb8a7c750, 0xbf8c0000)
  from space 5120K, 100% used [0xbfdc0000, 0xc02c0000, 0xc02c0000)
  to   space 5120K,   0% used [0xbf8c0000, 0xbf8c0000, 0xbfdc0000)
 tenured generation   total 349568K, used 289884K [0xcad50000, 0xe02b0000, 0xf5800000)
   the space 349568K,  82% used [0xcad50000, 0xdc8672d8, 0xdc867400, 0xe02b0000)
 compacting perm gen  total 37888K, used 37369K [0xf5800000, 0xf7d00000, 0xf9800000)
   the space 37888K,  98% used [0xf5800000, 0xf7c7e7b8, 0xf7c7e800, 0xf7d00000)

Local Time = Fri Jul  7 01:01:55 2006
Elapsed Time = 27753
#
# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002EF 01
# Please report this error at
# http://java.sun.com/cgi-bin/bugreport.cgi
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode)
#
0
Comment
Question by:yongsing
  • 9
  • 6
17 Comments
 
LVL 35

Expert Comment

by:girionis
ID: 17056941
Hi yongsing

Have a llok here: http://forum.java.sun.com/thread.jspa?threadID=307252&start=60 some say it's hardware problem some say it's a problem with JDK. A suggested solution is to get rid of -server switch. Maybe if you try JRockit you will have better results.

Cheers
0
 
LVL 9

Author Comment

by:yongsing
ID: 17057152
Thanks, girionis. That is quite a useful link.

I do not think that the application itself can cause the JVM to crash, as what our client thinks. I mean, if you run out of memory to create new objects, the JVM would have thrown OutOfMemoryError, right?

We'll try the suggested solutions to see if they solve our problem.
0
 
LVL 35

Expert Comment

by:girionis
ID: 17057254
> the JVM would have thrown OutOfMemoryError, right?

Yes this is true, unless the JVM implementation is buggy.
0
 
LVL 3

Expert Comment

by:anumalas
ID: 17057667
Hi

Try to increase the jvm heap size

for eg: java -Xms64m -Xmx512m


you can set this heap size in startWeblogic.sh for JAVA_OPTIONS environmental variable.

Hope this will help :)
0
 
LVL 9

Author Comment

by:yongsing
ID: 17067135
>> Try to increase the jvm heap size

We have already increased to 512, then 1G, but still getting the crash.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17070209
Does JRocket comes with WebLogic Server 8.1? Where is it?
0
 
LVL 35

Expert Comment

by:girionis
ID: 17070864
It does. It should be on the root folder of the BEA installation. On my computer it is in the D:\bea\jrockit81sp4_142_05
0
 
LVL 9

Author Comment

by:yongsing
ID: 17071546
It doesn't seem to come with the BEA installation on our Solaris machine. I have it on my own computer though.
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 9

Author Comment

by:yongsing
ID: 17071552
The -server option seems to be there when you set PRODUCTION_MODE to true. I think if we want to remove this -server option, PRODUCTION_MODE would have to be set to false. But I was told that in production, PRODUCTION_MODE should be set to true.
0
 
LVL 35

Expert Comment

by:girionis
ID: 17071573
Well, tbh there is little difference between production and development mode. The server just optimizes a few things. Just give it a go and see if the problem persists. If it does not then we know ehere the problem is and we can start working from there.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17071816
Thanks, we'll be trying that out tomorrow.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17113288
So far I've used the -Xmx and -Xms options. But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?
0
 
LVL 9

Author Comment

by:yongsing
ID: 17113293
We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our client said that it must be set to true. All their production servers are set to true.
0
 
LVL 35

Accepted Solution

by:
girionis earned 100 total points
ID: 17114115
> But there is also a -Xx:MaxPermSize option which I did not set. You know what it is for?

This is used for dynamic class generation purposes. Some applications need to dynamically generate and load classes, the MaxPermSize is used to set this permanent generation size.

Have a look here:

http://java.sun.com/docs/hotspot/gc/
http://forum.java.sun.com/thread.jspa?tstart=0&forumID=27&threadID=343789&trange=15

>We don't seem to be encountering the crash after I set the PRODUCTION_MODE to false. However, our
>client said that it must be set to true. All their production servers are set to true.

Then I think we might have found the problem. If you can't set the PRODUCTION_MODE to false then you will need to find a workaround.  Does the crash happen to all servers? If not then try to find out the specifications the servers that do not crash and see if you can have the same to those who crash. Otherwise try to upgrade/downgrade/change the VM (for example jrockit instead of Sun)/OS/ and see if the problem persists.
0
 
LVL 9

Author Comment

by:yongsing
ID: 17120780
Hi girionis,

(1) Is it -XX:MaxPermSize or -XmaxPermSize? I have encountered both on web sites I searched, but the former seems to be the correct one.

(2) Is there a MinPermSize as well?

(3) Do I set this to the Admin as well as the managed servers?
0
 
LVL 35

Expert Comment

by:girionis
ID: 17120797
1) I think is the latter (-XMaxPermSize) but I am not 100% sure.

2) Not sure.

3) YOu need to set it to all server instances that will generate classes, so yes, both managed servers and admin (although I am not sure if admin needs it since it won't be running any applications)
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Problem Imaging Computers With Clonezilla 2 82
Using Tomcat as raspberry pi java app server 13 134
Hosting application in Apache and Tomcat 1 91
unix solaris snoop command 6 84
This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
Configure Web Service (server application) I. Configure security for Web Services methods First, we need to protect Session bean which implements the service: 1. Open EJB deployment descriptor (ejb-jar.xml) in the EJB project that contains you…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

930 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now