?
Solved

JBOSS crashes

Posted on 2014-03-13
13
Medium Priority
?
1,233 Views
Last Modified: 2014-03-17
Dear Gurus,

I am getting following error

2014-03-13 15:21:19,101 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 15:21:19,101 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-22) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,902 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:24:30,917 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,917 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 20s:467ms
2014-03-13 15:26:34,593 INFO  [org.jboss.system.server.jmx.JMXKernel] (Thread-16) Server exit called, exiting the JVM now!

JBOSS 6.1.0
Operating System = Windows 2008 Enterprise Edition R2

THIS IS PRODUCTION SERVER AND IT GET CRASHED CAN YOU PLEASE HELP ME TO IDENTIFY WHY THIS SERVER GET CRASHED

Thanks
Netsol-NOS
0
Comment
Question by:Netsol-NOS
  • 7
  • 6
13 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928636
I am getting following error
There is no error shown there
0
 

Author Comment

by:Netsol-NOS
ID: 39928662
Dear Cehj,

Thank you for your reply.

I have increased the following settings:

FROM:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms128M -Xmx512M -XX:MaxPermSize=256M"

TO:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms256M -Xmx1024M -XX:MaxPermSize=512M"


Please note that when i check the task manager i realized that the java (JDK) is 32 bit and it is running on 64 bit OS.

So what do you suggest should i change the JDK into 64 bit ?

Secondly in 2 hours the log file get 20 + GB so i stopped the PRODUCTION Server and copy the JBOSS folder into D drive and then again start the server.
After starting the behavior is same the JBOSS Black Console is continuously running like FLASH and the log file is continuously increasing.

After crashed in morning i restart the server till now the server get crashed 2 times.

Please suggest i am in very big trouble.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928769
I've still to see any evidence of the problem. You need to post the ERROR trace from the log
0
Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

 

Author Comment

by:Netsol-NOS
ID: 39928786
2014-03-13 22:30:03,963 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 22:30:03,963 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-24) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 22s:136ms

2014-03-14 11:33:26,025 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/das].[jsp]] (http-0.0.0.0-8080-6) Servlet.service() for servlet jsp threw exception: java.lang.StackOverflowError
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:225) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at
org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:216) [:6.1.0.Final]
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [:1.0.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:734) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:541) [:6.1.0.Final]
      at
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928938
Looks like one of your filtered jsp servlets is causing some stack overflow. Maybe a loop or recursion failing to exit
0
 

Author Comment

by:Netsol-NOS
ID: 39929137
Now problem is not this..

Problem is Log files are continuously producing and JBOSS console (which is black window on desktop) continuously flashing very fast with above data which i mentioned.

Hence the log file get filled in 2 hours almost 20 GB

how can i stop producing this much log file (20 GB log)
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 2000 total points
ID: 39929517
0
 

Author Comment

by:Netsol-NOS
ID: 39929627
thank you let me check
0
 

Author Closing Comment

by:Netsol-NOS
ID: 39933642
Dear All,

We have performed some actions in the jboss-logging.xml and then the problem has been resolved.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933659
So what was causing the crash?
0
 

Author Comment

by:Netsol-NOS
ID: 39933667
Right now dont know the reason but may be following reasons.

11:12:42,600 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:24,568 INFO  [STDOUT] in setting procedure of querynull
11:19:31,978 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:39,825 INFO  [STDOUT] in setting procedure of querynull
11:19:54,005 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:39:35,994 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:53:34,125 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localho
t].[/das].[jsp]] Servlet.service() for servlet jsp threw exception: java.lang.O
tOfMemoryError: Java heap space

We have changed following things.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024



Shifted JAVA from 32 bit to 64 bit (JDK 1.6 64 bit)

Limit the log files and now JAVA is taking right now 2.3 GB

Thanks
0
 

Author Comment

by:Netsol-NOS
ID: 39933670
Can you please guide me

I have Windows 2008 R2 64 bit with 64 gb memory.

JBOSS is installed with JDK 1.6 64 bit.

I have set following JAVA HEAP SETTINGS.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024


My Task Manager shows the Java is now taking 1.9 GB only java.

Please suggest what will be the next step should we increase the following to 4 GB

Xmx has been increased to 2048 to 4096 or not.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933817
What you need to do first is to profile it and find out what is taking up all the memory. If there's a fundamental flaw, adjusting memory parameters isn't going to cure the problem necessarily
0

Featured Post

[Webinar On Demand] Database Backup and Recovery

Does your company store data on premises, off site, in the cloud, or a combination of these? If you answered “yes”, you need a data backup recovery plan that fits each and every platform. Watch now as as Percona teaches us how to build agile data backup recovery plan.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

INTRODUCTION Working with files is a moderately common task in Java.  For most projects hard coding the file names, using parameters in configuration files, or using command-line arguments is sufficient.   However, when your application has vi…
For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
Viewers learn about the third conditional statement “else if” and use it in an example program. Then additional information about conditional statements is provided, covering the topic thoroughly. Viewers learn about the third conditional statement …
Viewers will learn about basic arrays, how to declare them, and how to use them. Introduction and definition: Declare an array and cover the syntax of declaring them: Initialize every index in the created array: Example/Features of a basic arr…
Suggested Courses
Course of the Month8 days, 21 hours left to enroll

621 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question