Solved

JBOSS crashes

Posted on 2014-03-13
13
986 Views
Last Modified: 2014-03-17
Dear Gurus,

I am getting following error

2014-03-13 15:21:19,101 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 15:21:19,101 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-22) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,902 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:24:30,917 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,917 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 20s:467ms
2014-03-13 15:26:34,593 INFO  [org.jboss.system.server.jmx.JMXKernel] (Thread-16) Server exit called, exiting the JVM now!

JBOSS 6.1.0
Operating System = Windows 2008 Enterprise Edition R2

THIS IS PRODUCTION SERVER AND IT GET CRASHED CAN YOU PLEASE HELP ME TO IDENTIFY WHY THIS SERVER GET CRASHED

Thanks
Netsol-NOS
0
Comment
Question by:Netsol-NOS
  • 7
  • 6
13 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928636
I am getting following error
There is no error shown there
0
 

Author Comment

by:Netsol-NOS
ID: 39928662
Dear Cehj,

Thank you for your reply.

I have increased the following settings:

FROM:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms128M -Xmx512M -XX:MaxPermSize=256M"

TO:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms256M -Xmx1024M -XX:MaxPermSize=512M"


Please note that when i check the task manager i realized that the java (JDK) is 32 bit and it is running on 64 bit OS.

So what do you suggest should i change the JDK into 64 bit ?

Secondly in 2 hours the log file get 20 + GB so i stopped the PRODUCTION Server and copy the JBOSS folder into D drive and then again start the server.
After starting the behavior is same the JBOSS Black Console is continuously running like FLASH and the log file is continuously increasing.

After crashed in morning i restart the server till now the server get crashed 2 times.

Please suggest i am in very big trouble.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928769
I've still to see any evidence of the problem. You need to post the ERROR trace from the log
0
NAS Cloud Backup Strategies

This article explains backup scenarios when using network storage. We review the so-called “3-2-1 strategy” and summarize the methods you can use to send NAS data to the cloud

 

Author Comment

by:Netsol-NOS
ID: 39928786
2014-03-13 22:30:03,963 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 22:30:03,963 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-24) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 22s:136ms

2014-03-14 11:33:26,025 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/das].[jsp]] (http-0.0.0.0-8080-6) Servlet.service() for servlet jsp threw exception: java.lang.StackOverflowError
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:225) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at
org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:216) [:6.1.0.Final]
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [:1.0.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:734) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:541) [:6.1.0.Final]
      at
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928938
Looks like one of your filtered jsp servlets is causing some stack overflow. Maybe a loop or recursion failing to exit
0
 

Author Comment

by:Netsol-NOS
ID: 39929137
Now problem is not this..

Problem is Log files are continuously producing and JBOSS console (which is black window on desktop) continuously flashing very fast with above data which i mentioned.

Hence the log file get filled in 2 hours almost 20 GB

how can i stop producing this much log file (20 GB log)
0
 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
ID: 39929517
0
 

Author Comment

by:Netsol-NOS
ID: 39929627
thank you let me check
0
 

Author Closing Comment

by:Netsol-NOS
ID: 39933642
Dear All,

We have performed some actions in the jboss-logging.xml and then the problem has been resolved.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933659
So what was causing the crash?
0
 

Author Comment

by:Netsol-NOS
ID: 39933667
Right now dont know the reason but may be following reasons.

11:12:42,600 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:24,568 INFO  [STDOUT] in setting procedure of querynull
11:19:31,978 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:39,825 INFO  [STDOUT] in setting procedure of querynull
11:19:54,005 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:39:35,994 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:53:34,125 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localho
t].[/das].[jsp]] Servlet.service() for servlet jsp threw exception: java.lang.O
tOfMemoryError: Java heap space

We have changed following things.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024



Shifted JAVA from 32 bit to 64 bit (JDK 1.6 64 bit)

Limit the log files and now JAVA is taking right now 2.3 GB

Thanks
0
 

Author Comment

by:Netsol-NOS
ID: 39933670
Can you please guide me

I have Windows 2008 R2 64 bit with 64 gb memory.

JBOSS is installed with JDK 1.6 64 bit.

I have set following JAVA HEAP SETTINGS.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024


My Task Manager shows the Java is now taking 1.9 GB only java.

Please suggest what will be the next step should we increase the following to 4 GB

Xmx has been increased to 2048 to 4096 or not.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933817
What you need to do first is to profile it and find out what is taking up all the memory. If there's a fundamental flaw, adjusting memory parameters isn't going to cure the problem necessarily
0

Featured Post

Master Your Team's Linux and Cloud Stack!

The average business loses $13.5M per year to ineffective training (per 1,000 employees). Keep ahead of the competition and combine in-person quality with online cost and flexibility by training with Linux Academy.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
eclipse argument 14 61
replace a word with other 1 44
Oracle SQL syntax check  without executing 6 54
tomcat startup error 5 64
For beginner Java programmers or at least those new to the Eclipse IDE, the following tutorial will show some (four) ways in which you can import your Java projects to your Eclipse workbench. Introduction While learning Java can be done with…
This was posted to the Netbeans forum a Feb, 2010 and I also sent it to Verisign. Who didn't help much in my struggles to get my application signed. ------------------------- Start The idea here is to target your cell phones with the correct…
Viewers learn about the “while” loop and how to utilize it correctly in Java. Additionally, viewers begin exploring how to include conditional statements within a while loop and avoid an endless loop. Define While Loop: Basic Example: Explanatio…
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:

773 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question