Solved

JBOSS crashes

Posted on 2014-03-13
13
966 Views
Last Modified: 2014-03-17
Dear Gurus,

I am getting following error

2014-03-13 15:21:19,101 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 15:21:19,101 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-22) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,902 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:24:30,917 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,917 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 20s:467ms
2014-03-13 15:26:34,593 INFO  [org.jboss.system.server.jmx.JMXKernel] (Thread-16) Server exit called, exiting the JVM now!

JBOSS 6.1.0
Operating System = Windows 2008 Enterprise Edition R2

THIS IS PRODUCTION SERVER AND IT GET CRASHED CAN YOU PLEASE HELP ME TO IDENTIFY WHY THIS SERVER GET CRASHED

Thanks
Netsol-NOS
0
Comment
Question by:Netsol-NOS
  • 7
  • 6
13 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928636
I am getting following error
There is no error shown there
0
 

Author Comment

by:Netsol-NOS
ID: 39928662
Dear Cehj,

Thank you for your reply.

I have increased the following settings:

FROM:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms128M -Xmx512M -XX:MaxPermSize=256M"

TO:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms256M -Xmx1024M -XX:MaxPermSize=512M"


Please note that when i check the task manager i realized that the java (JDK) is 32 bit and it is running on 64 bit OS.

So what do you suggest should i change the JDK into 64 bit ?

Secondly in 2 hours the log file get 20 + GB so i stopped the PRODUCTION Server and copy the JBOSS folder into D drive and then again start the server.
After starting the behavior is same the JBOSS Black Console is continuously running like FLASH and the log file is continuously increasing.

After crashed in morning i restart the server till now the server get crashed 2 times.

Please suggest i am in very big trouble.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928769
I've still to see any evidence of the problem. You need to post the ERROR trace from the log
0
 

Author Comment

by:Netsol-NOS
ID: 39928786
2014-03-13 22:30:03,963 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 22:30:03,963 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-24) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 22s:136ms

2014-03-14 11:33:26,025 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/das].[jsp]] (http-0.0.0.0-8080-6) Servlet.service() for servlet jsp threw exception: java.lang.StackOverflowError
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:225) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at
org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:216) [:6.1.0.Final]
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [:1.0.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:734) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:541) [:6.1.0.Final]
      at
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928938
Looks like one of your filtered jsp servlets is causing some stack overflow. Maybe a loop or recursion failing to exit
0
 

Author Comment

by:Netsol-NOS
ID: 39929137
Now problem is not this..

Problem is Log files are continuously producing and JBOSS console (which is black window on desktop) continuously flashing very fast with above data which i mentioned.

Hence the log file get filled in 2 hours almost 20 GB

how can i stop producing this much log file (20 GB log)
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
ID: 39929517
0
 

Author Comment

by:Netsol-NOS
ID: 39929627
thank you let me check
0
 

Author Closing Comment

by:Netsol-NOS
ID: 39933642
Dear All,

We have performed some actions in the jboss-logging.xml and then the problem has been resolved.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933659
So what was causing the crash?
0
 

Author Comment

by:Netsol-NOS
ID: 39933667
Right now dont know the reason but may be following reasons.

11:12:42,600 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:24,568 INFO  [STDOUT] in setting procedure of querynull
11:19:31,978 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:39,825 INFO  [STDOUT] in setting procedure of querynull
11:19:54,005 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:39:35,994 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:53:34,125 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localho
t].[/das].[jsp]] Servlet.service() for servlet jsp threw exception: java.lang.O
tOfMemoryError: Java heap space

We have changed following things.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024



Shifted JAVA from 32 bit to 64 bit (JDK 1.6 64 bit)

Limit the log files and now JAVA is taking right now 2.3 GB

Thanks
0
 

Author Comment

by:Netsol-NOS
ID: 39933670
Can you please guide me

I have Windows 2008 R2 64 bit with 64 gb memory.

JBOSS is installed with JDK 1.6 64 bit.

I have set following JAVA HEAP SETTINGS.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024


My Task Manager shows the Java is now taking 1.9 GB only java.

Please suggest what will be the next step should we increase the following to 4 GB

Xmx has been increased to 2048 to 4096 or not.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933817
What you need to do first is to profile it and find out what is taking up all the memory. If there's a fundamental flaw, adjusting memory parameters isn't going to cure the problem necessarily
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

After being asked a question last year, I went into one of my moods where I did some research and code just for the fun and learning of it all.  Subsequently, from this journey, I put together this article on "Range Searching Using Visual Basic.NET …
Introduction This article is the last of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers our test design approach and then goes through a simple test case example, how …
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:
Viewers will learn how to properly install Eclipse with the necessary JDK, and will take a look at an introductory Java program. Download Eclipse installation zip file: Extract files from zip file: Download and install JDK 8: Open Eclipse and …

910 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now