Solved

JBOSS crashes

Posted on 2014-03-13
13
893 Views
Last Modified: 2014-03-17
Dear Gurus,

I am getting following error

2014-03-13 15:21:19,101 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 15:21:19,101 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-22) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,117 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-22) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:21:19,132 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,902 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 15:24:30,917 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 15:24:30,917 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 20s:467ms
2014-03-13 15:26:34,593 INFO  [org.jboss.system.server.jmx.JMXKernel] (Thread-16) Server exit called, exiting the JVM now!

JBOSS 6.1.0
Operating System = Windows 2008 Enterprise Edition R2

THIS IS PRODUCTION SERVER AND IT GET CRASHED CAN YOU PLEASE HELP ME TO IDENTIFY WHY THIS SERVER GET CRASHED

Thanks
Netsol-NOS
0
Comment
Question by:Netsol-NOS
  • 7
  • 6
13 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928636
I am getting following error
There is no error shown there
0
 

Author Comment

by:Netsol-NOS
ID: 39928662
Dear Cehj,

Thank you for your reply.

I have increased the following settings:

FROM:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms128M -Xmx512M -XX:MaxPermSize=256M"

TO:
rem # JVM memory allocation pool parameters - modify as appropriate.
set "JAVA_OPTS=-Xms256M -Xmx1024M -XX:MaxPermSize=512M"


Please note that when i check the task manager i realized that the java (JDK) is 32 bit and it is running on 64 bit OS.

So what do you suggest should i change the JDK into 64 bit ?

Secondly in 2 hours the log file get 20 + GB so i stopped the PRODUCTION Server and copy the JBOSS folder into D drive and then again start the server.
After starting the behavior is same the JBOSS Black Console is continuously running like FLASH and the log file is continuously increasing.

After crashed in morning i restart the server till now the server get crashed 2 times.

Please suggest i am in very big trouble.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928769
I've still to see any evidence of the problem. You need to post the ERROR trace from the log
0
 

Author Comment

by:Netsol-NOS
ID: 39928786
2014-03-13 22:30:03,963 INFO  [STDOUT] (Thread-1) Posting Shutdown Request to the server...

2014-03-13 22:30:03,963 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-24) Stopping: JBossAS [6.1.0.Final "Neo"]
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,963 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-24) Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-13 22:30:03,994 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-24) Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.http11.Http11Protocol] (Thread-2) Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
2014-03-14 11:31:45,984 INFO  [org.apache.coyote.ajp.AjpProtocol] (Thread-2) Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
2014-03-14 11:31:45,984 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 22s:136ms

2014-03-14 11:33:26,025 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/das].[jsp]] (http-0.0.0.0-8080-6) Servlet.service() for servlet jsp threw exception: java.lang.StackOverflowError
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:225) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at
org.apache.catalina.core.ApplicationHttpRequest.getAttribute(ApplicationHttpRequest.java:227) [:6.1.0.Final]
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:216) [:6.1.0.Final]
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [:1.0.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:734) [:6.1.0.Final]
      at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:541) [:6.1.0.Final]
      at
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39928938
Looks like one of your filtered jsp servlets is causing some stack overflow. Maybe a loop or recursion failing to exit
0
 

Author Comment

by:Netsol-NOS
ID: 39929137
Now problem is not this..

Problem is Log files are continuously producing and JBOSS console (which is black window on desktop) continuously flashing very fast with above data which i mentioned.

Hence the log file get filled in 2 hours almost 20 GB

how can i stop producing this much log file (20 GB log)
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 86

Accepted Solution

by:
CEHJ earned 500 total points
ID: 39929517
0
 

Author Comment

by:Netsol-NOS
ID: 39929627
thank you let me check
0
 

Author Closing Comment

by:Netsol-NOS
ID: 39933642
Dear All,

We have performed some actions in the jboss-logging.xml and then the problem has been resolved.

Thanks
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933659
So what was causing the crash?
0
 

Author Comment

by:Netsol-NOS
ID: 39933667
Right now dont know the reason but may be following reasons.

11:12:42,600 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:24,568 INFO  [STDOUT] in setting procedure of querynull
11:19:31,978 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:19:39,825 INFO  [STDOUT] in setting procedure of querynull
11:19:54,005 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:39:35,994 INFO  [STDOUT] java.sql.SQLRecoverableException: Io ¿¿: Connecti
n reset by peer: socket write error
11:53:34,125 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localho
t].[/das].[jsp]] Servlet.service() for servlet jsp threw exception: java.lang.O
tOfMemoryError: Java heap space

We have changed following things.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024



Shifted JAVA from 32 bit to 64 bit (JDK 1.6 64 bit)

Limit the log files and now JAVA is taking right now 2.3 GB

Thanks
0
 

Author Comment

by:Netsol-NOS
ID: 39933670
Can you please guide me

I have Windows 2008 R2 64 bit with 64 gb memory.

JBOSS is installed with JDK 1.6 64 bit.

I have set following JAVA HEAP SETTINGS.

Xms has been increased to 512
Xmx has been increased to 2048
Maxpermsize has been increased to 1024


My Task Manager shows the Java is now taking 1.9 GB only java.

Please suggest what will be the next step should we increase the following to 4 GB

Xmx has been increased to 2048 to 4096 or not.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 39933817
What you need to do first is to profile it and find out what is taking up all the memory. If there's a fundamental flaw, adjusting memory parameters isn't going to cure the problem necessarily
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

Suggested Solutions

Java contains several comparison operators (e.g., <, <=, >, >=, ==, !=) that allow you to compare primitive values. However, these operators cannot be used to compare the contents of objects. Interface Comparable is used to allow objects of a cl…
In this post we will learn how to connect and configure Android Device (Smartphone etc.) with Android Studio. After that we will run a simple Hello World Program.
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
Viewers will learn one way to get user input in Java. Introduce the Scanner object: Declare the variable that stores the user input: An example prompting the user for input: Methods you need to invoke in order to properly get  user input:

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now