Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

java jvm - exit code 137 (128 + signal number for SIGKILL) on RHEL6

Posted on 2012-09-18
10
Medium Priority
?
6,628 Views
Last Modified: 2012-12-10
We have a JVM that is stopping with an exit code 137 (128 + signal number for SIGKILL) and no application logs about the stop.

I assume that the JVM would never kill itself, so this must mean that another process/user killed the JVM?

Also is there some root log that might hold details of what killed the JVM process?
0
Comment
Question by:rmundkowsky
  • 5
  • 4
10 Comments
 
LVL 86

Expert Comment

by:CEHJ
ID: 38410300
0
 

Author Comment

by:rmundkowsky
ID: 38410766
I have read that if the kernel sends a sigkill then it is in the syslog and should be see by using dmesg. But I see no kill or SIGKILL info in that log.

Regardless, my main question is would the JVM issue a SIGKILL to itself? Or is SIGKILL only produced by other process sending SIGKILL to the JVM?
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 38411494
It wouldn't normally kill itself, no
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 12

Expert Comment

by:Sharon Seth
ID: 38412721
-XX:ErrorFile=<path>  . This was added starting java 5 to specify the path for the hs_err log file . But not sure if a SIGKILL would create a log . You can try that JVM arg and see if a log gets created at the specified path
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 38412923
It wouldn't normally kill itself, no
In fact, afaik, it's not possible for a process to kill itself

Did you eliminate the OOM killer possibility?
0
 

Author Comment

by:rmundkowsky
ID: 38413405
Not sure if OOM Killer is logged, but I will look into it.  

I was using -XX:OnOutOfMemoryError and -XX:OnError and got nothing, also STDERR and STDOUT are logged. And log4j shows no errors, just was working fine and then cuts off without any exceptions.

Since there was nothing in the dmesg log, I am thinking some admin process or person killed it and not the kernel.
0
 

Author Comment

by:rmundkowsky
ID: 38511428
Well, I have not have not been able to replicate the problem again. The system Admins have stated that their were no logs of a SIGKILL, but I did note that our systems are configured to have kill signals in the audit log.  And I found you can search the logs via: ausearch -sc kill . So I am hoping this problem will reoccur and that we will get more audit info on it when it does occur. Until then, I can't note if there is a solution in this thread, but thanks for inputs.
0
 
LVL 86

Expert Comment

by:CEHJ
ID: 38511442
OK
0
 

Accepted Solution

by:
rmundkowsky earned 0 total points
ID: 38663114
Learned a little more. Not all kill signals are seen in dmesg logs.  I have seen some OOM Kills in the dmesg logs which are not related to my problem.  Also I noted that -XX:OnError and -XX:OnOutOfMemoryError work for the OOM Kills or at least Max Heap full errors and not for my problem.  So in order to get more info on kills, we modified the /etc/audit/audit.rules to include:

-a exit,always -F arch=b64 -S kill
-a exit,always -F arch=b32 -S kill

I ran some tests and "ausearch -sc kill -i" does display info on these kills. Now planning to run tests to try to get the mystery kill to occur again.

I am closing this question, though the mystery has not been fully solved yet.
0
 

Author Closing Comment

by:rmundkowsky
ID: 38675220
No solution has been found yet, but the auditing I have noted should help determine who did the kill.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I have written articles previously comparing SARDU and YUMI.  I also included a couple of lines about Easy2boot (easy2boot.com).  I have now been using, and enjoying easy2boot as my sole multiboot utility for some years and realize that it deserves …
Often times it's very very easy to extend a volume on a Linux instance in AWS, but impossible to shrink it. I wanted to contribute to the experts-exchange community a way of providing a procedure that works on an AWS instance. It can also be used on…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses
Course of the Month11 days, 6 hours left to enroll

572 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question