?
Solved

Find out which process caused a crash!

Posted on 2004-08-30
7
Medium Priority
?
170 Views
Last Modified: 2013-12-15
Hi:

Our server crashed two days ago, and it looks like it ran out of memory and got overloaded.  Is there any way to get an idea of which process caused the crash?  

I ran:

sar -u sa28

and discovered that between 12:40pm and 12:50pm something took over the cpu and didn't let go until it final crashed 5.5 hours later.

My questions are: how can I get a listing of the processes that were running and/or created during that 10 minute period?  Is there any way to tell which processes where demanding the most cpu time at the time of the crash?

Thanks for your help.

-Charlie
0
Comment
Question by:gothamww
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
7 Comments
 
LVL 40

Accepted Solution

by:
jlevie earned 500 total points
ID: 11937341
sar only maintains past statistics so you can't get mich more than you have gotten from it. Unless you were running a much more extensive set of logging tools or the process in question caused something to be written to the messages file you probably can tell what ran away with the CPU. But, you might be able to guess who the likely culprits were from what service this machine provides. What services does this box provide?
0
 

Author Comment

by:gothamww
ID: 11942192
mainly it's used for a dynamic web site - so: httpd and mysql would be the main services.  
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 500 total points
ID: 11942419
What scripting language is being used (Perl, PHP, etc). Unless someone has dinked with php.ini and disabled the failsafes PHP should be able to do this as it has runtime memory and execution limits. Perl or C code is another matter.

I'd suggest looking at the web logs for the time of interest and see if it provides any clues as to what was happening.
0
CHALLENGE LAB: Troubleshooting Connectivity Issues

Goal: Fix the connectivity issue in the lab's AWS environment so that you can SSH into the provided EC2 instance.  

 

Author Comment

by:gothamww
ID: 11942589
Thanks for the suggestion, I'll check the web logs.

We use both perl and php.  Just so I understand better what do you mean by "runtime memory and execution limits"?  
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 500 total points
ID: 11945269
In the php.ini file there are limits set for various things, like how much memory a PHP page can, max html page size, max cpu time, etc. Those are there to keep a runaway PHP script from killing the Web server. Perl doesn't have any imposed limits so a Perl script that gets into an infinite loop can kill the server.
0
 

Author Comment

by:gothamww
ID: 11946194

thanks so much - just one last question - is there software out there that will alert the administrator when the load on the server has gotten too high for an extended period of time?  It would be nice to get notified BEFORE the server crashes, if possible.
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 500 total points
ID: 11946720
There are a number of server monitoring packages out there, like BigBrother (http://bb4.com/), Nagios (http://www.nagios.org/), etc. They can be configured to alert on a number of things, including load average.
0

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

How many times have you wanted to quickly do the same thing to a list but found yourself typing it again and again? I first figured out a small time saver with the up arrow to recall the last command but that can only get you so far if you have a bi…
Fine Tune your automatic Updates for Ubuntu / Debian
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
Suggested Courses
Course of the Month12 days, 11 hours left to enroll

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question