Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 171
  • Last Modified:

Find out which process caused a crash!

Hi:

Our server crashed two days ago, and it looks like it ran out of memory and got overloaded.  Is there any way to get an idea of which process caused the crash?  

I ran:

sar -u sa28

and discovered that between 12:40pm and 12:50pm something took over the cpu and didn't let go until it final crashed 5.5 hours later.

My questions are: how can I get a listing of the processes that were running and/or created during that 10 minute period?  Is there any way to tell which processes where demanding the most cpu time at the time of the crash?

Thanks for your help.

-Charlie
0
gothamww
Asked:
gothamww
  • 4
  • 3
4 Solutions
 
jlevieCommented:
sar only maintains past statistics so you can't get mich more than you have gotten from it. Unless you were running a much more extensive set of logging tools or the process in question caused something to be written to the messages file you probably can tell what ran away with the CPU. But, you might be able to guess who the likely culprits were from what service this machine provides. What services does this box provide?
0
 
gothamwwAuthor Commented:
mainly it's used for a dynamic web site - so: httpd and mysql would be the main services.  
0
 
jlevieCommented:
What scripting language is being used (Perl, PHP, etc). Unless someone has dinked with php.ini and disabled the failsafes PHP should be able to do this as it has runtime memory and execution limits. Perl or C code is another matter.

I'd suggest looking at the web logs for the time of interest and see if it provides any clues as to what was happening.
0
Learn Veeam advantages over legacy backup

Every day, more and more legacy backup customers switch to Veeam. Technologies designed for the client-server era cannot restore any IT service running in the hybrid cloud within seconds. Learn top Veeam advantages over legacy backup and get Veeam for the price of your renewal

 
gothamwwAuthor Commented:
Thanks for the suggestion, I'll check the web logs.

We use both perl and php.  Just so I understand better what do you mean by "runtime memory and execution limits"?  
0
 
jlevieCommented:
In the php.ini file there are limits set for various things, like how much memory a PHP page can, max html page size, max cpu time, etc. Those are there to keep a runaway PHP script from killing the Web server. Perl doesn't have any imposed limits so a Perl script that gets into an infinite loop can kill the server.
0
 
gothamwwAuthor Commented:

thanks so much - just one last question - is there software out there that will alert the administrator when the load on the server has gotten too high for an extended period of time?  It would be nice to get notified BEFORE the server crashes, if possible.
0
 
jlevieCommented:
There are a number of server monitoring packages out there, like BigBrother (http://bb4.com/), Nagios (http://www.nagios.org/), etc. They can be configured to alert on a number of things, including load average.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

  • 4
  • 3
Tackle projects and never again get stuck behind a technical roadblock.
Join Now