Solved

Find out which process caused a crash!

Posted on 2004-08-30
7
164 Views
Last Modified: 2013-12-15
Hi:

Our server crashed two days ago, and it looks like it ran out of memory and got overloaded.  Is there any way to get an idea of which process caused the crash?  

I ran:

sar -u sa28

and discovered that between 12:40pm and 12:50pm something took over the cpu and didn't let go until it final crashed 5.5 hours later.

My questions are: how can I get a listing of the processes that were running and/or created during that 10 minute period?  Is there any way to tell which processes where demanding the most cpu time at the time of the crash?

Thanks for your help.

-Charlie
0
Comment
Question by:gothamww
  • 4
  • 3
7 Comments
 
LVL 40

Accepted Solution

by:
jlevie earned 125 total points
ID: 11937341
sar only maintains past statistics so you can't get mich more than you have gotten from it. Unless you were running a much more extensive set of logging tools or the process in question caused something to be written to the messages file you probably can tell what ran away with the CPU. But, you might be able to guess who the likely culprits were from what service this machine provides. What services does this box provide?
0
 

Author Comment

by:gothamww
ID: 11942192
mainly it's used for a dynamic web site - so: httpd and mysql would be the main services.  
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 125 total points
ID: 11942419
What scripting language is being used (Perl, PHP, etc). Unless someone has dinked with php.ini and disabled the failsafes PHP should be able to do this as it has runtime memory and execution limits. Perl or C code is another matter.

I'd suggest looking at the web logs for the time of interest and see if it provides any clues as to what was happening.
0
6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

 

Author Comment

by:gothamww
ID: 11942589
Thanks for the suggestion, I'll check the web logs.

We use both perl and php.  Just so I understand better what do you mean by "runtime memory and execution limits"?  
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 125 total points
ID: 11945269
In the php.ini file there are limits set for various things, like how much memory a PHP page can, max html page size, max cpu time, etc. Those are there to keep a runaway PHP script from killing the Web server. Perl doesn't have any imposed limits so a Perl script that gets into an infinite loop can kill the server.
0
 

Author Comment

by:gothamww
ID: 11946194

thanks so much - just one last question - is there software out there that will alert the administrator when the load on the server has gotten too high for an extended period of time?  It would be nice to get notified BEFORE the server crashes, if possible.
0
 
LVL 40

Assisted Solution

by:jlevie
jlevie earned 125 total points
ID: 11946720
There are a number of server monitoring packages out there, like BigBrother (http://bb4.com/), Nagios (http://www.nagios.org/), etc. They can be configured to alert on a number of things, including load average.
0

Featured Post

Comprehensive Backup Solutions for Microsoft

Acronis protects the complete Microsoft technology stack: Windows Server, Windows PC, laptop and Surface data; Microsoft business applications; Microsoft Hyper-V; Azure VMs; Microsoft Windows Server 2016; Microsoft Exchange 2016 and SQL Server 2016.

Join & Write a Comment

Currently, there is not an RPM package available under the RHEL/Fedora/CentOS distributions that gives you a quick and easy way to allow PHP to interface with Oracle. As a result, I have included a set of instructions on how to do this with minimal …
If you have a server on collocation with the super-fast CPU, that doesn't mean that you get it running at full power. Here is a preamble. When doing inventory of Linux servers, that I'm administering, I've found that some of them are running on l…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now