Linux Server Crashing Issues

Is this by chance a server hosted on Liquidweb?

No, it's godaddy.

When this happens, are you even able to ping your server and get responses, or is it totally frozen?

Totally frozen.

Okay, please log into your server as root, open /var/log/messages in a text editor, and look for the date/time of the last freeze, and paste as many lines leading up to that freeze as you can grab.

I'm surprised the Godaddy admins aren't doing this for you...

The server is unmanaged, so they don't troubleshoot issues like this unless I were to upgrade to managed.  It's pretty expensive though and since the sites hosted on this server don't make an real $ it's not an option to upgrade.

I have putty open with the messages file open, how do I select and copy from it as it's a lot of messages and can't seem to highlight/pageup with keyboard.

Is there any way you can copy the file to somewhere in your web server tree, zip it up, and post a link to it so I can download it directly?

So:

cp /var/log/messages /path/to/my/webcontent
gzip -9 /path/to/my/webcontent/messages

Ok here you go.  It happened ~8:40am this morning.  So appears it happened at this point in the file:

Dec 11 08:36:26 - line 10714

But I'm not sure as I'm no expert in this area.
messages

Not any webpages.  I've added a teamspeak and jabber server in the past couple of months.

How can I tell which php code is spawning and not dying off?

There's really no easy way of knowing which php processes were spawned by which pages without literally blocking the site to the outside world, and then visiting your pages one by one and looking in the process list ("ps ax | grep php") after you've hit each site.

How many php processes are running right now?

ps ax | grep php | wc -l

Since it actually shows php (and not just httpd) in the process list, that means you have some kind of web content that actually spawns external processes, because under normal circumstances, you would not see this.  Do you have pages that kick off file system functionality, or perhaps an external database command?  It may just be one rogue app doing this.

That code returned the number
2

I'm not aware of any web content that spawns external processes.  I have websites and forums on the server.  The teamspeak and jabber messenger I'm not sure what all they touch.

Is there monitoring software or something I can install to drill down more?

Well, the fact that it returned 2, means there is one process with the name "php" in it running.  I figure that number will grow over time.

There's not clear way of knowing whether the hundreds of running PHP processes at the time of your crash were a symptom or the cause, but in my experience, it's always a runaway web process spawning children that don't complete their operations and/or die off.

You might Google around and see if you can find anybody else having issues with Apache/PHP web servers after installing teamspeak and/or jabber.

Hmm...what about something like?
http://www.nagios.com/

Hi,

Turn on logging in your PHP ini  file and have it write all actions to the log filelink, also make sure your apache server is logging all actions/requests.

Wait for the system to die then capture all logged code (in that period) on both files and post them here. This should help to identify what code is causing your issues.

D

Hm...I turned on logging via the method above, but it still isn't logging.

Never figured out what it was other than high traffic. Moved from apache to nginx and now it's working.