Solved

Linux Server Crashing Issues

Posted on 2013-12-11
20
370 Views
Last Modified: 2014-01-31
My dedicated server every day or two will become unresponsive.  I can't get to any of my sites, WHM, or even SSH in via putty when this happens.  I have to request a power cycle and then the server and everything comes back up.

I want to figure out how/why this is happening.  I'm on a CentOS box running whm/panel.  How/where do I look at log files to see what is causing this issue?
0
Comment
Question by:N R
  • 10
  • 7
20 Comments
 
LVL 19

Expert Comment

by:xterm
ID: 39711831
Is this by chance a server hosted on Liquidweb?
0
 
LVL 11

Author Comment

by:N R
ID: 39711930
No, it's godaddy.
0
 
LVL 19

Expert Comment

by:xterm
ID: 39711935
When this happens, are you even able to ping your server and get responses, or is it totally frozen?
0
 
LVL 11

Author Comment

by:N R
ID: 39711937
Totally frozen.
0
 
LVL 19

Expert Comment

by:xterm
ID: 39711948
Okay, please log into your server as root, open /var/log/messages in a text editor, and look for the date/time of the last freeze, and paste as many lines leading up to that freeze as you can grab.

I'm surprised the Godaddy admins aren't doing this for you...
0
 
LVL 11

Author Comment

by:N R
ID: 39711970
The server is unmanaged, so they don't troubleshoot issues like this unless I were to upgrade to managed.  It's pretty expensive though and since the sites hosted on this server don't make an real $ it's not an option to upgrade.

I have putty open with the messages file open, how do I select and copy from it as it's a lot of messages and can't seem to highlight/pageup with keyboard.
0
 
LVL 19

Expert Comment

by:xterm
ID: 39711982
Is there any way you can copy the file to somewhere in your web server tree, zip it up, and post a link to it so I can download it directly?

So:

cp /var/log/messages /path/to/my/webcontent
gzip -9 /path/to/my/webcontent/messages
0
 
LVL 11

Author Comment

by:N R
ID: 39712006
Ok here you go.  It happened ~8:40am this morning.  So appears it happened at this point in the file:

Dec 11 08:36:26 - line 10714

But I'm not sure as I'm no expert in this area.
messages
0
 
LVL 19

Assisted Solution

by:xterm
xterm earned 500 total points
ID: 39712047
Okay, it's pretty clear what is happening - you have some PHP code on your webserver which spawns over and over again and the processes don't die off.  The machine runs out of memory, and tries to invoke the oom_killer which kills off memory hog processes, but does not always succeed in saving the system.

Have you added any new web apps via WHM lately which may be responsible for this?
0
Give your grad a cloud of their own!

With up to 8TB of storage, give your favorite graduate their own personal cloud to centralize all their photos, videos and music in one safe place. They can save, sync and share all their stuff, and automatic photo backup helps free up space on their smartphone and tablet.

 
LVL 11

Author Comment

by:N R
ID: 39712053
Not any webpages.  I've added a teamspeak and jabber server in the past couple of months.

How can I tell which php code is spawning and not dying off?
0
 
LVL 19

Expert Comment

by:xterm
ID: 39712090
There's really no easy way of knowing which php processes were spawned by which pages without literally blocking the site to the outside world, and then visiting your pages one by one and looking in the process list ("ps ax | grep php") after you've hit each site.

How many php processes are running right now?

ps ax | grep php | wc -l

Since it actually shows php (and not just httpd) in the process list, that means you have some kind of web content that actually spawns external processes, because under normal circumstances, you would not see this.  Do you have pages that kick off file system functionality, or perhaps an external database command?  It may just be one rogue app doing this.
0
 
LVL 11

Author Comment

by:N R
ID: 39712123
That code returned the number
2

I'm not aware of any web content that spawns external processes.  I have websites and forums on the server.  The teamspeak and jabber messenger I'm not sure what all they touch.

Is there monitoring software or something I can install to drill down more?
0
 
LVL 19

Expert Comment

by:xterm
ID: 39712278
Well, the fact that it returned 2, means there is one process with the name "php" in it running.  I figure that number will grow over time.

There's not clear way of knowing whether the hundreds of running PHP processes at the time of your crash were a symptom or the cause, but in my experience, it's always a runaway web process spawning children that don't complete their operations and/or die off.

You might Google around and see if you can find anybody else having issues with Apache/PHP web servers after installing teamspeak and/or jabber.
0
 
LVL 11

Author Comment

by:N R
ID: 39717469
Hmm...what about something like?
http://www.nagios.com/
0
 
LVL 13

Expert Comment

by:darren-w-
ID: 39723839
Hi,

Turn on logging in your PHP ini  file and have it write all actions to the log filelink, also make sure your apache server is logging all actions/requests.

Wait for the system to die then capture all logged code (in that period) on both files and post them here. This should help to identify what code is causing your issues.

D
0
 
LVL 11

Author Comment

by:N R
ID: 39807296
Hm...I turned on logging via the method above, but it still isn't logging.
0
 
LVL 11

Accepted Solution

by:
N R earned 0 total points
ID: 39810536
I never figured out the issue.  Moved to nginx instead of apache and now all my issues are resolved.
0
 
LVL 11

Author Closing Comment

by:N R
ID: 39823537
Never figured out what it was other than high traffic. Moved from apache to nginx and now it's working.
0

Featured Post

What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction This article explores the design of a cache system that can improve the performance of a web site or web application.  The assumption is that the web site has many more “read” operations than “write” operations (this is commonly the ca…
When it comes to showing a 404 error page to your visitors, you do not want that generic page to show, and you especially do not want your hosting provider’s ad error page to show either. In this article, I will show you how to enable the custom 40…
The viewer will learn how to count occurrences of each item in an array.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now