Solved

Defunt and zombie php processes causing high cpu load

Posted on 2014-10-03
11
567 Views
Last Modified: 2016-03-02
Hi,

On a dedicated server with 16 GB RAM running a single Magento site, we have been constantly facing high cpu loads. Running top, we find several php and defunct php processes generating as high as 98% cpu loads.

But, here's the strange thing - when we identify the parent pid of the defunct process and then strace the parent pid, we get the message that there is no such process. Sometimes, we get the same results for the non-defunct php processes as well. I am attaching a screen shot of the results strace -p pid command.

And, all these calls are to Magento's index.php page, which is usually the case with any CMS site.

We have done a fair bit of tweaking over the past week to fix front end issues through Y!Slow and Google Page Speed Insight, optimize Apache by moving to Worker MPM, optimize MySQL by running the mysqltuner.pl script every 24 hours etc., but nothing has an impact on these defunct and php processes and the high cpu loads they generate.

Does anyone have an idea how to identify these processes and stop them from generating such high cpu loads?

Thanks in advance.
php-zombie-process.jpg
0
Comment
Question by:scmeeven
  • 5
  • 4
  • 2
11 Comments
 
LVL 34

Accepted Solution

by:
gr8gonzo earned 500 total points
ID: 40359497
First, your screenshot shows two commands that BOTH indicate the process 10721 doesn't exist.

Second, I would suggest running some code profiling:
http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/A_12179-PHP-Code-Profiling-and-Analyzing-Performance.html

That should help tell you exactly what the PHP processes are doing.
0
 
LVL 58

Expert Comment

by:Gary
ID: 40359570
Is this shared hosting?
Is it a busy site?
Try disabling any plugins, firstly disable all plugins and see if it occurs, if not then reenable them one by one.
0
 

Author Comment

by:scmeeven
ID: 40359953
@gr8gonzo, 10721 was actually marked as defunct in top, so I ran the first command to identify its parent id, which came back as 11253. The intriguing thing about these php processes is that I get the no such process error even if I run strace within seconds of seeing them in PuTTY (opened two sessions to do so). But, the high cpu load they generate and their impact on the server is unmistakable. I will check out your suggestion on code profiling.

@Gary, as I mentioned, this is a dedicated server. Yes, it's a pretty busy site, but probably in the dozens to hundred visitors per hour range.
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 
LVL 58

Expert Comment

by:Gary
ID: 40359976
I would really start with going thru the plugins, are you using a cache on the server?
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40360291
No intention of being negative, but dozens to hundred visitors per hour isn't a busy site, at least not in the sense that you should be having load issues. Even a hundred visitors per hour is slightly over one per minute on average, which means those processes have a pretty decent amount of elbowroom to spin up and shut down.

Just because some people have some different ideas of terminology - is it truly a completely-independent, physical dedicated server? Some people think that VPS is a dedicated server, or that virtualized servers (e.g. a virtual instance via KVM or other virtual manager) is considered a dedicated server, but any virtualized solution is still sharing resources on the backend. A fully-dedicated server is a physical machine that doesn't share any of its resources.

Gary's suggestions about plugins (I think Magento calls them extensions) is a good one. All it takes is one poorly-written plugin to turn a speedy site into a slug. The code profiling may help you identify which plugin is doing the damage, assuming you can replicate the problem by yourself.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40360295
Also, I can't recall if I included this in my article on code profiling, but don't leave XDebug enabled on the site once you're done. It's a heavy debugging extension that can have a significant performance impact, depending on the code.
0
 

Author Comment

by:scmeeven
ID: 40361522
@gr8gonzo, this is a genuine, physical, dedicated server.  And, thanks for the tip about disabling xdebug. I will update this question with the results of that test.
0
 

Author Comment

by:scmeeven
ID: 40361776
I have installed xdebug, added the correct path to the Zend extension in php.ini, set trigger to 1 and output dir to /tmp and restarted Apache.

Then I ran www.domain.com/index.php?XDEBUG_PROFILE=1 but there's no cachegrind.out file generated in the /tmp directory. Because CMSes like Magento call the index.php file, I then tried www.domain.com/?XDEBUG_PROFILE=1 but still with no resulting cachegrind.out file.

Is there something I am doing wrong?

I am attaching a screen shot of the /tmp directory of the server.
no-xdebug-intmp.jpg
0
 

Author Comment

by:scmeeven
ID: 40361827
Fixed it. I had forgotten to add extension=xdebug.so under the Dynamic extensions section of php.ini, so the module was not loading automatically.

My next problem is with KCacheGrind. After installing it and running it on the cachegrind.out.pid file, I get the following error:
root@xxxxxxx [~]# kcachegrind cachegrind.out.13777
kcachegrind: cannot connect to X server

I am unsure how to proceed.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40362123
Kcachegrind is a GUI app so it is looking for a desktop environment. I usually just download thd files to my Windows box and use WinCacheGrind . You can also post the file here.
0
 

Author Comment

by:scmeeven
ID: 40391179
Sorry for the delay. My laptop went dead and I couldn't find time at work to update this thread.

@gr8gronzo, I found your suggestion to do code profiling very useful and will accept that as my solution.
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
OS Selection Menu 1 46
Wordpress Security 29 46
Is there a way to bind the ID to the text value of the search result? 28 51
Renaming with batch file 4 39
Build an array called $myWeek which will hold the array elements Today, Yesterday and then builds up the rest of the week by the name of the day going back 1 week.   (CODE) (CODE) Then you just need to pass your date to the function. If i…
This article discusses four methods for overlaying images in a container on a web page
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

792 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question