Solved

Defunt and zombie php processes causing high cpu load

Posted on 2014-10-03
11
553 Views
Last Modified: 2016-03-02
Hi,

On a dedicated server with 16 GB RAM running a single Magento site, we have been constantly facing high cpu loads. Running top, we find several php and defunct php processes generating as high as 98% cpu loads.

But, here's the strange thing - when we identify the parent pid of the defunct process and then strace the parent pid, we get the message that there is no such process. Sometimes, we get the same results for the non-defunct php processes as well. I am attaching a screen shot of the results strace -p pid command.

And, all these calls are to Magento's index.php page, which is usually the case with any CMS site.

We have done a fair bit of tweaking over the past week to fix front end issues through Y!Slow and Google Page Speed Insight, optimize Apache by moving to Worker MPM, optimize MySQL by running the mysqltuner.pl script every 24 hours etc., but nothing has an impact on these defunct and php processes and the high cpu loads they generate.

Does anyone have an idea how to identify these processes and stop them from generating such high cpu loads?

Thanks in advance.
php-zombie-process.jpg
0
Comment
Question by:scmeeven
  • 5
  • 4
  • 2
11 Comments
 
LVL 34

Accepted Solution

by:
gr8gonzo earned 500 total points
ID: 40359497
First, your screenshot shows two commands that BOTH indicate the process 10721 doesn't exist.

Second, I would suggest running some code profiling:
http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/A_12179-PHP-Code-Profiling-and-Analyzing-Performance.html

That should help tell you exactly what the PHP processes are doing.
0
 
LVL 58

Expert Comment

by:Gary
ID: 40359570
Is this shared hosting?
Is it a busy site?
Try disabling any plugins, firstly disable all plugins and see if it occurs, if not then reenable them one by one.
0
 

Author Comment

by:scmeeven
ID: 40359953
@gr8gonzo, 10721 was actually marked as defunct in top, so I ran the first command to identify its parent id, which came back as 11253. The intriguing thing about these php processes is that I get the no such process error even if I run strace within seconds of seeing them in PuTTY (opened two sessions to do so). But, the high cpu load they generate and their impact on the server is unmistakable. I will check out your suggestion on code profiling.

@Gary, as I mentioned, this is a dedicated server. Yes, it's a pretty busy site, but probably in the dozens to hundred visitors per hour range.
0
DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

 
LVL 58

Expert Comment

by:Gary
ID: 40359976
I would really start with going thru the plugins, are you using a cache on the server?
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40360291
No intention of being negative, but dozens to hundred visitors per hour isn't a busy site, at least not in the sense that you should be having load issues. Even a hundred visitors per hour is slightly over one per minute on average, which means those processes have a pretty decent amount of elbowroom to spin up and shut down.

Just because some people have some different ideas of terminology - is it truly a completely-independent, physical dedicated server? Some people think that VPS is a dedicated server, or that virtualized servers (e.g. a virtual instance via KVM or other virtual manager) is considered a dedicated server, but any virtualized solution is still sharing resources on the backend. A fully-dedicated server is a physical machine that doesn't share any of its resources.

Gary's suggestions about plugins (I think Magento calls them extensions) is a good one. All it takes is one poorly-written plugin to turn a speedy site into a slug. The code profiling may help you identify which plugin is doing the damage, assuming you can replicate the problem by yourself.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40360295
Also, I can't recall if I included this in my article on code profiling, but don't leave XDebug enabled on the site once you're done. It's a heavy debugging extension that can have a significant performance impact, depending on the code.
0
 

Author Comment

by:scmeeven
ID: 40361522
@gr8gonzo, this is a genuine, physical, dedicated server.  And, thanks for the tip about disabling xdebug. I will update this question with the results of that test.
0
 

Author Comment

by:scmeeven
ID: 40361776
I have installed xdebug, added the correct path to the Zend extension in php.ini, set trigger to 1 and output dir to /tmp and restarted Apache.

Then I ran www.domain.com/index.php?XDEBUG_PROFILE=1 but there's no cachegrind.out file generated in the /tmp directory. Because CMSes like Magento call the index.php file, I then tried www.domain.com/?XDEBUG_PROFILE=1 but still with no resulting cachegrind.out file.

Is there something I am doing wrong?

I am attaching a screen shot of the /tmp directory of the server.
no-xdebug-intmp.jpg
0
 

Author Comment

by:scmeeven
ID: 40361827
Fixed it. I had forgotten to add extension=xdebug.so under the Dynamic extensions section of php.ini, so the module was not loading automatically.

My next problem is with KCacheGrind. After installing it and running it on the cachegrind.out.pid file, I get the following error:
root@xxxxxxx [~]# kcachegrind cachegrind.out.13777
kcachegrind: cannot connect to X server

I am unsure how to proceed.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40362123
Kcachegrind is a GUI app so it is looking for a desktop environment. I usually just download thd files to my Windows box and use WinCacheGrind . You can also post the file here.
0
 

Author Comment

by:scmeeven
ID: 40391179
Sorry for the delay. My laptop went dead and I couldn't find time at work to update this thread.

@gr8gronzo, I found your suggestion to do code profiling very useful and will accept that as my solution.
0

Featured Post

Gigs: Get Your Project Delivered by an Expert

Select from freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
designing in object programming 12 75
Blocking submission of a web-based form 6 20
comma true 6 32
Unwanted Popup 8 27
A publishing tool, a Version Control System, or a Collaboration Platform! These can be some of the defining words for the two very famous web-hosting Git repositories: Bitbucket and Github. Git is widely used amongst the programmers and developers f…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
This video teaches users how to migrate an existing Wordpress website to a new domain.
Wufoo.com provides powerful tools for surveying targeted groups, and utilizing data from completed surveys to find trends, discover areas of demand or customer expectation, and make business decisions on products or services.

785 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question