Solved

Defunt and zombie php processes causing high cpu load

Posted on 2014-10-03
11
521 Views
Last Modified: 2016-03-02
Hi,

On a dedicated server with 16 GB RAM running a single Magento site, we have been constantly facing high cpu loads. Running top, we find several php and defunct php processes generating as high as 98% cpu loads.

But, here's the strange thing - when we identify the parent pid of the defunct process and then strace the parent pid, we get the message that there is no such process. Sometimes, we get the same results for the non-defunct php processes as well. I am attaching a screen shot of the results strace -p pid command.

And, all these calls are to Magento's index.php page, which is usually the case with any CMS site.

We have done a fair bit of tweaking over the past week to fix front end issues through Y!Slow and Google Page Speed Insight, optimize Apache by moving to Worker MPM, optimize MySQL by running the mysqltuner.pl script every 24 hours etc., but nothing has an impact on these defunct and php processes and the high cpu loads they generate.

Does anyone have an idea how to identify these processes and stop them from generating such high cpu loads?

Thanks in advance.
php-zombie-process.jpg
0
Comment
Question by:scmeeven
  • 5
  • 4
  • 2
11 Comments
 
LVL 34

Accepted Solution

by:
gr8gonzo earned 500 total points
Comment Utility
First, your screenshot shows two commands that BOTH indicate the process 10721 doesn't exist.

Second, I would suggest running some code profiling:
http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/A_12179-PHP-Code-Profiling-and-Analyzing-Performance.html

That should help tell you exactly what the PHP processes are doing.
0
 
LVL 58

Expert Comment

by:Gary
Comment Utility
Is this shared hosting?
Is it a busy site?
Try disabling any plugins, firstly disable all plugins and see if it occurs, if not then reenable them one by one.
0
 

Author Comment

by:scmeeven
Comment Utility
@gr8gonzo, 10721 was actually marked as defunct in top, so I ran the first command to identify its parent id, which came back as 11253. The intriguing thing about these php processes is that I get the no such process error even if I run strace within seconds of seeing them in PuTTY (opened two sessions to do so). But, the high cpu load they generate and their impact on the server is unmistakable. I will check out your suggestion on code profiling.

@Gary, as I mentioned, this is a dedicated server. Yes, it's a pretty busy site, but probably in the dozens to hundred visitors per hour range.
0
 
LVL 58

Expert Comment

by:Gary
Comment Utility
I would really start with going thru the plugins, are you using a cache on the server?
0
 
LVL 34

Expert Comment

by:gr8gonzo
Comment Utility
No intention of being negative, but dozens to hundred visitors per hour isn't a busy site, at least not in the sense that you should be having load issues. Even a hundred visitors per hour is slightly over one per minute on average, which means those processes have a pretty decent amount of elbowroom to spin up and shut down.

Just because some people have some different ideas of terminology - is it truly a completely-independent, physical dedicated server? Some people think that VPS is a dedicated server, or that virtualized servers (e.g. a virtual instance via KVM or other virtual manager) is considered a dedicated server, but any virtualized solution is still sharing resources on the backend. A fully-dedicated server is a physical machine that doesn't share any of its resources.

Gary's suggestions about plugins (I think Magento calls them extensions) is a good one. All it takes is one poorly-written plugin to turn a speedy site into a slug. The code profiling may help you identify which plugin is doing the damage, assuming you can replicate the problem by yourself.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 34

Expert Comment

by:gr8gonzo
Comment Utility
Also, I can't recall if I included this in my article on code profiling, but don't leave XDebug enabled on the site once you're done. It's a heavy debugging extension that can have a significant performance impact, depending on the code.
0
 

Author Comment

by:scmeeven
Comment Utility
@gr8gonzo, this is a genuine, physical, dedicated server.  And, thanks for the tip about disabling xdebug. I will update this question with the results of that test.
0
 

Author Comment

by:scmeeven
Comment Utility
I have installed xdebug, added the correct path to the Zend extension in php.ini, set trigger to 1 and output dir to /tmp and restarted Apache.

Then I ran www.domain.com/index.php?XDEBUG_PROFILE=1 but there's no cachegrind.out file generated in the /tmp directory. Because CMSes like Magento call the index.php file, I then tried www.domain.com/?XDEBUG_PROFILE=1 but still with no resulting cachegrind.out file.

Is there something I am doing wrong?

I am attaching a screen shot of the /tmp directory of the server.
no-xdebug-intmp.jpg
0
 

Author Comment

by:scmeeven
Comment Utility
Fixed it. I had forgotten to add extension=xdebug.so under the Dynamic extensions section of php.ini, so the module was not loading automatically.

My next problem is with KCacheGrind. After installing it and running it on the cachegrind.out.pid file, I get the following error:
root@xxxxxxx [~]# kcachegrind cachegrind.out.13777
kcachegrind: cannot connect to X server

I am unsure how to proceed.
0
 
LVL 34

Expert Comment

by:gr8gonzo
Comment Utility
Kcachegrind is a GUI app so it is looking for a desktop environment. I usually just download thd files to my Windows box and use WinCacheGrind . You can also post the file here.
0
 

Author Comment

by:scmeeven
Comment Utility
Sorry for the delay. My laptop went dead and I couldn't find time at work to update this thread.

@gr8gronzo, I found your suggestion to do code profiling very useful and will accept that as my solution.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Suggested Solutions

Real-time is more about the business, not the technology. In day-to-day life, to make real-time decisions like buying or investing, business needs the latest information(e.g. Gold Rate/Stock Rate). Unlike traditional days, you need not wait for a fe…
These days socially coordinated efforts have turned into a critical requirement for enterprises.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
You have products, that come in variants and want to set different prices for them? Watch this micro tutorial that describes how to configure prices for Magento super attributes. Assigning simple products to configurable: We assigned simple products…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now