Defunt and zombie php processes causing high cpu load

Hi,

On a dedicated server with 16 GB RAM running a single Magento site, we have been constantly facing high cpu loads. Running top, we find several php and defunct php processes generating as high as 98% cpu loads.

But, here's the strange thing - when we identify the parent pid of the defunct process and then strace the parent pid, we get the message that there is no such process. Sometimes, we get the same results for the non-defunct php processes as well. I am attaching a screen shot of the results strace -p pid command.

And, all these calls are to Magento's index.php page, which is usually the case with any CMS site.

We have done a fair bit of tweaking over the past week to fix front end issues through Y!Slow and Google Page Speed Insight, optimize Apache by moving to Worker MPM, optimize MySQL by running the mysqltuner.pl script every 24 hours etc., but nothing has an impact on these defunct and php processes and the high cpu loads they generate.

Does anyone have an idea how to identify these processes and stop them from generating such high cpu loads?

Thanks in advance.
php-zombie-process.jpg
scmeevenAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gr8gonzoConsultantCommented:
First, your screenshot shows two commands that BOTH indicate the process 10721 doesn't exist.

Second, I would suggest running some code profiling:
http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/A_12179-PHP-Code-Profiling-and-Analyzing-Performance.html

That should help tell you exactly what the PHP processes are doing.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
GaryCommented:
Is this shared hosting?
Is it a busy site?
Try disabling any plugins, firstly disable all plugins and see if it occurs, if not then reenable them one by one.
0
scmeevenAuthor Commented:
@gr8gonzo, 10721 was actually marked as defunct in top, so I ran the first command to identify its parent id, which came back as 11253. The intriguing thing about these php processes is that I get the no such process error even if I run strace within seconds of seeing them in PuTTY (opened two sessions to do so). But, the high cpu load they generate and their impact on the server is unmistakable. I will check out your suggestion on code profiling.

@Gary, as I mentioned, this is a dedicated server. Yes, it's a pretty busy site, but probably in the dozens to hundred visitors per hour range.
0
Cloud Class® Course: Microsoft Exchange Server

The MCTS: Microsoft Exchange Server 2010 certification validates your skills in supporting the maintenance and administration of the Exchange servers in an enterprise environment. Learn everything you need to know with this course.

GaryCommented:
I would really start with going thru the plugins, are you using a cache on the server?
0
gr8gonzoConsultantCommented:
No intention of being negative, but dozens to hundred visitors per hour isn't a busy site, at least not in the sense that you should be having load issues. Even a hundred visitors per hour is slightly over one per minute on average, which means those processes have a pretty decent amount of elbowroom to spin up and shut down.

Just because some people have some different ideas of terminology - is it truly a completely-independent, physical dedicated server? Some people think that VPS is a dedicated server, or that virtualized servers (e.g. a virtual instance via KVM or other virtual manager) is considered a dedicated server, but any virtualized solution is still sharing resources on the backend. A fully-dedicated server is a physical machine that doesn't share any of its resources.

Gary's suggestions about plugins (I think Magento calls them extensions) is a good one. All it takes is one poorly-written plugin to turn a speedy site into a slug. The code profiling may help you identify which plugin is doing the damage, assuming you can replicate the problem by yourself.
0
gr8gonzoConsultantCommented:
Also, I can't recall if I included this in my article on code profiling, but don't leave XDebug enabled on the site once you're done. It's a heavy debugging extension that can have a significant performance impact, depending on the code.
0
scmeevenAuthor Commented:
@gr8gonzo, this is a genuine, physical, dedicated server.  And, thanks for the tip about disabling xdebug. I will update this question with the results of that test.
0
scmeevenAuthor Commented:
I have installed xdebug, added the correct path to the Zend extension in php.ini, set trigger to 1 and output dir to /tmp and restarted Apache.

Then I ran www.domain.com/index.php?XDEBUG_PROFILE=1 but there's no cachegrind.out file generated in the /tmp directory. Because CMSes like Magento call the index.php file, I then tried www.domain.com/?XDEBUG_PROFILE=1 but still with no resulting cachegrind.out file.

Is there something I am doing wrong?

I am attaching a screen shot of the /tmp directory of the server.
no-xdebug-intmp.jpg
0
scmeevenAuthor Commented:
Fixed it. I had forgotten to add extension=xdebug.so under the Dynamic extensions section of php.ini, so the module was not loading automatically.

My next problem is with KCacheGrind. After installing it and running it on the cachegrind.out.pid file, I get the following error:
root@xxxxxxx [~]# kcachegrind cachegrind.out.13777
kcachegrind: cannot connect to X server

I am unsure how to proceed.
0
gr8gonzoConsultantCommented:
Kcachegrind is a GUI app so it is looking for a desktop environment. I usually just download thd files to my Windows box and use WinCacheGrind . You can also post the file here.
0
scmeevenAuthor Commented:
Sorry for the delay. My laptop went dead and I couldn't find time at work to update this thread.

@gr8gronzo, I found your suggestion to do code profiling very useful and will accept that as my solution.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Scripting Languages

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.