Solved

Defunt and zombie php processes causing high cpu load

Posted on 2014-10-03
11
540 Views
Last Modified: 2016-03-02
Hi,

On a dedicated server with 16 GB RAM running a single Magento site, we have been constantly facing high cpu loads. Running top, we find several php and defunct php processes generating as high as 98% cpu loads.

But, here's the strange thing - when we identify the parent pid of the defunct process and then strace the parent pid, we get the message that there is no such process. Sometimes, we get the same results for the non-defunct php processes as well. I am attaching a screen shot of the results strace -p pid command.

And, all these calls are to Magento's index.php page, which is usually the case with any CMS site.

We have done a fair bit of tweaking over the past week to fix front end issues through Y!Slow and Google Page Speed Insight, optimize Apache by moving to Worker MPM, optimize MySQL by running the mysqltuner.pl script every 24 hours etc., but nothing has an impact on these defunct and php processes and the high cpu loads they generate.

Does anyone have an idea how to identify these processes and stop them from generating such high cpu loads?

Thanks in advance.
php-zombie-process.jpg
0
Comment
Question by:scmeeven
  • 5
  • 4
  • 2
11 Comments
 
LVL 34

Accepted Solution

by:
gr8gonzo earned 500 total points
ID: 40359497
First, your screenshot shows two commands that BOTH indicate the process 10721 doesn't exist.

Second, I would suggest running some code profiling:
http://www.experts-exchange.com/Programming/Languages/Scripting/PHP/A_12179-PHP-Code-Profiling-and-Analyzing-Performance.html

That should help tell you exactly what the PHP processes are doing.
0
 
LVL 58

Expert Comment

by:Gary
ID: 40359570
Is this shared hosting?
Is it a busy site?
Try disabling any plugins, firstly disable all plugins and see if it occurs, if not then reenable them one by one.
0
 

Author Comment

by:scmeeven
ID: 40359953
@gr8gonzo, 10721 was actually marked as defunct in top, so I ran the first command to identify its parent id, which came back as 11253. The intriguing thing about these php processes is that I get the no such process error even if I run strace within seconds of seeing them in PuTTY (opened two sessions to do so). But, the high cpu load they generate and their impact on the server is unmistakable. I will check out your suggestion on code profiling.

@Gary, as I mentioned, this is a dedicated server. Yes, it's a pretty busy site, but probably in the dozens to hundred visitors per hour range.
0
 
LVL 58

Expert Comment

by:Gary
ID: 40359976
I would really start with going thru the plugins, are you using a cache on the server?
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40360291
No intention of being negative, but dozens to hundred visitors per hour isn't a busy site, at least not in the sense that you should be having load issues. Even a hundred visitors per hour is slightly over one per minute on average, which means those processes have a pretty decent amount of elbowroom to spin up and shut down.

Just because some people have some different ideas of terminology - is it truly a completely-independent, physical dedicated server? Some people think that VPS is a dedicated server, or that virtualized servers (e.g. a virtual instance via KVM or other virtual manager) is considered a dedicated server, but any virtualized solution is still sharing resources on the backend. A fully-dedicated server is a physical machine that doesn't share any of its resources.

Gary's suggestions about plugins (I think Magento calls them extensions) is a good one. All it takes is one poorly-written plugin to turn a speedy site into a slug. The code profiling may help you identify which plugin is doing the damage, assuming you can replicate the problem by yourself.
0
DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40360295
Also, I can't recall if I included this in my article on code profiling, but don't leave XDebug enabled on the site once you're done. It's a heavy debugging extension that can have a significant performance impact, depending on the code.
0
 

Author Comment

by:scmeeven
ID: 40361522
@gr8gonzo, this is a genuine, physical, dedicated server.  And, thanks for the tip about disabling xdebug. I will update this question with the results of that test.
0
 

Author Comment

by:scmeeven
ID: 40361776
I have installed xdebug, added the correct path to the Zend extension in php.ini, set trigger to 1 and output dir to /tmp and restarted Apache.

Then I ran www.domain.com/index.php?XDEBUG_PROFILE=1 but there's no cachegrind.out file generated in the /tmp directory. Because CMSes like Magento call the index.php file, I then tried www.domain.com/?XDEBUG_PROFILE=1 but still with no resulting cachegrind.out file.

Is there something I am doing wrong?

I am attaching a screen shot of the /tmp directory of the server.
no-xdebug-intmp.jpg
0
 

Author Comment

by:scmeeven
ID: 40361827
Fixed it. I had forgotten to add extension=xdebug.so under the Dynamic extensions section of php.ini, so the module was not loading automatically.

My next problem is with KCacheGrind. After installing it and running it on the cachegrind.out.pid file, I get the following error:
root@xxxxxxx [~]# kcachegrind cachegrind.out.13777
kcachegrind: cannot connect to X server

I am unsure how to proceed.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40362123
Kcachegrind is a GUI app so it is looking for a desktop environment. I usually just download thd files to my Windows box and use WinCacheGrind . You can also post the file here.
0
 

Author Comment

by:scmeeven
ID: 40391179
Sorry for the delay. My laptop went dead and I couldn't find time at work to update this thread.

@gr8gronzo, I found your suggestion to do code profiling very useful and will accept that as my solution.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
php date - compare 4 40
How to use md5 hashing 3 26
Could you possibly point a workaround on this jQuery/ plugin datatable error? 5 41
is this a cms? 8 34
Build an array called $myWeek which will hold the array elements Today, Yesterday and then builds up the rest of the week by the name of the day going back 1 week.   (CODE) (CODE) Then you just need to pass your date to the function. If i…
This article will show, step by step, how to integrate R code into a R Sweave document
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…
In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…

911 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now