Run PHP script as daemon and auto scale

Posted on 2014-11-10
Medium Priority
Last Modified: 2014-12-16
Run PHP script as daemon and auto scale

Have a PHP program to process events stored in database,
event table
id      description
1      event 1
2       event 2
3      event 3

do loop {
get an event from table
process the event
remove the event from table

Want to run PHP program as daemon.
nice php process_event.php
Here it only runs a single worker.
Want to auto scale additional workers when needed.
For example, when table has 10 events, run 1 worker,
when table has 20 events, run 2 workers,
and automatically scale to run more or less workers according to number of events.
Question by:drupal_100
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
  • 2
LVL 35

Expert Comment

ID: 40435198
So what is the question?
LVL 62

Accepted Solution

gheist earned 1000 total points
ID: 40435577
Method 1:
use apache-worker/event +  mod_fcgid + PHP
Method 2:
use apache-worker/event or nginx + php-fpm

Apache 2.2 -> fits 64 threads per process, fixed 2 user connections for one backend connection
Apache 2.4 -> fits 64 threads per proces, you can select the ration
NGINX -> 1024 threads per process, auto adjusts up to 1024 front connections per backend worker

Author Comment

ID: 40483418
$pid = pcntl_fork();
fork process to scale up.
Create CentOS 7 Newton Packstack Running Keystone

A bug was filed against RDO for the installation of Keystone v3. This guide is designed to walk you through the configuration for using Keystone v3 with Packstack. You will accomplish this using various repos and the Answers file.


Author Comment

ID: 40483747
I've requested that this question be closed as follows:

Accepted answer: 0 points for drupal_100's comment #a40483418

for the following reason:

Get answer
LVL 35

Assisted Solution

gr8gonzo earned 1000 total points
ID: 40483528
(I'm going to assume your PHP script already handles race conditions so that multiple workers don't re-reprocess the same events).

pnctl_fork by itself isn't a good way to "scale", for anyone who might come across this thread. It's actually trying to re-invent the wheel and there's more to it. You still have process management tasks if you go this route (just Google for pnctl_fork and defunct child processes for a simple example). Also, if the parent process crashes or gets killed, then your child processes go with it, and you might not know where the child was in a particular state, leaving you with a dirty environment.

This is why most people, including gheist, recommend using an existing service (normally a web server) for this kind of situation. This is literally what a web server is doing - it's forking off new processes and handling the process management for you, and also taking care of memory allocation issues and all the other small details that go into parent/child process management. You can even run just a private, internal web server - it doesn't have to be a major public deployment.

Now, creating child processes doesn't solve scaling issues, either. Let's say you start up 5 different child workers to process 50 events. Now let's say that each child eats up 50%-100% CPU while it is working. If you run them all simultaneously, each child will be less and less effective with every additional child worker. Adding child workers doesn't increase CPU or memory - they each take up more of those same limited resources.

If you want a model that truly scales, then you need to be able to have other pieces of hardware (additional CPU and memory) participate in the event processing. Again, web servers are a good way to do this. You can have a parent script that sits on one server and sends out "job requests" to different pieces of hardware that all process the jobs independently. That way, workers are not competing for the same physical resources.

A multi-server environment may be more than you need, but I'd still recommend using a web service, per gheist's recommendation, for kicking off child processes. PHP can even use multi-curl to asynchronously start the requests and wait for them to finish.
LVL 62

Expert Comment

ID: 40483746
Your request does not concern programming, or manually forking process where you should be using a thread.

Both PHP-FPM and apache mod_fcgi exactly scales dynamically (untill 1024 parallel requests without tuning, that yields 20x that in users of site using it)
LVL 62

Expert Comment

ID: 40483748
You need to explain better.
LVL 35

Expert Comment

ID: 40496459
My recommendation: Split points:
http:#a40435577 (gheist) - 400 points
http:#a40483528 (gr8gonzo) - 100 points
LVL 62

Expert Comment

ID: 40496662
I think both ways is acceptable like 50/50 I move threads out of process while you emphasize on hardships of thread programming.

Featured Post

How Blockchain Is Impacting Every Industry

Blockchain expert Alex Tapscott talks to Acronis VP Frank Jablonski about this revolutionary technology and how it's making inroads into other industries and facets of everyday life.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Build an array called $myWeek which will hold the array elements Today, Yesterday and then builds up the rest of the week by the name of the day going back 1 week.   (CODE) (CODE) Then you just need to pass your date to the function. If i…
This article discusses how to implement server side field validation and display customized error messages to the client.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
Suggested Courses

801 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question