Run PHP script as daemon and auto scale

Posted on 2014-11-10
Last Modified: 2014-12-16
Run PHP script as daemon and auto scale

Have a PHP program to process events stored in database,
event table
id      description
1      event 1
2       event 2
3      event 3

do loop {
get an event from table
process the event
remove the event from table

Want to run PHP program as daemon.
nice php process_event.php
Here it only runs a single worker.
Want to auto scale additional workers when needed.
For example, when table has 10 events, run 1 worker,
when table has 20 events, run 2 workers,
and automatically scale to run more or less workers according to number of events.
Question by:drupal_100
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 4
  • 3
  • 2
LVL 35

Expert Comment

ID: 40435198
So what is the question?
LVL 62

Accepted Solution

gheist earned 250 total points
ID: 40435577
Method 1:
use apache-worker/event +  mod_fcgid + PHP
Method 2:
use apache-worker/event or nginx + php-fpm

Apache 2.2 -> fits 64 threads per process, fixed 2 user connections for one backend connection
Apache 2.4 -> fits 64 threads per proces, you can select the ration
NGINX -> 1024 threads per process, auto adjusts up to 1024 front connections per backend worker

Author Comment

ID: 40483418
$pid = pcntl_fork();
fork process to scale up.
Stressed Out?

Watch some penguins on the livecam!


Author Comment

ID: 40483747
I've requested that this question be closed as follows:

Accepted answer: 0 points for drupal_100's comment #a40483418

for the following reason:

Get answer
LVL 35

Assisted Solution

gr8gonzo earned 250 total points
ID: 40483528
(I'm going to assume your PHP script already handles race conditions so that multiple workers don't re-reprocess the same events).

pnctl_fork by itself isn't a good way to "scale", for anyone who might come across this thread. It's actually trying to re-invent the wheel and there's more to it. You still have process management tasks if you go this route (just Google for pnctl_fork and defunct child processes for a simple example). Also, if the parent process crashes or gets killed, then your child processes go with it, and you might not know where the child was in a particular state, leaving you with a dirty environment.

This is why most people, including gheist, recommend using an existing service (normally a web server) for this kind of situation. This is literally what a web server is doing - it's forking off new processes and handling the process management for you, and also taking care of memory allocation issues and all the other small details that go into parent/child process management. You can even run just a private, internal web server - it doesn't have to be a major public deployment.

Now, creating child processes doesn't solve scaling issues, either. Let's say you start up 5 different child workers to process 50 events. Now let's say that each child eats up 50%-100% CPU while it is working. If you run them all simultaneously, each child will be less and less effective with every additional child worker. Adding child workers doesn't increase CPU or memory - they each take up more of those same limited resources.

If you want a model that truly scales, then you need to be able to have other pieces of hardware (additional CPU and memory) participate in the event processing. Again, web servers are a good way to do this. You can have a parent script that sits on one server and sends out "job requests" to different pieces of hardware that all process the jobs independently. That way, workers are not competing for the same physical resources.

A multi-server environment may be more than you need, but I'd still recommend using a web service, per gheist's recommendation, for kicking off child processes. PHP can even use multi-curl to asynchronously start the requests and wait for them to finish.
LVL 62

Expert Comment

ID: 40483746
Your request does not concern programming, or manually forking process where you should be using a thread.

Both PHP-FPM and apache mod_fcgi exactly scales dynamically (untill 1024 parallel requests without tuning, that yields 20x that in users of site using it)
LVL 62

Expert Comment

ID: 40483748
You need to explain better.
LVL 35

Expert Comment

ID: 40496459
My recommendation: Split points:
http:#a40435577 (gheist) - 400 points
http:#a40483528 (gr8gonzo) - 100 points
LVL 62

Expert Comment

ID: 40496662
I think both ways is acceptable like 50/50 I move threads out of process while you emphasize on hardships of thread programming.

Featured Post

Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension ( This reminded me of questions tha…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

724 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question