Run PHP script as daemon and auto scale

Posted on 2014-11-10
Last Modified: 2014-12-16
Run PHP script as daemon and auto scale

Have a PHP program to process events stored in database,
event table
id      description
1      event 1
2       event 2
3      event 3

do loop {
get an event from table
process the event
remove the event from table

Want to run PHP program as daemon.
nice php process_event.php
Here it only runs a single worker.
Want to auto scale additional workers when needed.
For example, when table has 10 events, run 1 worker,
when table has 20 events, run 2 workers,
and automatically scale to run more or less workers according to number of events.
Question by:drupal_100
  • 4
  • 3
  • 2
LVL 34

Expert Comment

ID: 40435198
So what is the question?
LVL 61

Accepted Solution

gheist earned 250 total points
ID: 40435577
Method 1:
use apache-worker/event +  mod_fcgid + PHP
Method 2:
use apache-worker/event or nginx + php-fpm

Apache 2.2 -> fits 64 threads per process, fixed 2 user connections for one backend connection
Apache 2.4 -> fits 64 threads per proces, you can select the ration
NGINX -> 1024 threads per process, auto adjusts up to 1024 front connections per backend worker

Author Comment

ID: 40483418
$pid = pcntl_fork();
fork process to scale up.

Author Comment

ID: 40483747
I've requested that this question be closed as follows:

Accepted answer: 0 points for drupal_100's comment #a40483418

for the following reason:

Get answer
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

LVL 34

Assisted Solution

gr8gonzo earned 250 total points
ID: 40483528
(I'm going to assume your PHP script already handles race conditions so that multiple workers don't re-reprocess the same events).

pnctl_fork by itself isn't a good way to "scale", for anyone who might come across this thread. It's actually trying to re-invent the wheel and there's more to it. You still have process management tasks if you go this route (just Google for pnctl_fork and defunct child processes for a simple example). Also, if the parent process crashes or gets killed, then your child processes go with it, and you might not know where the child was in a particular state, leaving you with a dirty environment.

This is why most people, including gheist, recommend using an existing service (normally a web server) for this kind of situation. This is literally what a web server is doing - it's forking off new processes and handling the process management for you, and also taking care of memory allocation issues and all the other small details that go into parent/child process management. You can even run just a private, internal web server - it doesn't have to be a major public deployment.

Now, creating child processes doesn't solve scaling issues, either. Let's say you start up 5 different child workers to process 50 events. Now let's say that each child eats up 50%-100% CPU while it is working. If you run them all simultaneously, each child will be less and less effective with every additional child worker. Adding child workers doesn't increase CPU or memory - they each take up more of those same limited resources.

If you want a model that truly scales, then you need to be able to have other pieces of hardware (additional CPU and memory) participate in the event processing. Again, web servers are a good way to do this. You can have a parent script that sits on one server and sends out "job requests" to different pieces of hardware that all process the jobs independently. That way, workers are not competing for the same physical resources.

A multi-server environment may be more than you need, but I'd still recommend using a web service, per gheist's recommendation, for kicking off child processes. PHP can even use multi-curl to asynchronously start the requests and wait for them to finish.
LVL 61

Expert Comment

ID: 40483746
Your request does not concern programming, or manually forking process where you should be using a thread.

Both PHP-FPM and apache mod_fcgi exactly scales dynamically (untill 1024 parallel requests without tuning, that yields 20x that in users of site using it)
LVL 61

Expert Comment

ID: 40483748
You need to explain better.
LVL 34

Expert Comment

ID: 40496459
My recommendation: Split points:
http:#a40435577 (gheist) - 400 points
http:#a40483528 (gr8gonzo) - 100 points
LVL 61

Expert Comment

ID: 40496662
I think both ways is acceptable like 50/50 I move threads out of process while you emphasize on hardships of thread programming.

Featured Post

Complete VMware vSphere® ESX(i) & Hyper-V Backup

Capture your entire system, including the host, with patented disk imaging integrated with VMware VADP / Microsoft VSS and RCT. RTOs is as low as 15 seconds with Acronis Active Restore™. You can enjoy unlimited P2V/V2V migrations from any source (even from a different hypervisor)

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Developers of all skill levels should learn to use current best practices when developing websites. However many developers, new and old, fall into the trap of using deprecated features because this is what so many tutorials and books tell them to u…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

932 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now