Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 235
  • Last Modified:

Run PHP script as daemon and auto scale

Run PHP script as daemon and auto scale

Have a PHP program to process events stored in database,
event table
id      description
1      event 1
2       event 2
3      event 3
...

process_event.php
<?php
do loop {
get an event from table
process the event
remove the event from table
}
?>

Want to run PHP program as daemon.
process_event.sh
#!/bin/bash
nice php process_event.php
 
Here it only runs a single worker.
Want to auto scale additional workers when needed.
For example, when table has 10 events, run 1 worker,
when table has 20 events, run 2 workers,
and automatically scale to run more or less workers according to number of events.
0
drupal_100
Asked:
drupal_100
  • 4
  • 3
  • 2
2 Solutions
 
gr8gonzoConsultantCommented:
So what is the question?
0
 
gheistCommented:
Method 1:
use apache-worker/event +  mod_fcgid + PHP
Method 2:
use apache-worker/event or nginx + php-fpm

Apache 2.2 -> fits 64 threads per process, fixed 2 user connections for one backend connection
Apache 2.4 -> fits 64 threads per proces, you can select the ration
NGINX -> 1024 threads per process, auto adjusts up to 1024 front connections per backend worker
0
 
drupal_100Author Commented:
$pid = pcntl_fork();
fork process to scale up.
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
drupal_100Author Commented:
I've requested that this question be closed as follows:

Accepted answer: 0 points for drupal_100's comment #a40483418

for the following reason:

Get answer
0
 
gr8gonzoConsultantCommented:
(I'm going to assume your PHP script already handles race conditions so that multiple workers don't re-reprocess the same events).

pnctl_fork by itself isn't a good way to "scale", for anyone who might come across this thread. It's actually trying to re-invent the wheel and there's more to it. You still have process management tasks if you go this route (just Google for pnctl_fork and defunct child processes for a simple example). Also, if the parent process crashes or gets killed, then your child processes go with it, and you might not know where the child was in a particular state, leaving you with a dirty environment.

This is why most people, including gheist, recommend using an existing service (normally a web server) for this kind of situation. This is literally what a web server is doing - it's forking off new processes and handling the process management for you, and also taking care of memory allocation issues and all the other small details that go into parent/child process management. You can even run just a private, internal web server - it doesn't have to be a major public deployment.

Now, creating child processes doesn't solve scaling issues, either. Let's say you start up 5 different child workers to process 50 events. Now let's say that each child eats up 50%-100% CPU while it is working. If you run them all simultaneously, each child will be less and less effective with every additional child worker. Adding child workers doesn't increase CPU or memory - they each take up more of those same limited resources.

If you want a model that truly scales, then you need to be able to have other pieces of hardware (additional CPU and memory) participate in the event processing. Again, web servers are a good way to do this. You can have a parent script that sits on one server and sends out "job requests" to different pieces of hardware that all process the jobs independently. That way, workers are not competing for the same physical resources.

A multi-server environment may be more than you need, but I'd still recommend using a web service, per gheist's recommendation, for kicking off child processes. PHP can even use multi-curl to asynchronously start the requests and wait for them to finish.
0
 
gheistCommented:
Your request does not concern programming, or manually forking process where you should be using a thread.

Both PHP-FPM and apache mod_fcgi exactly scales dynamically (untill 1024 parallel requests without tuning, that yields 20x that in users of site using it)
0
 
gheistCommented:
You need to explain better.
0
 
gr8gonzoConsultantCommented:
My recommendation: Split points:
http:#a40435577 (gheist) - 400 points
http:#a40483528 (gr8gonzo) - 100 points
0
 
gheistCommented:
I think both ways is acceptable like 50/50 I move threads out of process while you emphasize on hardships of thread programming.
0

Featured Post

Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

  • 4
  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now