Solved

Run PHP script as daemon and auto scale

Posted on 2014-11-10
11
186 Views
Last Modified: 2014-12-16
Run PHP script as daemon and auto scale

Have a PHP program to process events stored in database,
event table
id      description
1      event 1
2       event 2
3      event 3
...

process_event.php
<?php
do loop {
get an event from table
process the event
remove the event from table
}
?>

Want to run PHP program as daemon.
process_event.sh
#!/bin/bash
nice php process_event.php
 
Here it only runs a single worker.
Want to auto scale additional workers when needed.
For example, when table has 10 events, run 1 worker,
when table has 20 events, run 2 workers,
and automatically scale to run more or less workers according to number of events.
0
Comment
Question by:drupal_100
  • 4
  • 3
  • 2
11 Comments
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40435198
So what is the question?
0
 
LVL 61

Accepted Solution

by:
gheist earned 250 total points
ID: 40435577
Method 1:
use apache-worker/event +  mod_fcgid + PHP
Method 2:
use apache-worker/event or nginx + php-fpm

Apache 2.2 -> fits 64 threads per process, fixed 2 user connections for one backend connection
Apache 2.4 -> fits 64 threads per proces, you can select the ration
NGINX -> 1024 threads per process, auto adjusts up to 1024 front connections per backend worker
0
 

Author Comment

by:drupal_100
ID: 40483418
$pid = pcntl_fork();
fork process to scale up.
0
 

Author Comment

by:drupal_100
ID: 40483747
I've requested that this question be closed as follows:

Accepted answer: 0 points for drupal_100's comment #a40483418

for the following reason:

Get answer
0
Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

 
LVL 34

Assisted Solution

by:gr8gonzo
gr8gonzo earned 250 total points
ID: 40483528
(I'm going to assume your PHP script already handles race conditions so that multiple workers don't re-reprocess the same events).

pnctl_fork by itself isn't a good way to "scale", for anyone who might come across this thread. It's actually trying to re-invent the wheel and there's more to it. You still have process management tasks if you go this route (just Google for pnctl_fork and defunct child processes for a simple example). Also, if the parent process crashes or gets killed, then your child processes go with it, and you might not know where the child was in a particular state, leaving you with a dirty environment.

This is why most people, including gheist, recommend using an existing service (normally a web server) for this kind of situation. This is literally what a web server is doing - it's forking off new processes and handling the process management for you, and also taking care of memory allocation issues and all the other small details that go into parent/child process management. You can even run just a private, internal web server - it doesn't have to be a major public deployment.

Now, creating child processes doesn't solve scaling issues, either. Let's say you start up 5 different child workers to process 50 events. Now let's say that each child eats up 50%-100% CPU while it is working. If you run them all simultaneously, each child will be less and less effective with every additional child worker. Adding child workers doesn't increase CPU or memory - they each take up more of those same limited resources.

If you want a model that truly scales, then you need to be able to have other pieces of hardware (additional CPU and memory) participate in the event processing. Again, web servers are a good way to do this. You can have a parent script that sits on one server and sends out "job requests" to different pieces of hardware that all process the jobs independently. That way, workers are not competing for the same physical resources.

A multi-server environment may be more than you need, but I'd still recommend using a web service, per gheist's recommendation, for kicking off child processes. PHP can even use multi-curl to asynchronously start the requests and wait for them to finish.
0
 
LVL 61

Expert Comment

by:gheist
ID: 40483746
Your request does not concern programming, or manually forking process where you should be using a thread.

Both PHP-FPM and apache mod_fcgi exactly scales dynamically (untill 1024 parallel requests without tuning, that yields 20x that in users of site using it)
0
 
LVL 61

Expert Comment

by:gheist
ID: 40483748
You need to explain better.
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40496459
My recommendation: Split points:
http:#a40435577 (gheist) - 400 points
http:#a40483528 (gr8gonzo) - 100 points
0
 
LVL 61

Expert Comment

by:gheist
ID: 40496662
I think both ways is acceptable like 50/50 I move threads out of process while you emphasize on hardships of thread programming.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

SSH (Secure Shell) - Tips and Tricks As you all know SSH(Secure Shell) is a network protocol, which we use to access/transfer files securely between two networked devices. SSH was actually designed as a replacement for insecure protocols that sen…
This article discusses four methods for overlaying images in a container on a web page
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now