Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 2424
  • Last Modified:

PHP subprocessing: popen and fread blocking

I have a server process running in PHP 5.2.x on Windows.  It's purpose is to run subprocesses and monitor their status.  It runs as a windows service.  It calls popen() to start multiple subprocesses, keeping the file handlers of each to loop back later and read the output of each process and eventually returns the status back to the clients who started them.

All of this works well, but there is one issue I am having that I have abstracted here to the simplest example I could think to create.  Fread() does not seem to be able to run with a popen() file pointer in a nonblocking mode.  If a process script does not output anything for awhile, and you are in a loop fread()ing the handler of that process, the fread() will hang until it sees some output.  So that works great if I make all my processes extremely verbose, chattering all the time, but if I don't want all that extra yapping then the server script hangs intermittently.

I have tried using set_stream_blocking to false, but it returns false (meaning that it could not set the mode on that type of stream).  For examples, see the scripts attached.  Server1.php launches process1.php, and times only the fread portion.  In a nonblocking mode, the read times should always be 0 secs.    If you substitute for process2.php, you see that a chatty process works fine (but is annoying and inefficient).

My best attempt so far is somewhat lile the 'server2.php' file below.  By checking for the size of the output of the process (using fstat()) prior to running the fread(), I can make it not stall during regular processing.  The issue seems to be at the end of the script, I think because the end of file does not count in the file size reported, therefore the fread() never gets the eof char, and the server loop never meets the feof() criteria.

I'm looking for a creative solution to make this work, I don't really want to rewrite this with proc_open (partially because I am not certain it will fix the problem), but I will if needed.  Thanks for the help!

server1.php
<?
$buffer = '';
$fp = popen('process1.php', "r"); 
while(!feof($fp)) 
{ 
	$starttime = microtime(true);
	$buffer .= fread($fp, 2048); 
	$endtime = microtime(true);
	echo 'Time spent reading: '.round($endtime - $starttime,6)." seconds\n";
	sleep(1); // just so we only check the script every second for output
} 
echo 'Process reached EOF!\n';
fclose($fp);
?>
 
process1.php
<?
echo "Here's some output!\n";
for ($i=0;i<=10;$i++) {
	sleep(10); // long processing time
	echo "A status update!\n";
}
echo "Process is done!\n";
?>
 
process2.php
<?
echo "Here's some output!\n";
for ($i=0;i<=100;$i++) {
	sleep(1); // lots of frequent echo's here
	echo ".";
}
echo "Process is done!\n";
?>
 
 
server2.php
<?
$buffer = '';
$fp = popen('process1.php', "r"); 
while(!feof($fp)) 
{ 
	$stat = fstat($fp);
	if ($stat['size']) { // only run the read when there is something to read
		$starttime = microtime(true);
		$buffer .= fread($fp, 2048); 
		$endtime = microtime(true);
		echo 'Time spent reading: '.round($endtime - $starttime,6)." seconds\n";
	}
	sleep(1); // just so we only check the script every second for output
} 
echo 'Process closed EOF!\n';
fclose($fp);
?>

Open in new window

0
nbcit
Asked:
nbcit
  • 4
  • 4
3 Solutions
 
Richard QuadlingSenior Software DeverloperCommented:
Yep. Its a bug. PHP on Windows does NOT support non-blocking file i/o.

http://bugs.php.net/bug.php?id=47918

In looking at the source, blocking is simply bypassed for Windows.

Unfortunately, you can't use the 'n' parameter in fopen() as this is not supported by the Windows C runtime. Instead, you have to use completely different file i/o.


PHP is not the only product suffering from this issue.

The common solution is to use threads, but this would need to be in the C code for PHP. And whilst it is a "common" solution, it is not the right one.

But I'm not good enough at C coding to get this fixed and so the bug remains.

 
0
 
Richard QuadlingSenior Software DeverloperCommented:
The "creative" solution you are looking for is to create a comms folder per "thread".

This way, the thread code can open/write/close a status file which can be read by the launching code.

It is a LOT slower and you have to deal with locking on the file. But if you keep the time low on the open/write/close, and cache the datetime of the last modification, then you do have a fairly workable solution.


0
 
nbcitAuthor Commented:
I see what you mean.  As you would expect I am not thrilled about modifying the processes to add file based outputs.

By using fstat() on the file handler and it's 'size' value, I have been able to keep the polling from blocking, because I don't run the fread() unless the fstat() sees some script output.  The problem I am having now is that when the process completes, I am not aware that the process closed, and size never gets bigger than 0, and therefore it loops forever.

Can you think of any way (without using feof()) to determine if the process has closed?  Perhaps record the PID somehow and run a command line check to see if it is still running?  If so, I could handle that in my loop logic and I am sure this would work without the file-based output, and without hitting any blocking.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
nbcitAuthor Commented:
So I think I found a solution.

I could not find a reliable way to determine if the process created by popen() had finished properly.  However, if I switched to proc_open(), then I could use proc_get_status(), and it has a value called 'running' which gives me what I need to make my script run in a nonblocking manner (by using fstat() for 'size' of the output).

So here is my little script, which will only run the fread() if fstat() determines there is output, and quits the loop when the proc_get_status() says that the process has completed, but only after one final check for script output.

Let me know what you think... It certainly is easier (and should be much faster) than a file-based process communication.
server.php
<?
$fp = proc_open('process1.php', array(1 => array("pipe", "w")), $pipes); 
while(!@$bufferdone) { 
	$proc = proc_get_status($fp);
	$stat = fstat($pipes[1]);
	if ($stat['size']) { // only run the read when there is something to read
		$starttime = microtime(true);
		@$buffer .= fread($pipes[1], $stat['size']); 
		echo $stat['size']." bytes read in ".round(microtime(true) - $starttime,2)." seconds\n";
	} else echo "Skipped - nothing to read\n";
	if (!$proc['running']) $bufferdone = 1;
	sleep(1); // just so we only check the script every second for output
}
echo "Process closed!\n";
proc_close($fp);
echo "\nBUFFER OUTPUT (".strlen($buffer)." bytes):\n".$buffer;
?>
 
 
process1.php
<?
echo "Here's some output!\n";
for ($i=0;$i<=3;$i++) {
	sleep(10); // long processing time
	echo "A status update!\n";
}
echo "Process is done!\n";
?>

Open in new window

1
 
nbcitAuthor Commented:
I have reincorporated these changes into my application, and fread() blocking is no longer a problem for me with these changes.

Hope this helps someone else!
0
 
Richard QuadlingSenior Software DeverloperCommented:
I have a project which is completely hung on the non-blocking fread issue.

I'll be taking a long hard look at your code.

Thanks for the code and the points.
0
 
Richard QuadlingSenior Software DeverloperCommented:
Well. I've got it working with your code.

I would lower the sleep from 1 second though. The buffer seems to be around 2048 bytes (but not exactly?).

This is read in a very small amount of time.

So, using ...

usleep(10000);

Provides a much faster response and leaves neither process hanging around.


My code with a lot of comments. My child process is just a LONG dir (The C:\cygwin directory is over 21,000 files).
<?php
echo PHP_VERSION, ' ', PHP_OS, ' ', PHP_SAPI, PHP_EOL;
echo 'INI:', php_ini_loaded_file(), PHP_EOL;
 
// Define the descriptors.
$a_Descriptors = array(0 => array('pipe', 'rnt'), 1 => array('pipe', 'wnt'), 2 => array('pipe', 'wnt'));
 
// Provide a place for the pipes.
$a_Pipes = array();
 
// Create the thread.
$r_Thread = proc_open("DIR C:\\cygwin /B /S /A /C /N /4 /OGEN", $a_Descriptors, $a_Pipes, Null, $_ENV);
 
// Can we change the size of the write buffer?
echo (stream_set_write_buffer($a_Pipes[1], 4096) == 0 ? 'Successfully' : 'Failed to'), ' set write buffer size', PHP_EOL;
 
// Display the current STDOUT meta data.
print_r(stream_get_meta_data($a_Pipes[1]));
 
// Try to change the blocking mode to non-blocking.
echo (stream_set_blocking($a_Pipes[1], False) ? 'Successfully' : 'Failed to'), ' set blocking mode to non-blocking', PHP_EOL;
 
// Display the current STDOUT meta data.
print_r(stream_get_meta_data($a_Pipes[1]));
 
// Can we change the size of the write buffer?
echo (stream_set_write_buffer($a_Pipes[1], 4096) == 0 ? 'Successfully' : 'Failed to'), ' set write buffer size', PHP_EOL;
 
// Empty buffer and status.
$s_Buffer = '';
 
// Get process status.
$a_ProcStatus = proc_get_status($r_Thread);
 
// Get data from child process whilst it is running..
while($a_ProcStatus['running'])
	{
	// Get pipe stats.
	$a_PipeStat   = fstat($a_Pipes[1]);
 
	//// GREEDY GATHERING.
 
	// If we have anything to get.
	while ($a_PipeStat['size'] > 0)
		{
		// Record a start time.
		$dt_StartTime = microtime(True);
 
		// Gather all we can.
		$s_PartBuffer = fread($a_Pipes[1], 8192);
 
		// Add it to the main buffer.
		$s_Buffer .= $s_PartBuffer;
 
		// Report how much and how long.
		echo strlen($s_PartBuffer), ' byte(s) read in ', number_format(microtime(True) - $dt_StartTime, 8), ' seconds', PHP_EOL;
 
		// Update the pipe status.
		$a_PipeStat = fstat($a_Pipes[1]);
		}
 
	// To get here, the buffer must be empty.
	echo 'Waiting', PHP_EOL;
	
 
	// Update the  process status.
	$a_ProcStatus = proc_get_status($r_Thread);
 
	// Wait for 0.01s before trying again.
	usleep(10000);
	}
 
// Close the child process.
proc_close($r_Thread);
echo 'Process closed!', PHP_EOL;
 
// Output the results.
echo PHP_EOL, 'BUFFER OUTPUT (', strlen($s_Buffer), 'bytes:', PHP_EOL, $s_Buffer;

Open in new window

0
 
nbcitAuthor Commented:
RQuadling,

Yeah, the sleep is way long in that example.  It was just that though - an example - in my true application there is no sleeping going on, just a series of loops for gathering status on multiple suprocesses, and an entire php-based web daemon for checking status of active jobs - part of a large application that handles both interactive and offline processing.

I'm glad it worked for you - and I'm very pleased I was able to find some kind of solution.  I REALLY didn't want to rewrite it another way.  Take care!
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 4
  • 4
Tackle projects and never again get stuck behind a technical roadblock.
Join Now