asked on

PHP subprocessing: popen and fread blocking

I have a server process running in PHP 5.2.x on Windows. It's purpose is to run subprocesses and monitor their status. It runs as a windows service. It calls popen() to start multiple subprocesses, keeping the file handlers of each to loop back later and read the output of each process and eventually returns the status back to the clients who started them.

All of this works well, but there is one issue I am having that I have abstracted here to the simplest example I could think to create. Fread() does not seem to be able to run with a popen() file pointer in a nonblocking mode. If a process script does not output anything for awhile, and you are in a loop fread()ing the handler of that process, the fread() will hang until it sees some output. So that works great if I make all my processes extremely verbose, chattering all the time, but if I don't want all that extra yapping then the server script hangs intermittently.

I have tried using set_stream_blocking to false, but it returns false (meaning that it could not set the mode on that type of stream). For examples, see the scripts attached. Server1.php launches process1.php, and times only the fread portion. In a nonblocking mode, the read times should always be 0 secs. If you substitute for process2.php, you see that a chatty process works fine (but is annoying and inefficient).

My best attempt so far is somewhat lile the 'server2.php' file below. By checking for the size of the output of the process (using fstat()) prior to running the fread(), I can make it not stall during regular processing. The issue seems to be at the end of the script, I think because the end of file does not count in the file size reported, therefore the fread() never gets the eof char, and the server loop never meets the feof() criteria.

I'm looking for a creative solution to make this work, I don't really want to rewrite this with proc_open (partially because I am not certain it will fix the problem), but I will if needed. Thanks for the help!

server1.php
<?
$buffer = '';
$fp = popen('process1.php', "r"); 
while(!feof($fp)) 
{ 
	$starttime = microtime(true);
	$buffer .= fread($fp, 2048); 
	$endtime = microtime(true);
	echo 'Time spent reading: '.round($endtime - $starttime,6)." seconds\n";
	sleep(1); // just so we only check the script every second for output
} 
echo 'Process reached EOF!\n';
fclose($fp);
?>
 
process1.php
<?
echo "Here's some output!\n";
for ($i=0;i<=10;$i++) {
	sleep(10); // long processing time
	echo "A status update!\n";
}
echo "Process is done!\n";
?>
 
process2.php
<?
echo "Here's some output!\n";
for ($i=0;i<=100;$i++) {
	sleep(1); // lots of frequent echo's here
	echo ".";
}
echo "Process is done!\n";
?>
 
 
server2.php
<?
$buffer = '';
$fp = popen('process1.php', "r"); 
while(!feof($fp)) 
{ 
	$stat = fstat($fp);
	if ($stat['size']) { // only run the read when there is something to read
		$starttime = microtime(true);
		$buffer .= fread($fp, 2048); 
		$endtime = microtime(true);
		echo 'Time spent reading: '.round($endtime - $starttime,6)." seconds\n";
	}
	sleep(1); // just so we only check the script every second for output
} 
echo 'Process closed EOF!\n';
fclose($fp);
?>

Open in new window

SOLUTION

Richard Quadling

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

SOLUTION

Richard Quadling

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

nbcit

ASKER

I see what you mean. As you would expect I am not thrilled about modifying the processes to add file based outputs.

By using fstat() on the file handler and it's 'size' value, I have been able to keep the polling from blocking, because I don't run the fread() unless the fstat() sees some script output. The problem I am having now is that when the process completes, I am not aware that the process closed, and size never gets bigger than 0, and therefore it loops forever.

Can you think of any way (without using feof()) to determine if the process has closed? Perhaps record the PID somehow and run a command line check to see if it is still running? If so, I could handle that in my loop logic and I am sure this would work without the file-based output, and without hitting any blocking.

ASKER CERTIFIED SOLUTION

nbcit

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

nbcit

ASKER

I have reincorporated these changes into my application, and fread() blocking is no longer a problem for me with these changes.

Hope this helps someone else!

Richard Quadling

I have a project which is completely hung on the non-blocking fread issue.

I'll be taking a long hard look at your code.

Thanks for the code and the points.

Richard Quadling

Well. I've got it working with your code.

I would lower the sleep from 1 second though. The buffer seems to be around 2048 bytes (but not exactly?).

This is read in a very small amount of time.

So, using ...

usleep(10000);

Provides a much faster response and leaves neither process hanging around.

My code with a lot of comments. My child process is just a LONG dir (The C:\cygwin directory is over 21,000 files).

<?php
echo PHP_VERSION, ' ', PHP_OS, ' ', PHP_SAPI, PHP_EOL;
echo 'INI:', php_ini_loaded_file(), PHP_EOL;
 
// Define the descriptors.
$a_Descriptors = array(0 => array('pipe', 'rnt'), 1 => array('pipe', 'wnt'), 2 => array('pipe', 'wnt'));
 
// Provide a place for the pipes.
$a_Pipes = array();
 
// Create the thread.
$r_Thread = proc_open("DIR C:\\cygwin /B /S /A /C /N /4 /OGEN", $a_Descriptors, $a_Pipes, Null, $_ENV);
 
// Can we change the size of the write buffer?
echo (stream_set_write_buffer($a_Pipes[1], 4096) == 0 ? 'Successfully' : 'Failed to'), ' set write buffer size', PHP_EOL;
 
// Display the current STDOUT meta data.
print_r(stream_get_meta_data($a_Pipes[1]));
 
// Try to change the blocking mode to non-blocking.
echo (stream_set_blocking($a_Pipes[1], False) ? 'Successfully' : 'Failed to'), ' set blocking mode to non-blocking', PHP_EOL;
 
// Display the current STDOUT meta data.
print_r(stream_get_meta_data($a_Pipes[1]));
 
// Can we change the size of the write buffer?
echo (stream_set_write_buffer($a_Pipes[1], 4096) == 0 ? 'Successfully' : 'Failed to'), ' set write buffer size', PHP_EOL;
 
// Empty buffer and status.
$s_Buffer = '';
 
// Get process status.
$a_ProcStatus = proc_get_status($r_Thread);
 
// Get data from child process whilst it is running..
while($a_ProcStatus['running'])
	{
	// Get pipe stats.
	$a_PipeStat   = fstat($a_Pipes[1]);
 
	//// GREEDY GATHERING.
 
	// If we have anything to get.
	while ($a_PipeStat['size'] > 0)
		{
		// Record a start time.
		$dt_StartTime = microtime(True);
 
		// Gather all we can.
		$s_PartBuffer = fread($a_Pipes[1], 8192);
 
		// Add it to the main buffer.
		$s_Buffer .= $s_PartBuffer;
 
		// Report how much and how long.
		echo strlen($s_PartBuffer), ' byte(s) read in ', number_format(microtime(True) - $dt_StartTime, 8), ' seconds', PHP_EOL;
 
		// Update the pipe status.
		$a_PipeStat = fstat($a_Pipes[1]);
		}
 
	// To get here, the buffer must be empty.
	echo 'Waiting', PHP_EOL;
	
 
	// Update the  process status.
	$a_ProcStatus = proc_get_status($r_Thread);
 
	// Wait for 0.01s before trying again.
	usleep(10000);
	}
 
// Close the child process.
proc_close($r_Thread);
echo 'Process closed!', PHP_EOL;
 
// Output the results.
echo PHP_EOL, 'BUFFER OUTPUT (', strlen($s_Buffer), 'bytes:', PHP_EOL, $s_Buffer;

Open in new window

nbcit

ASKER

RQuadling,

Yeah, the sleep is way long in that example. It was just that though - an example - in my true application there is no sleeping going on, just a series of loops for gathering status on multiple suprocesses, and an entire php-based web daemon for checking status of active jobs - part of a large application that handles both interactive and offline processing.

I'm glad it worked for you - and I'm very pleased I was able to find some kind of solution. I REALLY didn't want to rewrite it another way. Take care!