Solved

php sleep until file has been fully written

Posted on 2014-04-13
8
2,028 Views
Last Modified: 2016-09-14
how do you wait until a file has been fully written by an external process in php?

i'm trying to use this
function sleepUntilWritten($filename){
	while(true){
		$filesize_old = filesize($filename);
		sleep(4);
		$filesize_new = filesize($filename);
		if($filesize_old ==$filesize_new){
			echo 'done writing<br>';
			echo 'new = '.$filesize_new;
			echo 'old = '.$filesize_old;
			break;
			
		}
	}
}

Open in new window


but it breaks before the file has been fully written. i've also tried this:

function waitUntilFileWritten($filename){
	while(filectime($filename) >= time() - 2 ){
		sleep(1);
	}
}

Open in new window


but neither are getting the job done. i'm guessing the problem is more fundamental.
0
Comment
Question by:ChipmunkRumbleStud
  • 2
  • 2
  • 2
  • +2
8 Comments
 
LVL 82

Expert Comment

by:Dave Baldwin
ID: 39997572
The only way that the operating system knows that a program is done writing to a file is when that program closes the file handle.  I don't think PHP can 'see' that a file handle is open or closed.
0
 
LVL 34

Accepted Solution

by:
gr8gonzo earned 500 total points
ID: 39999020
Dave is partially correct - you'll have a hard time finding a way to query whether or not a file is opened by a separate application, but you can use flock() with LOCK_EX to try and obtain an exclusive writing lock on the file. If another app has the file open, then this would likely fail:

function check_is_still_writing($file)
{
  $is_still_writing = true;
  if($fp = fopen($file, "r+"))
  {
    if (flock($fp, LOCK_EX)) // Try to get an exclusive lock on the file
    {  
      flock($fp, LOCK_UN); // Unlock - we're just checking
      $is_still_writing = false;
    }
    fclose($fp);
  }
  return $is_still_writing;
}

Open in new window


The only problem with this is that if you have an application that does not keep the file handle open while writing, you could get a false positive. For example:

<?php
file_put_contents("file.txt", "blah blah");
sleep(10);
file_put_contents("file.txt", "blah blah", FILE_APPEND);
sleep(10);
file_put_contents("file.txt", "blah blah", FILE_APPEND);
?>

Open in new window


Each file_put_contents call opens the file handle, writes the data, and closes it. So the above script would take 20 seconds before it writes the final time, but it only has the file handle open for a few milliseconds in order to write out the blah blahs.

This can be common with logging, and because of that methodology, it becomes impossible to know when the writing has been finished because each "entry" is technically a completed/finished write. In that situation, you can do one of two things:

1. Use a much longer sleep() value to check the filesize differences (10-20 seconds instead of 4). Depending on what's writing to the file, there's usually some good number for a delay. Make SURE you use clearstatcache() to clear the filesystem info cache before checking the filesize again, since filesize is a cached function call:

http://www.php.net/manual/en/function.clearstatcache.php

It's possible even the four-second sleep() you had in your initial code is valid but the second filesize() call used the cached information because you didn't clear the filesystem info cache.

2. If you're exclusively on Linux, you can use shell_exec to run lsof on the various process IDs to try and discover what files they have open and try to find the one that is writing to that file, and then sleep until the process ends. Again, if the app opens/closes the file handle, this could be tricky to find the right process. If this PHP code is exceuted via a web server, you may need sudo to run lsof with the proper privileges.
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 40003860
PHP is not designed for monitoring an external process.  You would want the external process to monitor itself and send a signal to a PHP script to tell it when the external process completes.  You might follow the example of PayPal Instant Payment Notification, which uses a POST-method request to start an "IPN" script on your server whenever a payment completes.  The IPN script has access to both the payment information and the information on your server, and it can be used to make data base entries, start other processes, etc.

If you're new to PHP and want to get a good foundation, this article can help lead you in the right directions (and away from the wrong directions).
http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/A_11769-And-by-the-way-I-am-new-to-PHP.html
0
 

Author Comment

by:ChipmunkRumbleStud
ID: 40008294
@Ray Paseur so when you say 'sends a signal to php', what kind of signal is that and how do i implement it? I am salling the sound exchange binary to concatenate wave files and I want to make sure they are concatenated and the files are done before sending them to the client. If I enter

exec(sox one.wav two.wav oneandtwo.wav);
fopen(oneandtwo.wav);

Open in new window


will it work? The reason why I'm not testing it is because I'm having trouble calling binaries from php, even though 'dirs' works in the command line
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 108

Expert Comment

by:Ray Paseur
ID: 40008581
In computer science, "external process" is a term of art.  Now that I see your code example, I see that you've got an inline process using exec().  I think you probably want to try some tests just as you have written above, and experiment some.  I have always deliberately avoided exec() and I'm not conversant on how sox works.  What you'll be looking for is some affirmative signal that sox is finished and has created the file.  It may be as simple as trying to open the file, or it may take a little more work.

I'll sign off now, best of luck with your project, ~Ray
0
 
LVL 34

Expert Comment

by:gr8gonzo
ID: 40008735
If you're running PHP from the web, then PHP ends up inheriting the often-LOWER privileges that the web server uses. So PHP ends up getting "crippled" and cannot access many binaries with exec/shell_exec/etc... The only way around this is to either elevate Apache's own permissions (not usually a good idea), or use sudo. There are a lot of guides on the internet on how to appropriately install a sudo plugin for your web server. Assuming your web server is Apache, just google for "php apache sudo" (without the quotes) and you should find at least one resource that matches your setup.

That should allow you to run sox from PHP, and assuming it runs synchronously (meaning that it will run until it finishes), then that's all you need. If it runs asynchronously (meaning it starts and then indicates that it's finished so that PHP moves on in the code but a background process continues to run to do the work), then you'll need to monitor the file.

As far as monitoring the file goes, please try the example I provided using flock.
0
 

Author Closing Comment

by:ChipmunkRumbleStud
ID: 40033387
I was actually trying to use a file that had been created by and external binary called by php's exec function. The php script waits until the process is finished though, so this ended up not being a problem
0
 

Expert Comment

by:Aaron Lowe
ID: 41798323
You could store the date/time the file was last modified, do the operation, then sleep in a loop until that date/time changes.
Then you know the file has been changed and is fully closed and ready for reading.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

These days socially coordinated efforts have turned into a critical requirement for enterprises.
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now