Solved

How do I make sure a file is processed before it is altered?

Posted on 2011-02-16
12
236 Views
Last Modified: 2012-06-27
A remote server sends data to me (POST). I have a PHP script (receive.php) that receives the data and store it in a file. Then, a totally different script (process.php) reads that file, process its content and empty the file (this way it will prepare the file for the next write and it won't process the same data twice).

But the remote server may send data to my receive.php script while process.php script is still processing the file. How do I solve this problem?
0
Comment
Question by:cubic77
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
12 Comments
 
LVL 27

Expert Comment

by:Lukasz Chmielewski
ID: 34905897
0
 
LVL 4

Expert Comment

by:florjan
ID: 34905921
Php file operators. So check if file is not null. If it isnt write to other files. You could have a for sentence for less code. If you need anymore help I'll help you as soon as I come home.
0
 
LVL 34

Expert Comment

by:Beverley Portlock
ID: 34905956
A queue is your best bet.

Instead of allowing the scripts to access the file directly, get them to write into a table which holds the file's name and the next operation as well as any data. A separate script can then be run on a schedule to inspect the queue and process any work

-- UNTESTED --

CREATE TABLE processQueue (

  id INT NOT NULL auto_increment,
  filename VARCHAR(255) NOT NULL,
  command VARCHAR(32) NOT NULL,
  data TEXT NOT NULL,
  PRIMARY KEY(id)  

);


Then INSERT INTO processQueue ( filename, command, data ) VALUES (........ etc.....);

Have your scheduled script pull off the oldest record

SELECT * FROM processQueue ORDER BY filename, id LIMIT 1

apply the changes and then remove the work record

DELETE FROM processQueue where id = $id

This ensures that all operations on a file are performed sequentially. Have the job run every few minutes and if the queue is empty then just terminate the job. If you know that the changes will take fraction of a second then you might decide to process up to 10 commands or whatever works for you before you run out of processor time.

0
Get MongoDB database support online, now!

At Percona’s web store you can order your MongoDB database support needs in minutes. No hassles, no fuss, just pick and click. Pay online with a credit card. Handle your MongoDB database support now!

 
LVL 8

Expert Comment

by:rationalboss
ID: 34906867
Using a database will be helpful. But if you don't want to use a database, you can try the following.

Since you might still want to allow your application to receive updates even though there is still a file being processed, just allow the page to continue receiving requests. (that means, not locking the file)

My idea is to create a temporary directory that contains files you want processed. For example, you can create a queue/ directory. Each time a file is received, you put a file in that directory with a random file name. You can use file_put_contents() or move_uploaded_file() or simply rename(), you weren't able to mention how you receive files in receive.php.


Now, the other file which processes these files, loops to check for files in that directory:
<?php
$dir = "queue/";
if (is_dir($dir)) {
    if ($dh = opendir($dir)) {
        while (($file = readdir($dh)) !== false && filetype($dir . $file) == 'file') {
            // this is a file in the queue directory. Now, do something with $dir.$file
            // and make sure you delete the file after to indicate you're done processing it:
            // unlink($dir.$file);
        }
        closedir($dh);
    }
}
?>

... and delete the file after processing it. This way, you can accept multiple files for processing and have it processed sequentially and you don't need to use a database. The readdir() function would return the files in the order in which they are created in the file system.
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 34907173
This sounds like a strange application design.  Regarding this, "a totally different script (process.php) reads that file" -- how does that script get started?  And why would you want to re-use the file name?  Why not have the receive.php script collect the data, write the file, and then start the process.php script?  You would want to use a different file name (maybe using a timestamp) for each received data set.  Once process.php has completed its work, it could unlink() the file.
0
 

Author Comment

by:cubic77
ID: 34907873
@Ray_Paseur

> Why not have the receive.php script collect the data, write the file, and then start the process.php script?
Actually this is what I am doing right now.
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 34908374
If you're doing it that way and you use a different file name for each new file, you should be on firm ground, at least with this end of the application.
0
 

Author Comment

by:cubic77
ID: 34915112
>If you're doing it that way and you use a different file name for each new file, you should be on firm ground
Indeed it may be the best approach. Also, this way I can remove the need of using a data base to store my data. I can read the data directly from those files.
0
 
LVL 110

Expert Comment

by:Ray Paseur
ID: 34915456
That may be true.  Not sure what kind of data you have, but larger "blobs" of information like images or video are better in the file system, whereas smaller, succinct, searchable pieces seem to be most useful in a data base.  It's part art and part science ;)
0
 

Accepted Solution

by:
cubic77 earned 0 total points
ID: 34915495
I totally agree with databases. However, not storing the data in a DB has an important advantage to me. I can move the application (PHP script) and associated data to any server I want in less than 60 seconds. No DB setup nightmare.
0
 

Author Closing Comment

by:cubic77
ID: 36123171
no other solid solution provided
0

Featured Post

Will your db performance match your db growth?

In Percona’s white paper “Performance at Scale: Keeping Your Database on Its Toes,” we take a high-level approach to what you need to think about when planning for database scalability.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This article was originally published on Monitis Blog, you can check it here . Today it’s fairly well known that high-performing websites and applications bring in more visitors, higher SEO, and ultimately more sales. By the same token, downtime…
When it comes to security, close monitoring is a must. According to WhiteHat Security annual report, a substantial number of all web applications are vulnerable always. Monitis offers a new product - fully-featured Website security monitoring and pr…
The viewer will learn how to dynamically set the form action using jQuery.
Learn how to create flexible layouts using relative units in CSS.  New relative units added in CSS3 include vw(viewports width), vh(viewports height), vmin(minimum of viewports height and width), and vmax (maximum of viewports height and width).

630 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question