Go Premium for a chance to win a PS4. Enter to Win


Read/Write/Update a file at same time

Posted on 1997-12-23
Medium Priority
Last Modified: 2013-12-25
I am allowing users on my website to place their vote.  Their submissions go to my VOTE.CGI perl script.  It opens the VOTESUB.DAT file to add/update their vote tally.

Now if I have millions of users accessing VOTE.CGI and thus VOTESUB.DAT all at the same time, wouldn't VOTESUB.DAT get all screwed up?  I've got to hold back the user submissions and handle them one at a time so as to not screw up the numbers held with in it.  Is this true?  

If so I have thought about writing out a dummy file called HOLD.TMP everytime VOTE.CGI is started.  So, everytime VOTE.CGI is started it checks for the existence of HOLD.TMP.  If it exists then we know that another occurence of VOTE.CGI is still running right now.  I must 'pause'(sleep) until this file has dissapeared (deleted by that previous instance of VOTE.CGI once it completes itself).  When the file is gone immediately create a HOLD.TMP file (so as to pause any next VOTE.CGI instances) and now do the handling of the voter submission.

Is this the best method to handle multiple writes to a single file?  Or is there another method to pause all clients on a web page so as to do stuff like this?

One more thing: is it true that the only way to update a file (ie my VOTESUB.DAT) is to read it all in (or in parts) and then write it all out, even if all you wanted to do is add a single value at the half way point in the file?  So if So lets say VOTESUB.DAT looked like:
1: Hello Line 1
2: Hello Line 2
3: Hello Line 3

If I wanted to alter line 2 to read: Hello From Line 2
I MUST read in ALL of VOTESUB.DAT and write it back out again?!


Question by:mirror
  • 5
  • 3

Accepted Solution

alamo earned 80 total points
ID: 1831737
The first answer is: file locking- probably using flock(). (This was a FAQ here a couple of months ago).

Your hold.tmp idea is most people's first thought, but it won't work since two processes could write the file simultaneously. Much better to use actual file locking (though the standard "flock" method won't work on every system.

On the second question, you don't need to read and rewrite the entire file if the records are fixed length and you don't need to insert (just update). If you need inserts and have a lot of data, a dbm file (which is effectively a hash tied to a file) is a good way to do it. But i am assuming in your application your data file will never get much bigger than a few Kbytes, so I wouldn't bother with anything fancy since you wouldn't be saving much time and would be making the whole thing more complicated. So unless you use a DBM file the answer is yes, you have to read the entire thing in.

Here's all you need to do to update the file: (assumes the vote to be added is in $thisvote)

$LOCK_EX = 2; # Exclusive
open(DAT, "+<votesub.dat") || die "Can't open file: $!";
flock(DAT,$LOCK_EX) || die "Can't flock: $!";
# put a sleep(5); here to test locking, run script twice at same time
while (<DAT>) {
seek(DAT, 0, 0)  || die "Can't seek: $!";
foreach (keys %votearray) {
 print DAT "$_=$votearray{$_}\n";
 print "$_=$votearray{$_}\n";
close(DAT)   || die "Can't close: $!";

That's all you need... you do need to place a sleep there to verify flock works on your system - call the script multiple times and verify each waits for the one before it to finish and the data is correct, the 5 second sleep makes it easy to tell.

Good luck!

Expert Comment

ID: 1831738
oops, left in a debugging statement that echos the results to the screen:
print "$_=$votearray{$_}\n"

I tested it by adding $thisvote=$ARGV[0]; at the beginning and just calling it from the command line, this is a simple way to see how it works.


Author Comment

ID: 1831739
Once again, alamo.  Amazing.  I am off to try out what youi said.  Looks like I've got to change lots of code.

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.


Author Comment

ID: 1831740
Alamo, I have many more questions pertaining to DBM and FLOCK.  I can't seem to find decent info on them (everything book references another reference).  Please, can you email me directly so that I can talk to you.  I need answers quickly.


Author Comment

ID: 1831741
Alamo, what if I use my old method of locking my VOTESUB.DAT file but I handle itt this way.

When its ok, I write out the HOLD.TMP file as usual but inside this file I write the $$ (process Id #).  Then after writing this file, I read it in.  Is it my process Id #?  If not, continue waiting....If so continue on and handle the VOTESUB.DAT file.

Does this resolve the problem of writing the HOLD.TMP file at the same time?  

Merry Christmas.

Expert Comment

ID: 1831742
Your altered method is better but still not foolproof, since you can't completely ensure that someone won't come along and overwrite it after you verify the value has been written to it. Also, it's unclear whether what you read back is your own buffer or not.

flock is explained in the core perl documentation, it's pretty straughtforward. Are you saying the code above doesn't work on your system?

Author Comment

ID: 1831743
Here's my actual code I'm using Alamo.  Just before I actually use this CGI program, I call with SETLIGHT(DBASE,RED).  It checks if all is well then moves on in.  Is this not foolproof?  As you look at this I'll run off and check the FLOCK command in the perl docs.  Thanks for all your help....  And sorry for the messy code below.  It's probably better viewed if you copy the text and paste it in a text editor.  SOme of those comments seem to run pretty long.

#Check to see if we can handle data now
# All is ok now, handle program here

sub SetLight()

  if ($who eq "DBASE")   #We need to set a light on FTP.DAT

  if ($type eq "RED")  #Set an indicator to tell other FTP2.CGI                         that we're using FTP.DAT
  { my($tm)=1;
    while (-e "tmp/$fname")               #If we're not the first instance of FTP2.CGI, we must wait until the previous is done
    {$tm=($tm==1)?2:1; sleep($tm);}
    open (OUT,">tmp/$fname");             #The marker is off, quickly jump in by creating marker and becoming TOP priority.
    print OUT "$$"; close(OUT);           #Notice we write the PROCESS ID #.  A unique # for this process.

    open(IN,"tmp/$fname");                #We now verify that we wrote the HOLD file.  This prevents writes at same time to this file.  Verify PROCESS ID #
    $num=<IN>; close(IN);
    if ($num!=$$) {goto AGN;}             #If process ID isn't ours return back to the loop and wait
  else                                    #Green light. Kill any markers to allow other FTP2.CGI in


Author Comment

ID: 1831744
Any really quickly.  My database files which I created under DBM format (using the SDBM_FIle) are huge.  I am placing 5000 items in the one databse which has about 32 bytes per line.  I did some math and got around 160K.  But my DBM file is around 1.1 Megs.  Whats up with that and PLEASE assure me that this isn't all loaded into memory.  I really don't know much about servers but I'd hate for my website to be sucking all the servers memory and have me banned from my ISP.  (I left a more detailed message on the PERL section - HUGE DBM FILES)

Tanx again, man.

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In this tutorial I will focus on how to use WhizBase as a tool for sending ICQ messages to ICQ. Here I will use a new technology in WhizBase, published in WhizBase 5.1 version. In this tutorial I will use 3 files, pager.wbsp for the processing, e…
A quick Powershell script I wrote to find old program installations and check versions of a specific file across the network.
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…
The viewer will learn the basics of jQuery, including how to invoke it on a web page. Reference your jQuery libraries: (CODE) Include your new external js/jQuery file: (CODE) Write your first lines of code to setup your site for jQuery.: (CODE)
Suggested Courses

885 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question