Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17


Read/Write/Update a file at same time

Posted on 1997-12-23
Medium Priority
Last Modified: 2013-12-25
I am allowing users on my website to place their vote.  Their submissions go to my VOTE.CGI perl script.  It opens the VOTESUB.DAT file to add/update their vote tally.

Now if I have millions of users accessing VOTE.CGI and thus VOTESUB.DAT all at the same time, wouldn't VOTESUB.DAT get all screwed up?  I've got to hold back the user submissions and handle them one at a time so as to not screw up the numbers held with in it.  Is this true?  

If so I have thought about writing out a dummy file called HOLD.TMP everytime VOTE.CGI is started.  So, everytime VOTE.CGI is started it checks for the existence of HOLD.TMP.  If it exists then we know that another occurence of VOTE.CGI is still running right now.  I must 'pause'(sleep) until this file has dissapeared (deleted by that previous instance of VOTE.CGI once it completes itself).  When the file is gone immediately create a HOLD.TMP file (so as to pause any next VOTE.CGI instances) and now do the handling of the voter submission.

Is this the best method to handle multiple writes to a single file?  Or is there another method to pause all clients on a web page so as to do stuff like this?

One more thing: is it true that the only way to update a file (ie my VOTESUB.DAT) is to read it all in (or in parts) and then write it all out, even if all you wanted to do is add a single value at the half way point in the file?  So if So lets say VOTESUB.DAT looked like:
1: Hello Line 1
2: Hello Line 2
3: Hello Line 3

If I wanted to alter line 2 to read: Hello From Line 2
I MUST read in ALL of VOTESUB.DAT and write it back out again?!


Question by:mirror
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 3

Accepted Solution

alamo earned 80 total points
ID: 1831737
The first answer is: file locking- probably using flock(). (This was a FAQ here a couple of months ago).

Your hold.tmp idea is most people's first thought, but it won't work since two processes could write the file simultaneously. Much better to use actual file locking (though the standard "flock" method won't work on every system.

On the second question, you don't need to read and rewrite the entire file if the records are fixed length and you don't need to insert (just update). If you need inserts and have a lot of data, a dbm file (which is effectively a hash tied to a file) is a good way to do it. But i am assuming in your application your data file will never get much bigger than a few Kbytes, so I wouldn't bother with anything fancy since you wouldn't be saving much time and would be making the whole thing more complicated. So unless you use a DBM file the answer is yes, you have to read the entire thing in.

Here's all you need to do to update the file: (assumes the vote to be added is in $thisvote)

$LOCK_EX = 2; # Exclusive
open(DAT, "+<votesub.dat") || die "Can't open file: $!";
flock(DAT,$LOCK_EX) || die "Can't flock: $!";
# put a sleep(5); here to test locking, run script twice at same time
while (<DAT>) {
seek(DAT, 0, 0)  || die "Can't seek: $!";
foreach (keys %votearray) {
 print DAT "$_=$votearray{$_}\n";
 print "$_=$votearray{$_}\n";
close(DAT)   || die "Can't close: $!";

That's all you need... you do need to place a sleep there to verify flock works on your system - call the script multiple times and verify each waits for the one before it to finish and the data is correct, the 5 second sleep makes it easy to tell.

Good luck!

Expert Comment

ID: 1831738
oops, left in a debugging statement that echos the results to the screen:
print "$_=$votearray{$_}\n"

I tested it by adding $thisvote=$ARGV[0]; at the beginning and just calling it from the command line, this is a simple way to see how it works.


Author Comment

ID: 1831739
Once again, alamo.  Amazing.  I am off to try out what youi said.  Looks like I've got to change lots of code.

How to Create Failover DNS Record Sets in Route 53

Route 53 has the ability to easily configure DNS record sets specifically for failover scenarios. These failover record sets can be configured to failover to full-blown deployments in other regions or to a static HTML page that informs your customers of the issue.


Author Comment

ID: 1831740
Alamo, I have many more questions pertaining to DBM and FLOCK.  I can't seem to find decent info on them (everything book references another reference).  Please, can you email me directly so that I can talk to you.  I need answers quickly.

Author Comment

ID: 1831741
Alamo, what if I use my old method of locking my VOTESUB.DAT file but I handle itt this way.

When its ok, I write out the HOLD.TMP file as usual but inside this file I write the $$ (process Id #).  Then after writing this file, I read it in.  Is it my process Id #?  If not, continue waiting....If so continue on and handle the VOTESUB.DAT file.

Does this resolve the problem of writing the HOLD.TMP file at the same time?  

Merry Christmas.

Expert Comment

ID: 1831742
Your altered method is better but still not foolproof, since you can't completely ensure that someone won't come along and overwrite it after you verify the value has been written to it. Also, it's unclear whether what you read back is your own buffer or not.

flock is explained in the core perl documentation, it's pretty straughtforward. Are you saying the code above doesn't work on your system?

Author Comment

ID: 1831743
Here's my actual code I'm using Alamo.  Just before I actually use this CGI program, I call with SETLIGHT(DBASE,RED).  It checks if all is well then moves on in.  Is this not foolproof?  As you look at this I'll run off and check the FLOCK command in the perl docs.  Thanks for all your help....  And sorry for the messy code below.  It's probably better viewed if you copy the text and paste it in a text editor.  SOme of those comments seem to run pretty long.

#Check to see if we can handle data now
# All is ok now, handle program here

sub SetLight()

  if ($who eq "DBASE")   #We need to set a light on FTP.DAT

  if ($type eq "RED")  #Set an indicator to tell other FTP2.CGI                         that we're using FTP.DAT
  { my($tm)=1;
    while (-e "tmp/$fname")               #If we're not the first instance of FTP2.CGI, we must wait until the previous is done
    {$tm=($tm==1)?2:1; sleep($tm);}
    open (OUT,">tmp/$fname");             #The marker is off, quickly jump in by creating marker and becoming TOP priority.
    print OUT "$$"; close(OUT);           #Notice we write the PROCESS ID #.  A unique # for this process.

    open(IN,"tmp/$fname");                #We now verify that we wrote the HOLD file.  This prevents writes at same time to this file.  Verify PROCESS ID #
    $num=<IN>; close(IN);
    if ($num!=$$) {goto AGN;}             #If process ID isn't ours return back to the loop and wait
  else                                    #Green light. Kill any markers to allow other FTP2.CGI in


Author Comment

ID: 1831744
Any really quickly.  My database files which I created under DBM format (using the SDBM_FIle) are huge.  I am placing 5000 items in the one databse which has about 32 bytes per line.  I did some math and got around 160K.  But my DBM file is around 1.1 Megs.  Whats up with that and PLEASE assure me that this isn't all loaded into memory.  I really don't know much about servers but I'd hate for my website to be sucking all the servers memory and have me banned from my ISP.  (I left a more detailed message on the PERL section - HUGE DBM FILES)

Tanx again, man.

Featured Post

What is a Denial of Service (DoS)?

A DoS is a malicious attempt to prevent the normal operation of a computer system. You may frequently see the terms 'DDoS' (Distributed Denial of Service) and 'DoS' used interchangeably, but there are some subtle differences.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Recently I have been answering a lot of questions like this in IT forums that I frequent. The question posed is usually something along the lines of "We have software X installed and need to uninstall it for reason Y" or some other variant of the sa…
This article is meant to give a basic understanding of how to use R Sweave as a way to merge LaTeX and R code seamlessly into one presentable document.
Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…

722 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question