Solved

Read/Write/Update a file at same time

Posted on 1997-12-23
8
246 Views
Last Modified: 2013-12-25
I am allowing users on my website to place their vote.  Their submissions go to my VOTE.CGI perl script.  It opens the VOTESUB.DAT file to add/update their vote tally.

Now if I have millions of users accessing VOTE.CGI and thus VOTESUB.DAT all at the same time, wouldn't VOTESUB.DAT get all screwed up?  I've got to hold back the user submissions and handle them one at a time so as to not screw up the numbers held with in it.  Is this true?  

If so I have thought about writing out a dummy file called HOLD.TMP everytime VOTE.CGI is started.  So, everytime VOTE.CGI is started it checks for the existence of HOLD.TMP.  If it exists then we know that another occurence of VOTE.CGI is still running right now.  I must 'pause'(sleep) until this file has dissapeared (deleted by that previous instance of VOTE.CGI once it completes itself).  When the file is gone immediately create a HOLD.TMP file (so as to pause any next VOTE.CGI instances) and now do the handling of the voter submission.

Is this the best method to handle multiple writes to a single file?  Or is there another method to pause all clients on a web page so as to do stuff like this?

One more thing: is it true that the only way to update a file (ie my VOTESUB.DAT) is to read it all in (or in parts) and then write it all out, even if all you wanted to do is add a single value at the half way point in the file?  So if So lets say VOTESUB.DAT looked like:
------------
1: Hello Line 1
2: Hello Line 2
3: Hello Line 3
------------

If I wanted to alter line 2 to read: Hello From Line 2
I MUST read in ALL of VOTESUB.DAT and write it back out again?!

Tanx.


0
Comment
Question by:mirror
  • 5
  • 3
8 Comments
 
LVL 6

Accepted Solution

by:
alamo earned 20 total points
ID: 1831737
The first answer is: file locking- probably using flock(). (This was a FAQ here a couple of months ago).

Your hold.tmp idea is most people's first thought, but it won't work since two processes could write the file simultaneously. Much better to use actual file locking (though the standard "flock" method won't work on every system.

On the second question, you don't need to read and rewrite the entire file if the records are fixed length and you don't need to insert (just update). If you need inserts and have a lot of data, a dbm file (which is effectively a hash tied to a file) is a good way to do it. But i am assuming in your application your data file will never get much bigger than a few Kbytes, so I wouldn't bother with anything fancy since you wouldn't be saving much time and would be making the whole thing more complicated. So unless you use a DBM file the answer is yes, you have to read the entire thing in.

Here's all you need to do to update the file: (assumes the vote to be added is in $thisvote)

$LOCK_EX = 2; # Exclusive
open(DAT, "+<votesub.dat") || die "Can't open file: $!";
flock(DAT,$LOCK_EX) || die "Can't flock: $!";
# put a sleep(5); here to test locking, run script twice at same time
while (<DAT>) {
 chomp;
 ($option,$count)=split(/=/);
 $votearray{$option}=$count;
}
$votearray{$thisvote}++;
seek(DAT, 0, 0)  || die "Can't seek: $!";
foreach (keys %votearray) {
 print DAT "$_=$votearray{$_}\n";
 print "$_=$votearray{$_}\n";
}
close(DAT)   || die "Can't close: $!";

That's all you need... you do need to place a sleep there to verify flock works on your system - call the script multiple times and verify each waits for the one before it to finish and the data is correct, the 5 second sleep makes it easy to tell.

Good luck!
0
 
LVL 6

Expert Comment

by:alamo
ID: 1831738
oops, left in a debugging statement that echos the results to the screen:
print "$_=$votearray{$_}\n"

I tested it by adding $thisvote=$ARGV[0]; at the beginning and just calling it from the command line, this is a simple way to see how it works.

0
 

Author Comment

by:mirror
ID: 1831739
Once again, alamo.  Amazing.  I am off to try out what youi said.  Looks like I've got to change lots of code.

0
 

Author Comment

by:mirror
ID: 1831740
Alamo, I have many more questions pertaining to DBM and FLOCK.  I can't seem to find decent info on them (everything book references another reference).  Please, can you email me directly so that I can talk to you.  I need answers quickly.

ljaques@cscn.com
0
Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

 

Author Comment

by:mirror
ID: 1831741
Alamo, what if I use my old method of locking my VOTESUB.DAT file but I handle itt this way.

When its ok, I write out the HOLD.TMP file as usual but inside this file I write the $$ (process Id #).  Then after writing this file, I read it in.  Is it my process Id #?  If not, continue waiting....If so continue on and handle the VOTESUB.DAT file.

Does this resolve the problem of writing the HOLD.TMP file at the same time?  

Merry Christmas.
0
 
LVL 6

Expert Comment

by:alamo
ID: 1831742
Your altered method is better but still not foolproof, since you can't completely ensure that someone won't come along and overwrite it after you verify the value has been written to it. Also, it's unclear whether what you read back is your own buffer or not.

flock is explained in the core perl documentation, it's pretty straughtforward. Are you saying the code above doesn't work on your system?
0
 

Author Comment

by:mirror
ID: 1831743
Here's my actual code I'm using Alamo.  Just before I actually use this CGI program, I call with SETLIGHT(DBASE,RED).  It checks if all is well then moves on in.  Is this not foolproof?  As you look at this I'll run off and check the FLOCK command in the perl docs.  Thanks for all your help....  And sorry for the messy code below.  It's probably better viewed if you copy the text and paste it in a text editor.  SOme of those comments seem to run pretty long.

#Check to see if we can handle data now
SetLight(DBASE,RED);
# All is ok now, handle program here

sub SetLight()
{
  my($who,$type)=@_;
  my($fname,$num);

  if ($who eq "DBASE")   #We need to set a light on FTP.DAT
  {$fname="hold-db";$LIGHT1=($LIGHT1==1)?0:1;}

  if ($type eq "RED")  #Set an indicator to tell other FTP2.CGI                         that we're using FTP.DAT
  { my($tm)=1;
   AGN:
    while (-e "tmp/$fname")               #If we're not the first instance of FTP2.CGI, we must wait until the previous is done
    {$tm=($tm==1)?2:1; sleep($tm);}
    open (OUT,">tmp/$fname");             #The marker is off, quickly jump in by creating marker and becoming TOP priority.
    print OUT "$$"; close(OUT);           #Notice we write the PROCESS ID #.  A unique # for this process.

    open(IN,"tmp/$fname");                #We now verify that we wrote the HOLD file.  This prevents writes at same time to this file.  Verify PROCESS ID #
    $num=<IN>; close(IN);
    if ($num!=$$) {goto AGN;}             #If process ID isn't ours return back to the loop and wait
  }
  else                                    #Green light. Kill any markers to allow other FTP2.CGI in
  {unlink("tmp/$fname");}
}

0
 

Author Comment

by:mirror
ID: 1831744
Any really quickly.  My database files which I created under DBM format (using the SDBM_FIle) are huge.  I am placing 5000 items in the one databse which has about 32 bytes per line.  I did some math and got around 160K.  But my DBM file is around 1.1 Megs.  Whats up with that and PLEASE assure me that this isn't all loaded into memory.  I really don't know much about servers but I'd hate for my website to be sucking all the servers memory and have me banned from my ISP.  (I left a more detailed message on the PERL section - HUGE DBM FILES)

Tanx again, man.
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

This tutorial will discuss fancy secure registration forms, with AJAX technology support. In this article I assume you already know HTML and some JS. I will write the code using WhizBase Server Pages, so you need to know some basics in WBSP (you mig…
In this tutorial I will show you how to provide a dynamic RTF document on your website generated with data from your database. For this tutorial you will need Microsoft Word or WordPad, WhizBase and Microsoft Access. In this tutorial I will show …
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now