Solved

Windows File Locking Anomaly

Posted on 2004-09-10
20
871 Views
Last Modified: 2008-01-09
Hello,

I've run into a little quirk in dealing with Windows file locking -- or to be more precise, an apparent misunderstanding on how to determine when a file is being written to and when it's not.

A call made to determine if a file arriving in an FTP drop-box is free and clear of being written to is apparently failing to indicate that the file under test is indeed still being written to.

I wrote a test script to open a file for write access, then write a byte every second to that file for 5 minutes. The code below was then executed in another process against the test-write file. The results were that the code shown below returned a correct indication that it could not get an exlcusive lock on the file being written to in the other process. (An expected result.)

However, the FLOCK routine fails to indicate that a file is being written to when an FTP process has a file open for writing. Apparently, my subroutine CAN get an exclusive lock on the file.

Additionally, the file is checked for the Read-Only attribute using a Win32::GetFileAttribute call. (And if read-Only, the program attempts to reset the read-only bit.)

What happens next is that the partially completed file is copied to a destination. When the attempt is made to delete the file, only then does the OS return an error:

"The process cannot access the file because it is being used by another process"

Well excuse me, but I thought the FLOCK would have hinted at that fact, but it apparently does not.

So the question is, how do you tell for certain that any given file is NOT being used by another process and can be fully accessed? (i.e. READ/WRITE/MODIFY/DELETE)


=========
Code synopsis:
=========

use Fcntl ':flock';
use IO::File;
...

-->Get a list of files arriving into a directory from FTP process...
-->Send each path+filename string to a validation function that checks we're only handling the right type of file -- ( placed into $pathname variable )
-->Before returning a file type value (1-9) check that we can get an exclusive lock on the file (ensure it's not presently being written to by FTP process.)
...

$Ftype = validate($pathname);

if($Ftype < 9){

...
copy($pathname, $destination);
...
delete($pathname);   <--- FAILS here (with message quoted above)

}

########### SUBS ###########

########
sub validate {
if( test_lock($pathname)){   # See if the file is locked for writing
   error($ERRLOG,"Locked file ->[ $pathname ]");
   $retval = 9;                    # retval = 9 tells caller to leave the file alone this cycle
}
# end sub validate()


#########
sub test_lock {
       
my $File                   = $_[0];
my $mode       = ( -e $File ) ? '<' : '>';    
my $fh             = new IO::File $mode.$File;
my $status       = flock($fh, LOCK_NB | LOCK_EX );
       
   if(0 == $status){      # Status will be zero if file is locked
      print DBG "\nFound locked File =>".$File if($DEBUG);
      return($TRUE);          #TRUE = File IS locked
   }
      
flock($fh, LOCK_UN);       # Release the lock - we don't need it
return($FALSE);               # $FALSE = File is NOT locked
                         
} # end sub test_lock
0
Comment
Question by:Reddgum
  • 8
  • 7
  • 3
  • +2
20 Comments
 
LVL 3

Expert Comment

by:bugecdysis
ID: 12035497
Perhaps a loop to re-copy/re-delete the file when `delete($pathname)' throws an error, until it succeeeds or optionally times out?

my $timeout = time + 30;
my $success = 0;
while (time < $timeout) {
...
copy($pathname, $destination);
...
$success++, last if delete($pathname);

delete($destination); # might need this
}

die "Failed to copy file\n" unless $success;
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12039835
does your windows support flock() ?
0
 
LVL 6

Expert Comment

by:holli
ID: 12042550
FLOCK will only work if the process that writes the file does use FLOCK too. ist that the case?
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12044136
> ..  will only work if the process that writes the file does use FLOCK too.
*any* process relying on locks needs to use flock() then ;-)
0
 
LVL 3

Author Comment

by:Reddgum
ID: 12044486
The thought had crossed my mind re: using a function that first copies, then attempts to delete a file, but it goes against my engineer's sense of right and wrong. It's not that I won't write hacks, but in this case, I feel strongly enough that if there isn't an OS-level method of appropriately checking for the file being busy, there damn well should be. Aside from the obvious straw-man rips on Winblows, I can't believe that the only way in Perl to check a file is to first attempt to delete it.

One would ask - how is the DELETE (in this case unlink($pathname) ) function getting the BUSY status back from the OS itself? Obviously it's failing, and it fails for the exact reason it should - another process is using the file. So how do I access this same functionality that unlink is accessing?

Surely there's a Perl guru somewhere that knows the answer to this.



0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12044594
file locks are something bound to the filesystem, rather than the operating system
If all processes use the same libs to do the file locks, you can be shure which one it is.
But do all your processes use the same filesystem type to access the file? For example one process using NFS, the other SMB, things get realy complicated then ...
Still waiting for the answer to:  does your windows support flock() ?
0
 
LVL 6

Expert Comment

by:holli
ID: 12044694
ahoffman,

from perlfunc:

Two potentially non-obvious but traditional flock semantics are that it waits indefinitely until the lock is granted, and that its locks merely advisory. Such discretionary locks are more flexible, but offer fewer guarantees. This means that files locked with flock may be modified by programs that do not also use flock.


redgum,

this is from perlport

flock FILEHANDLE,OPERATION

Not implemented (Mac OS, VMS, RISC OS, VOS).
Available only on Windows NT (not on Windows 95). (Win32)


So what is you Windows/FileSystem?
0
 
LVL 6

Assisted Solution

by:holli
holli earned 150 total points
ID: 12044776
Im not so deep in the Win32-API but having a glance in the API-Reference you might use
LockFileEx-Function.

see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/lockfileex.asp
0
 
LVL 3

Author Comment

by:Reddgum
ID: 12044950
ahoffman & holli:

The OS is Windows 2000 Server Pro edition.  SP6 last time I looked. The OS doesn't support Unix calls per se, unless there's a library - and to be honest, I don't know if flock() is a native lib call or not.

ahoffman: Please note that I am using flock in the code snippet supplied. What gets me is that there does not appear to be an open method to extract information about files until you try to get destructive with them. I'm not convinced that such a massive oversight exists in Perl yet - I suspect it's just a semantics thing, or one of those less than obvious Perl-IO things that you must be told by someone who's a cousin of or the next door neighbor to Larry Wall.

FWIW: I've run into a similar item under windows. When using IIS on another server, I can read/write and modify a set of files anytime I wish. But when I attempt to delete them, the OS responds with the exact same message: can't access the file because another process blah blah blah. Even with IIS and it's cohorts shut off, the system refuses to allow the files to be deleted. While different in scope and a separate issue, they're connected by the same root cause.

Ideally:

use Win32:FileSystemSomethingorAnother

...

if( _isbusy($pathname)){
 
          wait_till_it's_not_busy_or_time_out();
 }



But noooooooo.....

0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12045068
well, I'm not sitting next door to Larry, but as I said: each program needs to use flock() to make it work.
I'm not a wondoze guru, but IIS probably uses SMB locks on your NTFS. Hence each attempt to change the file fails with this error message as long as another process claims to have access to the file, doesn't matter if it is read or write access.
That's glory M$ solution, please don't blaim Larry for that.
Read my previous comment: the filesystem (and its driver) is the culprit here.

Keep in mind that you even might get this message if no other process has a lock on the file, 'cause the (SMB) driver caches information and is lazy to write it back to the phsical disk. Shame on ...
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 3

Author Comment

by:Reddgum
ID: 12045209
No, not blaming Larry here - oversights aside, it just seems to be an issue that a lot of Win32 folks would run into and would be something at least reasonably common enough that I sort of expected it to be an almost pat answer. It seems to be a bigger issue altogether based on what you've written.

Fortunately, my process has a drop-through fail. If it sees the file in the directory, it just keeps copying it, writing over the prior copy and attempts to delete it through each iteration. After 4 or 5 loops, the file is copied one last time and then successfully deletes it. This isn't what I'd call graceful though and it marks up the log file with the errors.

Thanks for the inputs.

0
 
LVL 3

Author Comment

by:Reddgum
ID: 12045843
DOH - It's Windows Server 2000 SP4.  For some reason I'm thinking NT4 with SP6 and "Pro" for the workstation.

Where's my coffee dammit!?
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12045888
anyway: NTFS
need a coffe  too ;-)
0
 
LVL 5

Expert Comment

by:jpfx
ID: 12057645
I'm pretty certain flock has no effect on windows. I got around a similar problem by setting ftp sessions to time out after 299 seconds of inactivity and writing the script to work on files that were at least 300 seconds old.
0
 
LVL 3

Author Comment

by:Reddgum
ID: 12057864
Actually, flock() does appear to work. Like I mentioned in the description, I wrote a test script that opened a file and sent a byte to it every second for about 5 minutes. (To keep it open and busy.)

A second script called the function (published above) and returned a TRUE result each and every time while the file remained open for writing by the other script.

As you might appreciate, this is why it's bugging me :)

0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12058510
> ..  I wrote a test script ..
listen, that's exactly what I said all the time: if both applications use the same lib it works, obviously.
But in your case you have different authors of your applications, and so most likely using different libs ...
0
 
LVL 3

Author Comment

by:Reddgum
ID: 12058594
ahoffman:

Hmm. I'll try doing things a little different then. The ONLY similarity between the two scripts were that both were Perl scripts. FLOCK() was **not** used in the test-write script. It simply opened a file and wrote to it in  a timed loop.

I can put something together in C and perform the same basic test if you think the results are tainted because I used Perl for the test script and Perl for the lock test.





0
 
LVL 51

Accepted Solution

by:
ahoffmann earned 350 total points
ID: 12059491
> .. I used Perl for the test script and Perl for the lock test.
and hence both used the same native API. Or did you tell the scripts to do something different?

I'm not shure that it makes sence to write a C or whatever prog too, there are so many things to test, you're probably wasting time.
I'll stick to my comments http:#12044594 and http:#12045068
means you need to know what your programs do.
0
 
LVL 3

Author Comment

by:Reddgum
ID: 12059803
Thanks for your help guys - we'll just have to chalk this one up as a want for the next iteration of the Windoze OS...

Cheers,
           -Red
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 12061879
> ..  next iteration ..
LOL, the company developing windoze promises to make it all better "next" time, but I never have seen a date which defines "next". Good practice, hence this phrase "next time" is always true :-P
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

On Microsoft Windows, if  when you click or type the name of a .pl file, you get an error "is not recognized as an internal or external command, operable program or batch file", then this means you do not have the .pl file extension associated with …
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

8 Experts available now in Live!

Get 1:1 Help Now