Windows File Locking Anomaly

Hello,

I've run into a little quirk in dealing with Windows file locking -- or to be more precise, an apparent misunderstanding on how to determine when a file is being written to and when it's not.

A call made to determine if a file arriving in an FTP drop-box is free and clear of being written to is apparently failing to indicate that the file under test is indeed still being written to.

I wrote a test script to open a file for write access, then write a byte every second to that file for 5 minutes. The code below was then executed in another process against the test-write file. The results were that the code shown below returned a correct indication that it could not get an exlcusive lock on the file being written to in the other process. (An expected result.)

However, the FLOCK routine fails to indicate that a file is being written to when an FTP process has a file open for writing. Apparently, my subroutine CAN get an exclusive lock on the file.

Additionally, the file is checked for the Read-Only attribute using a Win32::GetFileAttribute call. (And if read-Only, the program attempts to reset the read-only bit.)

What happens next is that the partially completed file is copied to a destination. When the attempt is made to delete the file, only then does the OS return an error:

"The process cannot access the file because it is being used by another process"

Well excuse me, but I thought the FLOCK would have hinted at that fact, but it apparently does not.

So the question is, how do you tell for certain that any given file is NOT being used by another process and can be fully accessed? (i.e. READ/WRITE/MODIFY/DELETE)


=========
Code synopsis:
=========

use Fcntl ':flock';
use IO::File;
...

-->Get a list of files arriving into a directory from FTP process...
-->Send each path+filename string to a validation function that checks we're only handling the right type of file -- ( placed into $pathname variable )
-->Before returning a file type value (1-9) check that we can get an exclusive lock on the file (ensure it's not presently being written to by FTP process.)
...

$Ftype = validate($pathname);

if($Ftype < 9){

...
copy($pathname, $destination);
...
delete($pathname);   <--- FAILS here (with message quoted above)

}

########### SUBS ###########

########
sub validate {
if( test_lock($pathname)){   # See if the file is locked for writing
   error($ERRLOG,"Locked file ->[ $pathname ]");
   $retval = 9;                    # retval = 9 tells caller to leave the file alone this cycle
}
# end sub validate()


#########
sub test_lock {
       
my $File                   = $_[0];
my $mode       = ( -e $File ) ? '<' : '>';    
my $fh             = new IO::File $mode.$File;
my $status       = flock($fh, LOCK_NB | LOCK_EX );
       
   if(0 == $status){      # Status will be zero if file is locked
      print DBG "\nFound locked File =>".$File if($DEBUG);
      return($TRUE);          #TRUE = File IS locked
   }
      
flock($fh, LOCK_UN);       # Release the lock - we don't need it
return($FALSE);               # $FALSE = File is NOT locked
                         
} # end sub test_lock
LVL 3
ReddgumAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
ahoffmannConnect With a Mentor Commented:
> .. I used Perl for the test script and Perl for the lock test.
and hence both used the same native API. Or did you tell the scripts to do something different?

I'm not shure that it makes sence to write a C or whatever prog too, there are so many things to test, you're probably wasting time.
I'll stick to my comments http:#12044594 and http:#12045068
means you need to know what your programs do.
0
 
bugecdysisCommented:
Perhaps a loop to re-copy/re-delete the file when `delete($pathname)' throws an error, until it succeeeds or optionally times out?

my $timeout = time + 30;
my $success = 0;
while (time < $timeout) {
...
copy($pathname, $destination);
...
$success++, last if delete($pathname);

delete($destination); # might need this
}

die "Failed to copy file\n" unless $success;
0
 
ahoffmannCommented:
does your windows support flock() ?
0
Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

 
holliCommented:
FLOCK will only work if the process that writes the file does use FLOCK too. ist that the case?
0
 
ahoffmannCommented:
> ..  will only work if the process that writes the file does use FLOCK too.
*any* process relying on locks needs to use flock() then ;-)
0
 
ReddgumAuthor Commented:
The thought had crossed my mind re: using a function that first copies, then attempts to delete a file, but it goes against my engineer's sense of right and wrong. It's not that I won't write hacks, but in this case, I feel strongly enough that if there isn't an OS-level method of appropriately checking for the file being busy, there damn well should be. Aside from the obvious straw-man rips on Winblows, I can't believe that the only way in Perl to check a file is to first attempt to delete it.

One would ask - how is the DELETE (in this case unlink($pathname) ) function getting the BUSY status back from the OS itself? Obviously it's failing, and it fails for the exact reason it should - another process is using the file. So how do I access this same functionality that unlink is accessing?

Surely there's a Perl guru somewhere that knows the answer to this.



0
 
ahoffmannCommented:
file locks are something bound to the filesystem, rather than the operating system
If all processes use the same libs to do the file locks, you can be shure which one it is.
But do all your processes use the same filesystem type to access the file? For example one process using NFS, the other SMB, things get realy complicated then ...
Still waiting for the answer to:  does your windows support flock() ?
0
 
holliCommented:
ahoffman,

from perlfunc:

Two potentially non-obvious but traditional flock semantics are that it waits indefinitely until the lock is granted, and that its locks merely advisory. Such discretionary locks are more flexible, but offer fewer guarantees. This means that files locked with flock may be modified by programs that do not also use flock.


redgum,

this is from perlport

flock FILEHANDLE,OPERATION

Not implemented (Mac OS, VMS, RISC OS, VOS).
Available only on Windows NT (not on Windows 95). (Win32)


So what is you Windows/FileSystem?
0
 
holliConnect With a Mentor Commented:
Im not so deep in the Win32-API but having a glance in the API-Reference you might use
LockFileEx-Function.

see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/lockfileex.asp
0
 
ReddgumAuthor Commented:
ahoffman & holli:

The OS is Windows 2000 Server Pro edition.  SP6 last time I looked. The OS doesn't support Unix calls per se, unless there's a library - and to be honest, I don't know if flock() is a native lib call or not.

ahoffman: Please note that I am using flock in the code snippet supplied. What gets me is that there does not appear to be an open method to extract information about files until you try to get destructive with them. I'm not convinced that such a massive oversight exists in Perl yet - I suspect it's just a semantics thing, or one of those less than obvious Perl-IO things that you must be told by someone who's a cousin of or the next door neighbor to Larry Wall.

FWIW: I've run into a similar item under windows. When using IIS on another server, I can read/write and modify a set of files anytime I wish. But when I attempt to delete them, the OS responds with the exact same message: can't access the file because another process blah blah blah. Even with IIS and it's cohorts shut off, the system refuses to allow the files to be deleted. While different in scope and a separate issue, they're connected by the same root cause.

Ideally:

use Win32:FileSystemSomethingorAnother

...

if( _isbusy($pathname)){
 
          wait_till_it's_not_busy_or_time_out();
 }



But noooooooo.....

0
 
ahoffmannCommented:
well, I'm not sitting next door to Larry, but as I said: each program needs to use flock() to make it work.
I'm not a wondoze guru, but IIS probably uses SMB locks on your NTFS. Hence each attempt to change the file fails with this error message as long as another process claims to have access to the file, doesn't matter if it is read or write access.
That's glory M$ solution, please don't blaim Larry for that.
Read my previous comment: the filesystem (and its driver) is the culprit here.

Keep in mind that you even might get this message if no other process has a lock on the file, 'cause the (SMB) driver caches information and is lazy to write it back to the phsical disk. Shame on ...
0
 
ReddgumAuthor Commented:
No, not blaming Larry here - oversights aside, it just seems to be an issue that a lot of Win32 folks would run into and would be something at least reasonably common enough that I sort of expected it to be an almost pat answer. It seems to be a bigger issue altogether based on what you've written.

Fortunately, my process has a drop-through fail. If it sees the file in the directory, it just keeps copying it, writing over the prior copy and attempts to delete it through each iteration. After 4 or 5 loops, the file is copied one last time and then successfully deletes it. This isn't what I'd call graceful though and it marks up the log file with the errors.

Thanks for the inputs.

0
 
ReddgumAuthor Commented:
DOH - It's Windows Server 2000 SP4.  For some reason I'm thinking NT4 with SP6 and "Pro" for the workstation.

Where's my coffee dammit!?
0
 
ahoffmannCommented:
anyway: NTFS
need a coffe  too ;-)
0
 
jpfxCommented:
I'm pretty certain flock has no effect on windows. I got around a similar problem by setting ftp sessions to time out after 299 seconds of inactivity and writing the script to work on files that were at least 300 seconds old.
0
 
ReddgumAuthor Commented:
Actually, flock() does appear to work. Like I mentioned in the description, I wrote a test script that opened a file and sent a byte to it every second for about 5 minutes. (To keep it open and busy.)

A second script called the function (published above) and returned a TRUE result each and every time while the file remained open for writing by the other script.

As you might appreciate, this is why it's bugging me :)

0
 
ahoffmannCommented:
> ..  I wrote a test script ..
listen, that's exactly what I said all the time: if both applications use the same lib it works, obviously.
But in your case you have different authors of your applications, and so most likely using different libs ...
0
 
ReddgumAuthor Commented:
ahoffman:

Hmm. I'll try doing things a little different then. The ONLY similarity between the two scripts were that both were Perl scripts. FLOCK() was **not** used in the test-write script. It simply opened a file and wrote to it in  a timed loop.

I can put something together in C and perform the same basic test if you think the results are tainted because I used Perl for the test script and Perl for the lock test.





0
 
ReddgumAuthor Commented:
Thanks for your help guys - we'll just have to chalk this one up as a want for the next iteration of the Windoze OS...

Cheers,
           -Red
0
 
ahoffmannCommented:
> ..  next iteration ..
LOL, the company developing windoze promises to make it all better "next" time, but I never have seen a date which defines "next". Good practice, hence this phrase "next time" is always true :-P
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.