The file test of "if (-e $filename)" takes a billion years to come back even when the file is there!

Posted on 2005-04-13
Last Modified: 2013-12-25
Ok, I'll expand a bit more...

I have a Linux server with an NFS mounted directory on a neighbouring DG/UX machine.

I have a script, that part-way through checks for the existence of a file, that is on the remote machine (e.g.)

$filename = "/dir1/dir2/filename.txt";

if (-e $filename)
# Set some variables here

This file will be put here by a BASIC program running on the other machine, and therefore I need to keep checking until I see it there (I know it will be there categorically), so I put the above in a while loop, setting a "found" variable when I see it, and sleeping for a second before retrying.

The problem is, that this takes about 20-30 seconds to see the file, even though I can see it there within about 2 seconds.  I can prove this by changing the "If (-e..." statement above to a simple "sleep(4);" before setting my variables, it works fine.

Now you're thinking "why not just do that then?" - because, depending on what information is being processed on the other machine, it might take longer than (x) seconds to write the file, and if I check too soon, my script (and resulting web page) will fail :-(

Two questions then, does anybody know why it takes so long - is it because the file is remotely mounted via NFS?  Also, how can I make it faster - is there a better method of testing the existence of a file, and maybe is there a better entire loop that would check really fast, and respond as soon as it finds the file?

Hope you can help - much hair being removed :-)

Question by:NTIVER
    LVL 18

    Accepted Solution

    I suspect it _is_ due to the nfs mount. Can you try the same thing locally, just to rule out it's the -e test?

    What are you going to do with the file once it's available?

    Maybe simply trying to open it might be quicker:

        if(open F, $file) {
    LVL 48

    Expert Comment

    A better way would be for the BASIC prog to write the file with a temporary name, and then rename it at the end.  That way you don't need to worry about if the file is incomplete.
    LVL 51

    Expert Comment

    sounds like a NFS caching problem, probably your nfsd on the remote site
    LVL 1

    Author Comment

    Hi all - thank you for your comments.

    OK, I'll try the if (open F, $file) syntax today to see if that works.

    My process is:

    User fills in HTML form with criteria, this gets submitted to a perl script - using the entered fields, this then writes a file to a remotely mounted directory on my other box.  Here it is picked up by my BASIC program (running from a polling routine), processed, and put back in the same directory.  My perl script then grabs it (once it exists!), and builds a web page of results in a new HTML page.  We're basically enabling people to enquire on our database from the web.

    Tintin: With regard to your comment - the basic program takes less than a second to process the data and write the file, and writes it with a unique name (e.g. 13579.txt) any way, so the same file will never exist twice.  The reason for this is so that two people can use the web page at the same time, and each one has their own unique identifier.

    I'll let you know how I get on with the open...

    Thanks for all your comments, and if you think of anything else in the mean time - let me know :-)

    LVL 1

    Author Comment

    Hi all.

    OK, I've changed it to be...


    That HAS made it faster, however it still isn't seeing the file as soon as it is placed there ready, it still takes a good few seconds to see it.

    One thing I have noticed thouigh, if while the script is waiting for the file to appear, I do an "ls -l" against the remote directory to refresh my list of files, the script comes back immediately after.  I'm thinking, is there a polling interval or similar in which remote NFS directories are refreshed?  If so, is it paramterised or can it be made faster?

    Many thanks all for your help - we're getting there :-)

    Of course, I could cheat and set a script running every second to do the "ls -l" command and pipe it through to /dev/null, but that's a bit rubbish.

    LVL 18

    Expert Comment

    Looks like you do indeed need to tune your nfs setup. I'm not much of a sysadmin though. Try one of the OS topic areas here, I'm sure the folks there can help you with that.
    LVL 1

    Author Comment

    Thank you for all who contributed - I'll post a further question RE the NFS problem in a more appropriate area, as although my situation is better than before, it's not as fast as I think it should be.


    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Highfive Gives IT Their Time Back

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    In this tutorial I will aim to show you how simple is making a small application in WhizBase, how to add, remove and update data in the DB. I will make a small address book application where you can add, browse, update and remove addresses. I wi…
    This article will show, step by step, how to integrate R code into a R Sweave document
    Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…
    This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

    760 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    9 Experts available now in Live!

    Get 1:1 Help Now