Solved

How to be sure, file is written completely

Posted on 2004-10-24
275 Views
Last Modified: 2010-04-21
Hi!

I do a check, if a file exists "test -f <file>".
When it exists I do some processing ...

- but how can I be sure, that the file ist completely written, the moment I start the process?

thx
michi
0
Question by:witty
    10 Comments
     
    LVL 11

    Expert Comment

    by:avizit
    if your system supports the "lsof" command then it can give you the list if open files , which may be sufficient for your purpose that if a file is not open at the moment it is as good as being completely written for you

    but that still doesnt gurantee if a file is completely written .. files especially the log files etc can get written any time depending of how some program runs and the errors it comes across ( just to give you an example )
    0
     
    LVL 20

    Accepted Solution

    by:
    `fuser filename` does the same thing as lsof in this situation, but there is still the problem that the writing process may not have the file open at that moment...

    The usual method is to have the writing process create a temp file and rename it when it is complete,
    e.g.  proc > filename.tmp && mv filename.tmp filename.end

    Or create a flag file that the reading process looks for before it tries to read the data file, e.g.
    ftp myserver
    put filename
    put flagfile
    quit
    0
     
    LVL 60

    Expert Comment

    by:gheist
    some systems have same command under name fstat
    0
     
    LVL 5

    Expert Comment

    by:ITcrow
    If your script have write access on this file, try to create write lock on the file.
    If you can lock it, it's not open for write anywhere else.

    If the other process, which is generating this file is yours as well, then
    - Create Lock ( touch /tmp/.lock_logfile )
    - Generate file ( ls -R  > /tmp/logfile )
    - Release the lock ( /bin/rm /tmp/.lock_logfile )

    0
     
    LVL 51

    Expert Comment

    by:ahoffmann
    >  but how can I be sure, that the file ist completely written
    do mean to perform this check from within a shell script?
    If so, it's not that easy. You need to know the filesystem and its configuration (cacheing in particular)
    If you're on NFS too, then somthing like ITcrow suggest might be nbecessary too
    0
     
    LVL 84

    Expert Comment

    by:ozo
    Your process and the processes which may write to the file should agree on some protocol for determining when the writing is complete.
    Or, you might mv the file to a private place to ensure that it can no longer be written to.
    0
     
    LVL 1

    Author Comment

    by:witty
    ok - I see there is no general solution for this problem!

    I think the best way would be, telling the application to rename the file after finishing writing from a temporary name to the final name!
    - but I can't use this solution, because I can't change the application.

    so the best way would be using the "fuser"- and "lsof"-commands (I don't have "fstat").

    avizit: can you give me a hint how to use the lsof-command? then I will split points between you and tfewster!

    thx for all your comments
    michi
    0
     
    LVL 11

    Assisted Solution

    by:avizit
    you have to download 'lsof' if it doesnt exists already on your system

    http://ftp.cerias.purdue.edu/pub/tools/unix/sysutils/lsof/


    for usage you can read

    http://www.uwsg.iu.edu/UAU/advcomm/lsof.html
    0
     
    LVL 1

    Author Comment

    by:witty
    How can I splt points - I only see "Accept Answer", but not "Split Points".

    thx
    michi
    0
     
    LVL 20

    Expert Comment

    by:tfewster
    There is a "Split Points" option just above the "add a comment" box

    See http://www.experts-exchange.com/help.jsp#hi69 for more info on splitting points
    0

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Course: Foundations of Front-End Development

    Jump-start a lucrative career in front-end web development, with zero previous coding experience required. This course covers the basic programming concepts and languages required for creating engaging websites from scratch.

    This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
    Java performance on Solaris - Managing CPUs There are various resource controls in operating system which directly/indirectly influence the performance of application. one of the most important resource controls is "CPU".   In a multithreaded…
    Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
    This tutorial goes over how to archive and restore FreeBSD jails that are managed by ezjail.

    933 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now