How to be sure, file is written completely


I do a check, if a file exists "test -f <file>".
When it exists I do some processing ...

- but how can I be sure, that the file ist completely written, the moment I start the process?

Who is Participating?
tfewsterConnect With a Mentor Commented:
`fuser filename` does the same thing as lsof in this situation, but there is still the problem that the writing process may not have the file open at that moment...

The usual method is to have the writing process create a temp file and rename it when it is complete,
e.g.  proc > filename.tmp && mv filename.tmp filename.end

Or create a flag file that the reading process looks for before it tries to read the data file, e.g.
ftp myserver
put filename
put flagfile
if your system supports the "lsof" command then it can give you the list if open files , which may be sufficient for your purpose that if a file is not open at the moment it is as good as being completely written for you

but that still doesnt gurantee if a file is completely written .. files especially the log files etc can get written any time depending of how some program runs and the errors it comes across ( just to give you an example )
some systems have same command under name fstat
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

If your script have write access on this file, try to create write lock on the file.
If you can lock it, it's not open for write anywhere else.

If the other process, which is generating this file is yours as well, then
- Create Lock ( touch /tmp/.lock_logfile )
- Generate file ( ls -R  > /tmp/logfile )
- Release the lock ( /bin/rm /tmp/.lock_logfile )

>  but how can I be sure, that the file ist completely written
do mean to perform this check from within a shell script?
If so, it's not that easy. You need to know the filesystem and its configuration (cacheing in particular)
If you're on NFS too, then somthing like ITcrow suggest might be nbecessary too
Your process and the processes which may write to the file should agree on some protocol for determining when the writing is complete.
Or, you might mv the file to a private place to ensure that it can no longer be written to.
wittyAuthor Commented:
ok - I see there is no general solution for this problem!

I think the best way would be, telling the application to rename the file after finishing writing from a temporary name to the final name!
- but I can't use this solution, because I can't change the application.

so the best way would be using the "fuser"- and "lsof"-commands (I don't have "fstat").

avizit: can you give me a hint how to use the lsof-command? then I will split points between you and tfewster!

thx for all your comments
avizitConnect With a Mentor Commented:
you have to download 'lsof' if it doesnt exists already on your system

for usage you can read
wittyAuthor Commented:
How can I splt points - I only see "Accept Answer", but not "Split Points".

There is a "Split Points" option just above the "add a comment" box

See for more info on splitting points
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.