How to be sure, file is written completely

Posted on 2004-10-24
Medium Priority
Last Modified: 2010-04-21

I do a check, if a file exists "test -f <file>".
When it exists I do some processing ...

- but how can I be sure, that the file ist completely written, the moment I start the process?

Question by:witty
  • 2
  • 2
  • 2
  • +4
LVL 11

Expert Comment

ID: 12393629
if your system supports the "lsof" command then it can give you the list if open files , which may be sufficient for your purpose that if a file is not open at the moment it is as good as being completely written for you

but that still doesnt gurantee if a file is completely written .. files especially the log files etc can get written any time depending of how some program runs and the errors it comes across ( just to give you an example )
LVL 21

Accepted Solution

tfewster earned 100 total points
ID: 12393674
`fuser filename` does the same thing as lsof in this situation, but there is still the problem that the writing process may not have the file open at that moment...

The usual method is to have the writing process create a temp file and rename it when it is complete,
e.g.  proc > filename.tmp && mv filename.tmp filename.end

Or create a flag file that the reading process looks for before it tries to read the data file, e.g.
ftp myserver
put filename
put flagfile
LVL 62

Expert Comment

ID: 12393937
some systems have same command under name fstat
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.


Expert Comment

ID: 12429707
If your script have write access on this file, try to create write lock on the file.
If you can lock it, it's not open for write anywhere else.

If the other process, which is generating this file is yours as well, then
- Create Lock ( touch /tmp/.lock_logfile )
- Generate file ( ls -R  > /tmp/logfile )
- Release the lock ( /bin/rm /tmp/.lock_logfile )

LVL 51

Expert Comment

ID: 12456246
>  but how can I be sure, that the file ist completely written
do mean to perform this check from within a shell script?
If so, it's not that easy. You need to know the filesystem and its configuration (cacheing in particular)
If you're on NFS too, then somthing like ITcrow suggest might be nbecessary too
LVL 85

Expert Comment

ID: 12501748
Your process and the processes which may write to the file should agree on some protocol for determining when the writing is complete.
Or, you might mv the file to a private place to ensure that it can no longer be written to.

Author Comment

ID: 12509121
ok - I see there is no general solution for this problem!

I think the best way would be, telling the application to rename the file after finishing writing from a temporary name to the final name!
- but I can't use this solution, because I can't change the application.

so the best way would be using the "fuser"- and "lsof"-commands (I don't have "fstat").

avizit: can you give me a hint how to use the lsof-command? then I will split points between you and tfewster!

thx for all your comments
LVL 11

Assisted Solution

avizit earned 100 total points
ID: 12520082
you have to download 'lsof' if it doesnt exists already on your system


for usage you can read


Author Comment

ID: 12747144
How can I splt points - I only see "Accept Answer", but not "Split Points".

LVL 21

Expert Comment

ID: 12747609
There is a "Split Points" option just above the "add a comment" box

See http://www.experts-exchange.com/help.jsp#hi69 for more info on splitting points

Featured Post

Concerto's Cloud Advisory Services

Want to avoid the missteps to gaining all the benefits of the cloud? Learn more about the different assessment options from our Cloud Advisory team.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A metadevice consists of one or more devices (slices). It can be expanded by adding slices. Then, it can be grown to fill a larger space while the file system is in use. However, not all UNIX file systems (UFS) can be expanded this way. The conca…
This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

579 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question