[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now


How to be sure, file is written completely

Posted on 2004-10-24
Medium Priority
Last Modified: 2010-04-21

I do a check, if a file exists "test -f <file>".
When it exists I do some processing ...

- but how can I be sure, that the file ist completely written, the moment I start the process?

Question by:witty
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
  • 2
  • +4
LVL 11

Expert Comment

ID: 12393629
if your system supports the "lsof" command then it can give you the list if open files , which may be sufficient for your purpose that if a file is not open at the moment it is as good as being completely written for you

but that still doesnt gurantee if a file is completely written .. files especially the log files etc can get written any time depending of how some program runs and the errors it comes across ( just to give you an example )
LVL 21

Accepted Solution

tfewster earned 100 total points
ID: 12393674
`fuser filename` does the same thing as lsof in this situation, but there is still the problem that the writing process may not have the file open at that moment...

The usual method is to have the writing process create a temp file and rename it when it is complete,
e.g.  proc > filename.tmp && mv filename.tmp filename.end

Or create a flag file that the reading process looks for before it tries to read the data file, e.g.
ftp myserver
put filename
put flagfile
LVL 62

Expert Comment

ID: 12393937
some systems have same command under name fstat
What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.


Expert Comment

ID: 12429707
If your script have write access on this file, try to create write lock on the file.
If you can lock it, it's not open for write anywhere else.

If the other process, which is generating this file is yours as well, then
- Create Lock ( touch /tmp/.lock_logfile )
- Generate file ( ls -R  > /tmp/logfile )
- Release the lock ( /bin/rm /tmp/.lock_logfile )

LVL 51

Expert Comment

ID: 12456246
>  but how can I be sure, that the file ist completely written
do mean to perform this check from within a shell script?
If so, it's not that easy. You need to know the filesystem and its configuration (cacheing in particular)
If you're on NFS too, then somthing like ITcrow suggest might be nbecessary too
LVL 84

Expert Comment

ID: 12501748
Your process and the processes which may write to the file should agree on some protocol for determining when the writing is complete.
Or, you might mv the file to a private place to ensure that it can no longer be written to.

Author Comment

ID: 12509121
ok - I see there is no general solution for this problem!

I think the best way would be, telling the application to rename the file after finishing writing from a temporary name to the final name!
- but I can't use this solution, because I can't change the application.

so the best way would be using the "fuser"- and "lsof"-commands (I don't have "fstat").

avizit: can you give me a hint how to use the lsof-command? then I will split points between you and tfewster!

thx for all your comments
LVL 11

Assisted Solution

avizit earned 100 total points
ID: 12520082
you have to download 'lsof' if it doesnt exists already on your system


for usage you can read


Author Comment

ID: 12747144
How can I splt points - I only see "Accept Answer", but not "Split Points".

LVL 21

Expert Comment

ID: 12747609
There is a "Split Points" option just above the "add a comment" box

See http://www.experts-exchange.com/help.jsp#hi69 for more info on splitting points

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
Using libpcap/Jpcap to capture and send packets on Solaris version (10/11) Library used: 1.      Libpcap (http://www.tcpdump.org) Version 1.2 2.      Jpcap(http://netresearch.ics.uci.edu/kfujii/Jpcap/doc/index.html) Version 0.6 Prerequisite: 1.      GCC …
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.
Suggested Courses

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question