Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium


Linux File Size - File Generation

Posted on 2005-03-21
Medium Priority
Last Modified: 2010-04-20

I am not very familiar with Linux but i have been asked to research a problem we have been having at our company.
We have an application which picks up temporary files (generated by another application).   The problem is that the coding in the application which picks up the files is based around the Windows file system - which only gives files sizes once they have been fully generated, otherwised the file size is set to 0 bytes - and is set so that it picks up the file if the size is anything other than 0 bytes.

The problem is that the application which picks up the files, from Linux, is picking them up before they have completely generated - this is because when files are generated on Linux the file size is incremented continually until the full size is reached.

We dont want to have to change any of the applications so is there any way of locking the files until they have fully generated?  Or maybe there is a way in Linux to keep the file size at 0 bytes until it has been fully generated?

(we can increase the polling on the application but this doesnt actually solve the problem and there still exists a chance that it may poll as a file is being created).

Your help would be greatly appreciated,

Kind Regards,

Question by:dcsearle
  • 2

Assisted Solution

Robson earned 400 total points
ID: 13590469
Hmm... I've googled a bit for "linux mandatory file locking" and found this on fcntl(2) manpage:
Mandatory locking
(Non-POSIX.) The above record locks may be either advisory or mandatory, and are advisory by default. To make use of mandatory locks, mandatory locking must be enabled (using the "-o mand" option to mount(8)) for the file system containing the file to be locked and enabled on the file itself (by disabling group execute permission on the file and enabling the set-GID permission bit).

Advisory locks are not enforced and are useful only between cooperating processes. Mandatory locks are enforced for all processes.
..but that would reuire changing application that generates files. Have you got acces to their code and licence to modify it? If not, you've got problem... I don't believe you want to make changes to operating system in orde to make this application work :). Playing with LD_PRELOAD to override 'stat' system call (used to get files size) may also help, but both ways are much MUCH harder than simply rewritting some lines in application code.



Author Comment

ID: 13590614
Yeah the code is our own but it isnt really an option as it is very old software which is tried and tested.  However it is looking like we are going to have to consider that option.  Is there no way that Linux can be set to report file sizes as 0 bytes until they are generated?

thanks for the help,


Assisted Solution

Robson earned 400 total points
ID: 13590955
In linux file system there's no 'not yet closed' flag, option or something as far as I know therefore: no way to recognize 'not fully generated' files between others. Your applicaton may work well, but is not very well designed -- lockfiles or different file names for temporary files should be used in first place IMHO.

Good luck!


Accepted Solution

PeterDragonstaff earned 100 total points
ID: 13599353
You have a few options
1) Change the Linux app to create the file under another name, perhaps in another directory, and when finished rename it to one your temp file watcher is expecting
2) Change the Linux app and watcher to use lockfiles (best IMO)
3) Change your watcher to look at the file modification time (On linux do "man 2 stat", it is field st_mtime) and assume once the file is over, say, a minute old that the Linux app has finished. This is not a 100% solution but might work if the data is not critical.


Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension . This reminded me of questions that come up here at EE along the lines of, "How can I tell the type of file from its cont…
It’s 2016. Password authentication should be dead — or at least close to dying. But, unfortunately, it has not traversed Quagga stage yet. Using password authentication is like laundering hotel guest linens with a washboard — it’s Passé.
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
How to Install VMware Tools in Red Hat Enterprise Linux 6.4 (RHEL 6.4) Step-by-Step Tutorial
Suggested Courses
Course of the Month12 days, 3 hours left to enroll

564 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question