Linux File Size - File Generation

Posted on 2005-03-21
Medium Priority
Last Modified: 2010-04-20

I am not very familiar with Linux but i have been asked to research a problem we have been having at our company.
We have an application which picks up temporary files (generated by another application).   The problem is that the coding in the application which picks up the files is based around the Windows file system - which only gives files sizes once they have been fully generated, otherwised the file size is set to 0 bytes - and is set so that it picks up the file if the size is anything other than 0 bytes.

The problem is that the application which picks up the files, from Linux, is picking them up before they have completely generated - this is because when files are generated on Linux the file size is incremented continually until the full size is reached.

We dont want to have to change any of the applications so is there any way of locking the files until they have fully generated?  Or maybe there is a way in Linux to keep the file size at 0 bytes until it has been fully generated?

(we can increase the polling on the application but this doesnt actually solve the problem and there still exists a chance that it may poll as a file is being created).

Your help would be greatly appreciated,

Kind Regards,

Question by:dcsearle
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2

Assisted Solution

Robson earned 400 total points
ID: 13590469
Hmm... I've googled a bit for "linux mandatory file locking" and found this on fcntl(2) manpage:
Mandatory locking
(Non-POSIX.) The above record locks may be either advisory or mandatory, and are advisory by default. To make use of mandatory locks, mandatory locking must be enabled (using the "-o mand" option to mount(8)) for the file system containing the file to be locked and enabled on the file itself (by disabling group execute permission on the file and enabling the set-GID permission bit).

Advisory locks are not enforced and are useful only between cooperating processes. Mandatory locks are enforced for all processes.
..but that would reuire changing application that generates files. Have you got acces to their code and licence to modify it? If not, you've got problem... I don't believe you want to make changes to operating system in orde to make this application work :). Playing with LD_PRELOAD to override 'stat' system call (used to get files size) may also help, but both ways are much MUCH harder than simply rewritting some lines in application code.



Author Comment

ID: 13590614
Yeah the code is our own but it isnt really an option as it is very old software which is tried and tested.  However it is looking like we are going to have to consider that option.  Is there no way that Linux can be set to report file sizes as 0 bytes until they are generated?

thanks for the help,


Assisted Solution

Robson earned 400 total points
ID: 13590955
In linux file system there's no 'not yet closed' flag, option or something as far as I know therefore: no way to recognize 'not fully generated' files between others. Your applicaton may work well, but is not very well designed -- lockfiles or different file names for temporary files should be used in first place IMHO.

Good luck!


Accepted Solution

PeterDragonstaff earned 100 total points
ID: 13599353
You have a few options
1) Change the Linux app to create the file under another name, perhaps in another directory, and when finished rename it to one your temp file watcher is expecting
2) Change the Linux app and watcher to use lockfiles (best IMO)
3) Change your watcher to look at the file modification time (On linux do "man 2 stat", it is field st_mtime) and assume once the file is over, say, a minute old that the Linux app has finished. This is not a 100% solution but might work if the data is not critical.


Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

You ever wonder how to backup Linux system files just like Windows System Restore?  Well you can use Timeshift in Linux to perform those similar action.  This tutorial will show you how to backup your system files and keep regular intervals. Noteā€¦
The purpose of this article is to demonstrate how we can use conditional statements using Python.
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.
Suggested Courses
Course of the Month13 days, 17 hours left to enroll

800 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question