Most efficient way to write to a file

Posted on 2007-10-15
Last Modified: 2010-03-05
I need to know the most efficient way to write to a file. Currently I am just doing an open on the file and appending to the file, then I close the file.  This script is called programmatically every three minutes.  There is another script that accesses the same file.   This script opens the file and pulls into an array and closed the file and is called by an appicalion when needed, so it may run one a minute, second or hour.  I am concerned there may be file contention.
Question by:MatthewF
    LVL 5

    Expert Comment

    Your problem may be that your reading program may occasionally pick up a slightly out of date file. In the worst case scenario, your reading script will attempt to open the file while your appending script is attempting to write to it.  As a result, your reading script will not get the latest lines appended.
    LVL 5

    Accepted Solution

    You have a few options here:

    1) Keep things as they are.
    2) Combine the script that accesses the data with the one that requires the data so that fresh data will always be taken when needed.
    3) Cache the data in shared memory bypassing files altogether.
    4) Consider a messaging system.

    Problems happen with #1  and #2 when you deal with huge amounts of data. If the amount of data is always small, then there's really no reason to worry.

    If you happen to be dealing with a small set of data and you just keep on appending the latest to the end of the data file, you can use rotating files to ensure that your data file will not grow to be too large.

    With larger data sets, there exist perl modules for sharing memory. See:

    If you can think of updates to you data file as messages in a conversation between two processes, consider using messages instead. Perl has modules for sending messages over networks.
    LVL 28

    Expert Comment

    Are you using flock to lock the file?  If not, you should.

    perldoc -f flock

    Another option would be to put your data into a database instead of the plain text file.

    Featured Post

    What Is Threat Intelligence?

    Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

    Join & Write a Comment

    I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
    I have been pestered over the years to produce and distribute regular data extracts, and often the request have explicitly requested the data be emailed as an Excel attachement; specifically Excel, as it appears: CSV files confuse (no Red or Green h…
    Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
    Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…

    732 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    17 Experts available now in Live!

    Get 1:1 Help Now