• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 217
  • Last Modified:

Most efficient way to write to a file

I need to know the most efficient way to write to a file. Currently I am just doing an open on the file and appending to the file, then I close the file.  This script is called programmatically every three minutes.  There is another script that accesses the same file.   This script opens the file and pulls into an array and closed the file and is called by an appicalion when needed, so it may run one a minute, second or hour.  I am concerned there may be file contention.
  • 2
1 Solution
Your problem may be that your reading program may occasionally pick up a slightly out of date file. In the worst case scenario, your reading script will attempt to open the file while your appending script is attempting to write to it.  As a result, your reading script will not get the latest lines appended.
You have a few options here:

1) Keep things as they are.
2) Combine the script that accesses the data with the one that requires the data so that fresh data will always be taken when needed.
3) Cache the data in shared memory bypassing files altogether.
4) Consider a messaging system.

Problems happen with #1  and #2 when you deal with huge amounts of data. If the amount of data is always small, then there's really no reason to worry.

If you happen to be dealing with a small set of data and you just keep on appending the latest to the end of the data file, you can use rotating files to ensure that your data file will not grow to be too large.

With larger data sets, there exist perl modules for sharing memory. See:

If you can think of updates to you data file as messages in a conversation between two processes, consider using messages instead. Perl has modules for sending messages over networks.
Are you using flock to lock the file?  If not, you should.

perldoc -f flock

Another option would be to put your data into a database instead of the plain text file.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now