Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 214
  • Last Modified:

Most efficient way to write to a file

I need to know the most efficient way to write to a file. Currently I am just doing an open on the file and appending to the file, then I close the file.  This script is called programmatically every three minutes.  There is another script that accesses the same file.   This script opens the file and pulls into an array and closed the file and is called by an appicalion when needed, so it may run one a minute, second or hour.  I am concerned there may be file contention.
0
MatthewF
Asked:
MatthewF
  • 2
1 Solution
 
mkatmonkeyCommented:
Your problem may be that your reading program may occasionally pick up a slightly out of date file. In the worst case scenario, your reading script will attempt to open the file while your appending script is attempting to write to it.  As a result, your reading script will not get the latest lines appended.
0
 
mkatmonkeyCommented:
You have a few options here:

1) Keep things as they are.
2) Combine the script that accesses the data with the one that requires the data so that fresh data will always be taken when needed.
3) Cache the data in shared memory bypassing files altogether.
4) Consider a messaging system.

Problems happen with #1  and #2 when you deal with huge amounts of data. If the amount of data is always small, then there's really no reason to worry.

If you happen to be dealing with a small set of data and you just keep on appending the latest to the end of the data file, you can use rotating files to ensure that your data file will not grow to be too large.

With larger data sets, there exist perl modules for sharing memory. See:
http://search.cpan.org/~dwinters/Persistent-Base-0.52/lib/Persistent.pod

If you can think of updates to you data file as messages in a conversation between two processes, consider using messages instead. Perl has modules for sending messages over networks.
http://search.cpan.org/~rcaputo/POE-0.9999/lib/POE/Component/Client/TCP.pm
0
 
FishMongerCommented:
Are you using flock to lock the file?  If not, you should.

perldoc -f flock
http://search.cpan.org/~makoto/IO-File-Lockable-0.34/flock.pm
http://search.cpan.org/~nwclark/perl-5.8.8/ext/Fcntl/Fcntl.pm

Another option would be to put your data into a database instead of the plain text file.
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now