Solved

Is it possible to reuse file handle while changing the file name/location?

Posted on 2014-12-28
9
287 Views
Last Modified: 2015-01-02
Hello,

I was wondering if it was possible to reuse a file handle in Python and change where the file handle points to?

Basically what I want to do is, instead of...
for x in range(0, 10000)
    file = open('file_' + str(x) + '.txt', 'w')
    file.write('some text\n')
    file.close()

Open in new window


I want to do something like...
file = open('file_0.txt', 'w')
for x in range(0, 10000)
    file.write('some text\n')
    <magic code to change file location so I don't need to create a new file object>
file.close()

Open in new window


I'm asking this because I'll be creating about 500k files; and creating new file objects seems to be slowing down the script.

Appreciate any help on this!
0
Comment
Question by:Errang Genevre
  • 4
  • 4
9 Comments
 
LVL 77

Accepted Solution

by:
arnold earned 250 total points
ID: 40522082
your first example is the correct way to reuse file handle, close the existing, open with the new file name.

You are not testing whether the open is encountering an error such that if it does your process will lose data i.e. will write into a file handle that is not set.  

testing for a successful execution (opening a file) is highly recommended since you will not detect the error until it occurs and the data within is seen valuable by someone who requested access to it.
0
 
LVL 26

Assisted Solution

by:wilcoxon
wilcoxon earned 250 total points
ID: 40522191
There is no way to do this in Perl (I'm assuming there isn't in Python either).  As arnold said, you should be checking for the proper functioning of the calls.

in Perl, the equivalent (corrected) code snippet would be below.
use Fatal qw(open close);
for my $x (0..10000) {
    open OUT, ">file_$x.txt";
    print OUT "some text\n";
    close OUT;

Open in new window


If you don't have a recent enough version of Perl to include Fatal, you can do:
for my $x (0..10000) {
    open OUT, ">file_$x.txt" or die "could not write file_$x.txt: $!";
    print OUT "some text\n";
    close OUT or die "could not close handle for file_$x.txt: $!";

Open in new window

0
 

Author Closing Comment

by:Errang Genevre
ID: 40522315
Alright, thanks.

Didn't expect it to be possible, but just thought I'd check.

I'll try experimenting with the shell output redirection; hopefully that'll have a smaller footprint.
0
The New “Normal” in Modern Enterprise Operations

DevOps for the modern enterprise offers many benefits — increased agility, productivity, and more, but digital transformation isn’t easy, especially if you’re not addressing the right issues. Register for the webinar to dive into the “new normal” for enterprise modern ops.

 
LVL 77

Expert Comment

by:arnold
ID: 40522366
the comparative delay to close/open file handle is significantly smaller than your processing of data.

depending on what you are doing, you could use syslog/rsyslog as the destination for your output and have it manage the writeout into files.

pushing control out means you also are pushing the means of detecting errors in the process.
0
 

Author Comment

by:Errang Genevre
ID: 40527270
That's true.

The current process runs for 1.5+ hours, but its writing 500k files, and also creating over 2-3 times the number of directories. So I'm just trying to see if possible to use a combination of scripts to get it done quicker.

Still haven't found a decent enough solution for the file creation part; but yea, I still need to consider the Error handling part.
0
 
LVL 77

Expert Comment

by:arnold
ID: 40527284
If you comment out the file creation part and time the processing, dies it decrease.

Multiple directories suggests there is some additional logic to directories/file handles?
0
 

Author Comment

by:Errang Genevre
ID: 40527294
Since there are 500k files (file names are based on a numeric range), the folders are a way to group similar files, and not have Unix scream at us for having that many files in one location.
0
 
LVL 77

Expert Comment

by:arnold
ID: 40528587
Not sure whether running a daemon that will be charged with file creation to which your processing app will be sending its output is a worth while thing.

One thing to look at in your existing script is whether it is buffering the output.

I still think if the processing of the data can be sped up, that will likely be the way to go.
0
 

Author Comment

by:Errang Genevre
ID: 40528591
Alright, cool; thanks.
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Strings in Python are the set of characters that, once defined, cannot be changed by any other method like replace. Even if we use the replace method it still does not modify the original string that we use, but just copies the string and then modif…
Article by: Swadhin
Introduction of Lists in Python: There are six built-in types of sequences. Lists and tuples are the most common one. In this article we will see how to use Lists in python and how we can utilize it while doing our own program. In general we can al…
In this fifth video of the Xpdf series, we discuss and demonstrate the PDFdetach utility, which is able to list and, more importantly, extract attachments that are embedded in PDF files. It does this via a command line interface, making it suitable …
Video by: Mark
This lesson goes over how to construct ordered and unordered lists and how to create hyperlinks.

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question