Is it possible to reuse file handle while changing the file name/location?

Hello,

I was wondering if it was possible to reuse a file handle in Python and change where the file handle points to?

Basically what I want to do is, instead of...
for x in range(0, 10000)
    file = open('file_' + str(x) + '.txt', 'w')
    file.write('some text\n')
    file.close()

Open in new window


I want to do something like...
file = open('file_0.txt', 'w')
for x in range(0, 10000)
    file.write('some text\n')
    <magic code to change file location so I don't need to create a new file object>
file.close()

Open in new window


I'm asking this because I'll be creating about 500k files; and creating new file objects seems to be slowing down the script.

Appreciate any help on this!
Errang GenevreAsked:
Who is Participating?
 
arnoldCommented:
your first example is the correct way to reuse file handle, close the existing, open with the new file name.

You are not testing whether the open is encountering an error such that if it does your process will lose data i.e. will write into a file handle that is not set.  

testing for a successful execution (opening a file) is highly recommended since you will not detect the error until it occurs and the data within is seen valuable by someone who requested access to it.
0
 
wilcoxonCommented:
There is no way to do this in Perl (I'm assuming there isn't in Python either).  As arnold said, you should be checking for the proper functioning of the calls.

in Perl, the equivalent (corrected) code snippet would be below.
use Fatal qw(open close);
for my $x (0..10000) {
    open OUT, ">file_$x.txt";
    print OUT "some text\n";
    close OUT;

Open in new window


If you don't have a recent enough version of Perl to include Fatal, you can do:
for my $x (0..10000) {
    open OUT, ">file_$x.txt" or die "could not write file_$x.txt: $!";
    print OUT "some text\n";
    close OUT or die "could not close handle for file_$x.txt: $!";

Open in new window

0
 
Errang GenevreAuthor Commented:
Alright, thanks.

Didn't expect it to be possible, but just thought I'd check.

I'll try experimenting with the shell output redirection; hopefully that'll have a smaller footprint.
0
Cloud Class® Course: MCSA MCSE Windows Server 2012

This course teaches how to install and configure Windows Server 2012 R2.  It is the first step on your path to becoming a Microsoft Certified Solutions Expert (MCSE).

 
arnoldCommented:
the comparative delay to close/open file handle is significantly smaller than your processing of data.

depending on what you are doing, you could use syslog/rsyslog as the destination for your output and have it manage the writeout into files.

pushing control out means you also are pushing the means of detecting errors in the process.
0
 
Errang GenevreAuthor Commented:
That's true.

The current process runs for 1.5+ hours, but its writing 500k files, and also creating over 2-3 times the number of directories. So I'm just trying to see if possible to use a combination of scripts to get it done quicker.

Still haven't found a decent enough solution for the file creation part; but yea, I still need to consider the Error handling part.
0
 
arnoldCommented:
If you comment out the file creation part and time the processing, dies it decrease.

Multiple directories suggests there is some additional logic to directories/file handles?
0
 
Errang GenevreAuthor Commented:
Since there are 500k files (file names are based on a numeric range), the folders are a way to group similar files, and not have Unix scream at us for having that many files in one location.
0
 
arnoldCommented:
Not sure whether running a daemon that will be charged with file creation to which your processing app will be sending its output is a worth while thing.

One thing to look at in your existing script is whether it is buffering the output.

I still think if the processing of the data can be sped up, that will likely be the way to go.
0
 
Errang GenevreAuthor Commented:
Alright, cool; thanks.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.