futuremoose
asked on
Strategies for writing many small files to disk
I am working on a project that uses VB.Net and GDI+ to generate "lots" of fairly small bitmapped images. Since I am only generating 10's or 100's at a time, performance is not a real issue. However, I am looking at a job where I may have to generate up to a million at a time.
I am not too worried about the performance of generating the actual bitmaps in GDI+, however when I need to start writing so many files to the hard disk I am a bit concerned about that. Right now, I just do a "bitmap.save" -> <hard disk path> after I generate each one.
I am wondering if there are some strategies to optimize the writing out of "many" such small image files. For instance, letting them build up in RAM and writing out a bunch in a burst instead of "generate one, save, generate another, save, and so on..."
I realize the question is kind of vague, so possibly what I really need to know is the right question. Does anyone have some general advice, experience, resources, or comments that might address this issue?
Here are the specs of what I'm running now:
------- Hardware ---------
HP quad core, AMD A8-5500, 3.2GHz
8 Gb RAM
Western Digital WD Blue WD10EZEX 1TB drive
Windows 8 64 bit
------- Software ---------
VB.Net 2010, dot.net Framework 4
GDI+
resulting bitmap file size ~100k
Thanks.
I am not too worried about the performance of generating the actual bitmaps in GDI+, however when I need to start writing so many files to the hard disk I am a bit concerned about that. Right now, I just do a "bitmap.save" -> <hard disk path> after I generate each one.
I am wondering if there are some strategies to optimize the writing out of "many" such small image files. For instance, letting them build up in RAM and writing out a bunch in a burst instead of "generate one, save, generate another, save, and so on..."
I realize the question is kind of vague, so possibly what I really need to know is the right question. Does anyone have some general advice, experience, resources, or comments that might address this issue?
Here are the specs of what I'm running now:
------- Hardware ---------
HP quad core, AMD A8-5500, 3.2GHz
8 Gb RAM
Western Digital WD Blue WD10EZEX 1TB drive
Windows 8 64 bit
------- Software ---------
VB.Net 2010, dot.net Framework 4
GDI+
resulting bitmap file size ~100k
Thanks.
What do you mean by writing so files to the disk??? the writing you mention here is code?
ASKER
Using GDI+ I am creating a bitmap, which I then wish to save to the hard disk as a <name>.JPG file (or PNG, GIF, whatever). I am investigating the performance of saving the images (of which there may be many) to files on the hard drive.
you going to save the file one at a time?
ASKER
Since I am not generating so many now, just a few dozen or so, yes... I just save them one at a time. Not a problem.
My question is, if and when I am generating maybe 10,000 or so in a single session, what better way than "one at a time" is there to get them all saved to the hard disk.
My question is, if and when I am generating maybe 10,000 or so in a single session, what better way than "one at a time" is there to get them all saved to the hard disk.
You can write a program to generate it for you. what language you are familiar with?
or you can use picture converter ready made in the market.
or you can use picture converter ready made in the market.
ASKER
Generate what for me? I am already using VB.Net and GDI+ to generate the image... I'm just looking for a better way than GDI+'s Bitmap.Save to save them to the hard disk one at a time. I think my solution may be to save them a RAM disk initially, then transfer the whole bunch to the hard disk at once.
10K over what timescale?
Although the NTFS file system will cope with these kinds of numbers, (Max = 4GigaFiles) you need to be aware of maximum file name length and also max path lengths.
NB. Lots of small files have a detrimental effect on file system performance, so backups etc very slow traversing directories.
Although the NTFS file system will cope with these kinds of numbers, (Max = 4GigaFiles) you need to be aware of maximum file name length and also max path lengths.
NB. Lots of small files have a detrimental effect on file system performance, so backups etc very slow traversing directories.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Came up with my own solution, but thanks to Gerald for his comments.