Strategies for writing many small files to disk
Posted on 2014-07-16
I am working on a project that uses VB.Net and GDI+ to generate "lots" of fairly small bitmapped images. Since I am only generating 10's or 100's at a time, performance is not a real issue. However, I am looking at a job where I may have to generate up to a million at a time.
I am not too worried about the performance of generating the actual bitmaps in GDI+, however when I need to start writing so many files to the hard disk I am a bit concerned about that. Right now, I just do a "bitmap.save" -> <hard disk path> after I generate each one.
I am wondering if there are some strategies to optimize the writing out of "many" such small image files. For instance, letting them build up in RAM and writing out a bunch in a burst instead of "generate one, save, generate another, save, and so on..."
I realize the question is kind of vague, so possibly what I really need to know is the right question. Does anyone have some general advice, experience, resources, or comments that might address this issue?
Here are the specs of what I'm running now:
------- Hardware ---------
HP quad core, AMD A8-5500, 3.2GHz
8 Gb RAM
Western Digital WD Blue WD10EZEX 1TB drive
Windows 8 64 bit
------- Software ---------
VB.Net 2010, dot.net Framework 4
resulting bitmap file size ~100k