Strategies for writing many small files to disk

I am working on a project that uses VB.Net and GDI+ to generate "lots" of fairly small bitmapped images. Since I am only generating 10's or 100's at a time, performance is not a real issue. However, I am looking at a job where I may have to generate up to a million at a time.

I am not too worried about the performance of generating the actual bitmaps in GDI+, however when I need to start writing so many files to the hard disk I am a bit concerned about that. Right now, I just do a "" -> <hard disk path> after I generate each one.

I am wondering if there are some strategies to optimize the writing out of "many" such small image files. For instance, letting them build up in RAM and writing out a bunch in a burst instead of "generate one, save, generate another, save, and so on..."

I realize the question is kind of vague, so possibly what I really need to know is the right question. Does anyone have some general advice, experience, resources, or comments that might address this issue?

Here are the specs of what I'm running now:
------- Hardware ---------
HP quad core, AMD A8-5500, 3.2GHz
8 Gb RAM
Western Digital WD Blue WD10EZEX 1TB drive
Windows 8 64 bit

------- Software ---------
VB.Net 2010, Framework 4
resulting bitmap file size ~100k

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

What do you mean by writing so files to the disk??? the writing you mention here is code?
futuremooseAuthor Commented:
Using GDI+ I am creating a bitmap, which I then wish to save to the hard disk as a <name>.JPG file (or PNG, GIF, whatever). I am investigating the performance of saving the images (of which there may be many) to files on the hard drive.
you going to save the file one at a time?
futuremooseAuthor Commented:
Since I am not generating so many now, just a few dozen or so, yes... I just save them one at a time. Not a problem.

My question is, if and when I am generating maybe 10,000 or so in a single session, what better way than "one at a time" is there to get them all saved to the hard disk.
You can write a program to generate it for you. what language you are familiar with?

or you can use picture converter ready made in the market.
futuremooseAuthor Commented:
Generate what for me? I am already using VB.Net and GDI+ to generate the image... I'm just looking for a better way than GDI+'s Bitmap.Save to save them to the hard disk one at a time. I think my solution may be to save them a RAM disk initially, then transfer the whole bunch to the hard disk at once.
Gerald ConnollyCommented:
10K over what timescale?

Although the NTFS file system will cope with these kinds of numbers, (Max = 4GigaFiles) you need to be aware of maximum file name length and also max path lengths.

NB. Lots of small files have a detrimental effect on file system performance, so backups etc very slow traversing directories.
futuremooseAuthor Commented:
The time to create the images using GDI+ varies according to the complexity and resolution, but let's say I'm generating between 5 and 20 per second. At the worst, 5 per second, 10k would take 33 minutes.

Concerning the actual number of files being generated, my current thinking is this:
- generate in batches of no more than 10k
- do the initial ''s to a RAM disk
- ZIP together into a single file
- then xfer that to the hard disk, upload to cloud, or whatever

But any other thoughts or comments are welcome.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Gerald ConnollyCommented:
That sounds like a plan, and would reduce the filesystem overhead
futuremooseAuthor Commented:
Came up with my own solution, but thanks to Gerald for his comments.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Visual Basic.NET

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.