• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 200
  • Last Modified:

Strategies for writing many small files to disk

I am working on a project that uses VB.Net and GDI+ to generate "lots" of fairly small bitmapped images. Since I am only generating 10's or 100's at a time, performance is not a real issue. However, I am looking at a job where I may have to generate up to a million at a time.

I am not too worried about the performance of generating the actual bitmaps in GDI+, however when I need to start writing so many files to the hard disk I am a bit concerned about that. Right now, I just do a "bitmap.save" -> <hard disk path> after I generate each one.

I am wondering if there are some strategies to optimize the writing out of "many" such small image files. For instance, letting them build up in RAM and writing out a bunch in a burst instead of "generate one, save, generate another, save, and so on..."

I realize the question is kind of vague, so possibly what I really need to know is the right question. Does anyone have some general advice, experience, resources, or comments that might address this issue?

Here are the specs of what I'm running now:
------- Hardware ---------
HP quad core, AMD A8-5500, 3.2GHz
8 Gb RAM
Western Digital WD Blue WD10EZEX 1TB drive
Windows 8 64 bit

------- Software ---------
VB.Net 2010, dot.net Framework 4
GDI+
resulting bitmap file size ~100k

Thanks.
0
futuremoose
Asked:
futuremoose
  • 5
  • 3
  • 2
2 Solutions
 
tankergoblinCommented:
What do you mean by writing so files to the disk??? the writing you mention here is code?
0
 
futuremooseAuthor Commented:
Using GDI+ I am creating a bitmap, which I then wish to save to the hard disk as a <name>.JPG file (or PNG, GIF, whatever). I am investigating the performance of saving the images (of which there may be many) to files on the hard drive.
0
 
tankergoblinCommented:
you going to save the file one at a time?
0
 
futuremooseAuthor Commented:
Since I am not generating so many now, just a few dozen or so, yes... I just save them one at a time. Not a problem.

My question is, if and when I am generating maybe 10,000 or so in a single session, what better way than "one at a time" is there to get them all saved to the hard disk.
0
 
tankergoblinCommented:
You can write a program to generate it for you. what language you are familiar with?

or you can use picture converter ready made in the market.
0
 
futuremooseAuthor Commented:
Generate what for me? I am already using VB.Net and GDI+ to generate the image... I'm just looking for a better way than GDI+'s Bitmap.Save to save them to the hard disk one at a time. I think my solution may be to save them a RAM disk initially, then transfer the whole bunch to the hard disk at once.
0
 
Gerald ConnollyCommented:
10K over what timescale?

Although the NTFS file system will cope with these kinds of numbers, (Max = 4GigaFiles) you need to be aware of maximum file name length and also max path lengths.

NB. Lots of small files have a detrimental effect on file system performance, so backups etc very slow traversing directories.
0
 
futuremooseAuthor Commented:
The time to create the images using GDI+ varies according to the complexity and resolution, but let's say I'm generating between 5 and 20 per second. At the worst, 5 per second, 10k would take 33 minutes.

Concerning the actual number of files being generated, my current thinking is this:
- generate in batches of no more than 10k
- do the initial 'bitmap.save's to a RAM disk
- ZIP together into a single file
- then xfer that to the hard disk, upload to cloud, or whatever

But any other thoughts or comments are welcome.
0
 
Gerald ConnollyCommented:
That sounds like a plan, and would reduce the filesystem overhead
0
 
futuremooseAuthor Commented:
Came up with my own solution, but thanks to Gerald for his comments.
0
  • 5
  • 3
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now