I'm developing a program that will need to keep a record of millions of images, created at a rate of about 5 per second.
Right now, I'm simply writing them, separately (each image in its own separate file) into a folder.
However, as soon as the folder gets 5, 10, 20 thousand images, if you try to open it in "explorer", the desktop stops responding for a few seconds, getting increasingly worse with the number of images.
So, what I need to know is, what's the best way to handle such large amount of files?
I tried setting the folder as compressed, to see if the internal management of a large number of files would be more efficient - but it's the same.
My plausible options are:
1) splitting it up into different folders, each having just about 1000 images. (but causing a "management" nightmare in the software side)
2) storing the images inside a MySQL database (haven't tried it out - don't know if it would be very efficient storing binary images at those rates in there - creating a 200GB database)
3) any options to speed up windows folder "parsing"?
4) Whatever you can think of that might be of help