Storing the content of thousands of files

Posted on 2006-06-04
Medium Priority
Last Modified: 2013-11-20
I need the ability to store the contents of thousands or even tens of thousands of files; each will be attached to a unique account number (actually each account might have anywhere from 0 to hundreds of file).  I know of numerous ways to accomplish this in the Unix world, and almost all would be available in the Windows world as well.

My question, is there some sort of standard that is used in the Windows environment?  

If not, which ways work the best?

If I decide to keep the files in their native form, and just keep an index of their location, or hash their location, is there some magic number of files in a single directory, where the directory becomes HUGE, in the sense of Unix, where opening a file takes a lifetime, etc?

Thanks in advance, Rick
Question by:rickatseasoft
1 Comment
LVL 45

Accepted Solution

AndyAinscow earned 1000 total points
ID: 16830420
Standard way - not that I know of.  The criteria you would have are important.  (Disc space, how important is speed to find/load the file..)

Yes - having too many files in a directory can slow performance (I'd keep it less than 500).  At some point you would even run into a file system limit, which would be different for the formatting of the hard disk (FAT32, NTFS).  As can having too many characters in the overall path to the file and too many directories in the path.

Exactly what the performance hit would be however I can't say.  

What you could try is to just create a huge number of files (small app/bat file to do that) and see what the performance is.

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Introduction: Ownerdraw of the grid button.  A singleton class implentation and usage. Continuing from the fifth article about sudoku.   Open the project in visual studio. Go to the class view – CGridButton should be visible as a class.  R…
Exception Handling is in the core of any application that is able to dignify its name. In this article, I'll guide you through the process of writing a DRY (Don't Repeat Yourself) Exception Handling mechanism, using Aspect Oriented Programming.
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
If you are looking for an automated tool which can generate reports for Outlook emails and other items from PST file, then you can go for Kernel PST Reporter tool. The reports which are created by this tool are helpful to analyze and understand PST …

587 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question