Storing the content of thousands of files

Posted on 2006-06-04
Last Modified: 2013-11-20
I need the ability to store the contents of thousands or even tens of thousands of files; each will be attached to a unique account number (actually each account might have anywhere from 0 to hundreds of file).  I know of numerous ways to accomplish this in the Unix world, and almost all would be available in the Windows world as well.

My question, is there some sort of standard that is used in the Windows environment?  

If not, which ways work the best?

If I decide to keep the files in their native form, and just keep an index of their location, or hash their location, is there some magic number of files in a single directory, where the directory becomes HUGE, in the sense of Unix, where opening a file takes a lifetime, etc?

Thanks in advance, Rick
Question by:rickatseasoft
    1 Comment
    LVL 43

    Accepted Solution

    Standard way - not that I know of.  The criteria you would have are important.  (Disc space, how important is speed to find/load the file..)

    Yes - having too many files in a directory can slow performance (I'd keep it less than 500).  At some point you would even run into a file system limit, which would be different for the formatting of the hard disk (FAT32, NTFS).  As can having too many characters in the overall path to the file and too many directories in the path.

    Exactly what the performance hit would be however I can't say.  

    What you could try is to just create a huge number of files (small app/bat file to do that) and see what the performance is.

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Join & Write a Comment

    Suggested Solutions

    Introduction: Dynamic window placements and drawing on a form, simple usage of windows registry as a storage place for information. Continuing from the first article about sudoku.  There we have designed the application and put a lot of user int…
    Introduction: Displaying information on the statusbar.   Continuing from the third article about sudoku.   Open the project in visual studio. Status bar – let’s display the timestamp there.  We need to get the timestamp from the document s…
    This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.
    It is a freely distributed piece of software for such tasks as photo retouching, image composition and image authoring. It works on many operating systems, in many languages.

    755 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    21 Experts available now in Live!

    Get 1:1 Help Now