Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17


Storing Large Amounts of Files into Folders - Efficiently - and Naming Convention

Posted on 2006-07-20
Medium Priority
Last Modified: 2006-11-18
I have an application that will allow users to upload photos. Potentially, my application can reach into the 100,000+ user range. I was wondering if anyone knows an efficient way to create the file structure, or naming convention, to hold the photos. I've seen some examples based on UserIDs, Dates, and/or Splitting up the PhotoID into sections, but I'm still not very clear on these techniques. I was also wondering how many files I should store in each folder so that the files can be access on my website efficiently. I've read anywhere betweek 100 to 1,000 files per folder.

Right now I am thinking about storing the photo file on a web server and using Sql Server to store the file reference and file info (i.e. FileID, UserID, Date, FileName, FolderStructure, and perhaps a DomainID to prepare for situations where I will have to use multiple servers to store the photos).

Any help and insight on the topic will be appreciated. Thanks!
Question by:Skytide
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
LVL 24

Expert Comment

ID: 17150558
Based on the info above, I would say that it is probably a bad idea to store the images directly on the file system for such a large number of users/files.

You should probably store the files in a binary/image/BLOB column in a DB (e.g. SQL Server) rather than storing them directly on the filesystem. You could also store the filename, user, etc. in other columns (and index them) to provide efficient access to each file without worrying about folder names, etc. Storing them in a DB would also bypass any potential problems resulting from limits on filename/path lengths (max is 256 in Windows IIRC).

Author Comment

ID: 17151321
I would prefer to store the images on a web server since I don't want my database to become too large. I also might have to store images across servers in the future, so I want to avoid having to extract the images from the database when that happens. As a note too, the photos aren't sensitive, so I'm not really worried about the security of them.

Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs. Here is one: However, it's a really confusing topic since it seems like one could go either way.
LVL 24

Expert Comment

ID: 17155440
>> Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs.

There were good reasons for this in SQL Server 7.0 and earlier, but SQL Server 2000 and later have resolved those issues

Author Comment

ID: 17161909
Anymore input on this? I want to keep things open in case I decide to use a Content Delivery Network, so I won't be storing the images in the database.
LVL 24

Accepted Solution

Justin_W earned 2000 total points
ID: 17165000
Well, as I mentioned, file systems maximum length limits on filenames. Therefore, you may want to store full metadat info (full filename, dates, folders, etc.) in a DB, and use a GUID (or other reasonably short unique string) as the file system filename. That way, you could easily store and access full file data (including for reporting), but wouldn't have to worry about excessive string parsing or filenames. Also, you could always convert the filenames to something else later using a simple custom app.

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Today is the age of broadband.  More and more people are going this route determined to experience the web and it’s multitude of services as quickly and painlessly as possible. Coupled with the move to broadband, people are experiencing the web via …
It was really hard time for me to get the understanding of Delegates in C#. I went through many websites and articles but I found them very clumsy. After going through those sites, I noted down the points in a easy way so here I am sharing that unde…
This course is ideal for IT System Administrators working with VMware vSphere and its associated products in their company infrastructure. This course teaches you how to install and maintain this virtualization technology to store data, prevent vuln…
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA:…
Suggested Courses

670 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question