Storing Large Amounts of Files into Folders - Efficiently - and Naming Convention

Posted on 2006-07-20
Last Modified: 2006-11-18
I have an application that will allow users to upload photos. Potentially, my application can reach into the 100,000+ user range. I was wondering if anyone knows an efficient way to create the file structure, or naming convention, to hold the photos. I've seen some examples based on UserIDs, Dates, and/or Splitting up the PhotoID into sections, but I'm still not very clear on these techniques. I was also wondering how many files I should store in each folder so that the files can be access on my website efficiently. I've read anywhere betweek 100 to 1,000 files per folder.

Right now I am thinking about storing the photo file on a web server and using Sql Server to store the file reference and file info (i.e. FileID, UserID, Date, FileName, FolderStructure, and perhaps a DomainID to prepare for situations where I will have to use multiple servers to store the photos).

Any help and insight on the topic will be appreciated. Thanks!
Question by:Skytide
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
LVL 24

Expert Comment

ID: 17150558
Based on the info above, I would say that it is probably a bad idea to store the images directly on the file system for such a large number of users/files.

You should probably store the files in a binary/image/BLOB column in a DB (e.g. SQL Server) rather than storing them directly on the filesystem. You could also store the filename, user, etc. in other columns (and index them) to provide efficient access to each file without worrying about folder names, etc. Storing them in a DB would also bypass any potential problems resulting from limits on filename/path lengths (max is 256 in Windows IIRC).

Author Comment

ID: 17151321
I would prefer to store the images on a web server since I don't want my database to become too large. I also might have to store images across servers in the future, so I want to avoid having to extract the images from the database when that happens. As a note too, the photos aren't sensitive, so I'm not really worried about the security of them.

Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs. Here is one: However, it's a really confusing topic since it seems like one could go either way.
LVL 24

Expert Comment

ID: 17155440
>> Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs.

There were good reasons for this in SQL Server 7.0 and earlier, but SQL Server 2000 and later have resolved those issues

Author Comment

ID: 17161909
Anymore input on this? I want to keep things open in case I decide to use a Content Delivery Network, so I won't be storing the images in the database.
LVL 24

Accepted Solution

Justin_W earned 500 total points
ID: 17165000
Well, as I mentioned, file systems maximum length limits on filenames. Therefore, you may want to store full metadat info (full filename, dates, folders, etc.) in a DB, and use a GUID (or other reasonably short unique string) as the file system filename. That way, you could easily store and access full file data (including for reporting), but wouldn't have to worry about excessive string parsing or filenames. Also, you could always convert the filenames to something else later using a simple custom app.

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Sometimes in DotNetNuke module development you want to swap controls within the same module definition.  In doing this DNN (somewhat annoyingly) swaps the Skin and Container definitions to the default admin selections.  To get around this you need t…
Problem Hi all,    While many today have fast Internet connection, there are many still who do not, or are connecting through devices with a slower connect, so light web pages and fast load times are still popular.    If your ASP.NET page …
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…
Sometimes it takes a new vantage point, apart from our everyday security practices, to truly see our Active Directory (AD) vulnerabilities. We get used to implementing the same techniques and checking the same areas for a breach. This pattern can re…
Suggested Courses

623 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question