Storing Large Amounts of Files into Folders - Efficiently - and Naming Convention

Posted on 2006-07-20
Last Modified: 2006-11-18
I have an application that will allow users to upload photos. Potentially, my application can reach into the 100,000+ user range. I was wondering if anyone knows an efficient way to create the file structure, or naming convention, to hold the photos. I've seen some examples based on UserIDs, Dates, and/or Splitting up the PhotoID into sections, but I'm still not very clear on these techniques. I was also wondering how many files I should store in each folder so that the files can be access on my website efficiently. I've read anywhere betweek 100 to 1,000 files per folder.

Right now I am thinking about storing the photo file on a web server and using Sql Server to store the file reference and file info (i.e. FileID, UserID, Date, FileName, FolderStructure, and perhaps a DomainID to prepare for situations where I will have to use multiple servers to store the photos).

Any help and insight on the topic will be appreciated. Thanks!
Question by:Skytide
  • 3
  • 2
LVL 24

Expert Comment

ID: 17150558
Based on the info above, I would say that it is probably a bad idea to store the images directly on the file system for such a large number of users/files.

You should probably store the files in a binary/image/BLOB column in a DB (e.g. SQL Server) rather than storing them directly on the filesystem. You could also store the filename, user, etc. in other columns (and index them) to provide efficient access to each file without worrying about folder names, etc. Storing them in a DB would also bypass any potential problems resulting from limits on filename/path lengths (max is 256 in Windows IIRC).

Author Comment

ID: 17151321
I would prefer to store the images on a web server since I don't want my database to become too large. I also might have to store images across servers in the future, so I want to avoid having to extract the images from the database when that happens. As a note too, the photos aren't sensitive, so I'm not really worried about the security of them.

Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs. Here is one: However, it's a really confusing topic since it seems like one could go either way.
LVL 24

Expert Comment

ID: 17155440
>> Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs.

There were good reasons for this in SQL Server 7.0 and earlier, but SQL Server 2000 and later have resolved those issues

Author Comment

ID: 17161909
Anymore input on this? I want to keep things open in case I decide to use a Content Delivery Network, so I won't be storing the images in the database.
LVL 24

Accepted Solution

Justin_W earned 500 total points
ID: 17165000
Well, as I mentioned, file systems maximum length limits on filenames. Therefore, you may want to store full metadat info (full filename, dates, folders, etc.) in a DB, and use a GUID (or other reasonably short unique string) as the file system filename. That way, you could easily store and access full file data (including for reporting), but wouldn't have to worry about excessive string parsing or filenames. Also, you could always convert the filenames to something else later using a simple custom app.

Featured Post

Courses: Start Training Online With Pros, Today

Brush up on the basics or master the advanced techniques required to earn essential industry certifications, with Courses. Enroll in a course and start learning today. Training topics range from Android App Dev to the Xen Virtualization Platform.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A quick way to get a menu to work on our website, is using the Menu control and assign it to a web.sitemap using SiteMapDataSource. Example of web.sitemap file: (CODE) Sample code to add to the page menu: (CODE) Running the application, we wi…
Introduction This article shows how to use the open source plupload control to upload multiple images. The images are resized on the client side before uploading and the upload is done in chunks. Background I had to provide a way for user…
Although Jacob Bernoulli (1654-1705) has been credited as the creator of "Binomial Distribution Table", Gottfried Leibniz (1646-1716) did his dissertation on the subject in 1666; Leibniz you may recall is the co-inventor of "Calculus" and beat Isaac…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …

776 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question