Solved

Storing Large Amounts of Files into Folders - Efficiently - and Naming Convention

Posted on 2006-07-20
5
300 Views
Last Modified: 2006-11-18
I have an application that will allow users to upload photos. Potentially, my application can reach into the 100,000+ user range. I was wondering if anyone knows an efficient way to create the file structure, or naming convention, to hold the photos. I've seen some examples based on UserIDs, Dates, and/or Splitting up the PhotoID into sections, but I'm still not very clear on these techniques. I was also wondering how many files I should store in each folder so that the files can be access on my website efficiently. I've read anywhere betweek 100 to 1,000 files per folder.

Right now I am thinking about storing the photo file on a web server and using Sql Server to store the file reference and file info (i.e. FileID, UserID, Date, FileName, FolderStructure, and perhaps a DomainID to prepare for situations where I will have to use multiple servers to store the photos).

Any help and insight on the topic will be appreciated. Thanks!
0
Comment
Question by:Skytide
  • 3
  • 2
5 Comments
 
LVL 24

Expert Comment

by:Justin_W
ID: 17150558
Based on the info above, I would say that it is probably a bad idea to store the images directly on the file system for such a large number of users/files.

You should probably store the files in a binary/image/BLOB column in a DB (e.g. SQL Server) rather than storing them directly on the filesystem. You could also store the filename, user, etc. in other columns (and index them) to provide efficient access to each file without worrying about folder names, etc. Storing them in a DB would also bypass any potential problems resulting from limits on filename/path lengths (max is 256 in Windows IIRC).
0
 

Author Comment

by:Skytide
ID: 17151321
I would prefer to store the images on a web server since I don't want my database to become too large. I also might have to store images across servers in the future, so I want to avoid having to extract the images from the database when that happens. As a note too, the photos aren't sensitive, so I'm not really worried about the security of them.

Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs. Here is one: http://www.sqlteam.com/item.asp?ItemID=986 However, it's a really confusing topic since it seems like one could go either way.
0
 
LVL 24

Expert Comment

by:Justin_W
ID: 17155440
>> Most (but not all) of the articles and discussions I have read have also discourages storing images in blobs.

There were good reasons for this in SQL Server 7.0 and earlier, but SQL Server 2000 and later have resolved those issues
0
 

Author Comment

by:Skytide
ID: 17161909
Anymore input on this? I want to keep things open in case I decide to use a Content Delivery Network, so I won't be storing the images in the database.
0
 
LVL 24

Accepted Solution

by:
Justin_W earned 500 total points
ID: 17165000
Well, as I mentioned, file systems maximum length limits on filenames. Therefore, you may want to store full metadat info (full filename, dates, folders, etc.) in a DB, and use a GUID (or other reasonably short unique string) as the file system filename. That way, you could easily store and access full file data (including for reporting), but wouldn't have to worry about excessive string parsing or filenames. Also, you could always convert the filenames to something else later using a simple custom app.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

IntroductionWhile developing web applications, a single page might contain many regions and each region might contain many number of controls with the capability to perform  postback. Many times you might need to perform some action on an ASP.NET po…
Problem Hi all,    While many today have fast Internet connection, there are many still who do not, or are connecting through devices with a slower connect, so light web pages and fast load times are still popular.    If your ASP.NET page …
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
With Secure Portal Encryption, the recipient is sent a link to their email address directing them to the email laundry delivery page. From there, the recipient will be required to enter a user name and password to enter the page. Once the recipient …

948 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

23 Experts available now in Live!

Get 1:1 Help Now