How does Flickr and Facebook handle Pictures

Posted on 2010-01-06
Last Modified: 2012-05-08
Hello Experts,
 How does the big boys handle pictures of web?
1. Do they write the blob to a sql db?
2) Do the the build a folder for each user and store the pix on disk aas well as store an index to that directory/ picture in the sql db?

3) I am using a Hosting Company with a Virtual Server any suggestions of a larger outfit that may handle the large banwidth?  Or do you think that most hosting services are ok? I am anticiapating 40k users.

I am using C#, and SQL 2008.

I want to allow my users to add pictures. Any amount of pix but not larger than 10 mb.
is AJAX the way to go for File Uploads/Store?

I realize these questions are pedestrian, but I am working alone and have noone now to bounce these dumb questions off of.



Question by:nowmbie
    LVL 4

    Assisted Solution

    For both security and performance, you should consider a dedicated server with that many users, especially if a large percentage will be using it simultaneously.

    I don't really know, but I'd be willing to bet the big guys use databases for images.  Many sites use AJAX for their upload setups because it looks fancier and can be customized more than a simple "browse" button will give you, such as thumbnails with checkboxes.  I notice the same sites let you opt out for a simpler upload interface.
    LVL 29

    Accepted Solution

    1. To db is the only way it could possibly scale. File access is slow and inherently risky from a security standpoint, although they could dump the uploaded files onto the file system before streaming the binary data to the db. Much the way e-mails are dumped to an outbound directory by one process and actually sent out by another process. That would prevent the upload from bogging down under load, and allow anti-virus anti-malware to check the files before dumping them in a db.

    2. I'd be looking for a dedicated server with the thought of mirroring the photo data across many servers dispersed geographically. This is how larger companies handle large repositories of binary data which will be accessed by a large audience. In fact, there are media hosting operations which just handle the data for other web sites.  Inktomi for example acts as a a proxy cache for web traffic and on-demand streaming media.  (Images, Ads, and audio.) Yahoo bought them and they are used by many hosts to serve media to users from the closest server.

    3. At far less than 40K users, a financial services site I developed (read project lead with a team of programmers) also had the web server mirrored at three locations in the US, New York, Texas, and California.  From my point of view it was one site to maintiain, but the company paid a pretty penny for the hosting provider to keep the dedicated servers mirrored and running.

    The right web hosting provider (notice how I avoid a direct recommendation) will be happy to provide a copy of their SAS 70 certification and at least some clients of similar size as your proposed operation for you to contact for reference. They won't give you their entire client portfolio, but should have some clients you can contact.  They will also negotiate their prices for a sizable web operation, but may require an up front long term capital commitment to negotiate the best prices.

    Sounds like more than a one person development project to me.  

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    Course: Create Mobile App Prototypes with Adobe XD

    This is a project-based course: we go through all the steps of creating a prototype from start to finish, using all the tools and features currently available in Adobe XD. You can complete the course in less than a day, plus all project files and fonts are included.

    Foreword (July, 2015) Since I first wrote this article, years ago, a great many more people have begun using the internet.  They are coming online from every part of the globe, learning, reading, shopping and spending money at an ever-increasing ra…
    Using SQL Scripts we can save all the SQL queries as files that we use very frequently on our database later point of time. This is one of the feature present under SQL Workshop in Oracle Application Express.
    This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
    The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.

    737 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    21 Experts available now in Live!

    Get 1:1 Help Now