Solved

Where do big websites store all their uploads?

Posted on 2012-03-17
7
377 Views
Last Modified: 2012-03-18
Probably kind of a newb question here, but I guess I don't really know the answer.

Sites like YouTube, Facebook, which probably have tens of thousands of uploads to them per day... do they just continually add hard drives to increase storage space for all this? Seems kind of impractical.  I remember back in the day, you were only allowed to upload a certain amount of data per user account. How do they handle the mass amounts of data constantly being uploaded to those sites? It must be thousands of gigs if not terabytes per day. Where's it all being stored?
0
Comment
Question by:Tymetwister
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
  • 2
  • +1
7 Comments
 
LVL 3

Accepted Solution

by:
IMIronMan earned 170 total points
ID: 37733423
If I tried to explain it....you wouldn't believe me, so I let you take a look a how Google does it.  It's AMAZING!

http://www.youtube.com/watch?v=zRwPSFpLX8I
0
 
LVL 83

Expert Comment

by:Dave Baldwin
ID: 37733457
That's a cool video.  Here's more info: http://www.google.com/about/datacenters/  If you click on the 'Locations' link, you'll see that Google has at least 10 locations for data centers.  Microsoft, Yahoo, Facebook, Apple, Godaddy and other major internet sites have similar setups.  And there are data centers for people you never heard of.  One data center in Colorado has 1,000,000 sq ft of floor space and room for 262,000 servers and of course, all that support equipment to run them.
0
 
LVL 8

Author Comment

by:Tymetwister
ID: 37733537
I guess I'm still not absorbing the heart of what I was asking about. So... it pretty much is just adding an unthinkable amount of HDD's and storage space to house all of it in datacenters?
0
Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.

 
LVL 83

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 130 total points
ID: 37733640
Yep.  That Google video said it held 45,000 servers and each one would be a new machine in I think they said 2009.  That probably means 500GB or larger drives in each server.  And that is only 1 of 10 locations that they have around the world.  So some quick arithmetic comes to at least 450,000 hard drives with at least 22,000 Terabytes... and that is Just Google (Youtube is part of Google now).  Granted, it takes a while to install all those servers so you could say it just keeps on growing.  I wonder how long their servers last and how often they replace them with new machines.

You can Google info on the data centers for most of the large organizations.
0
 
LVL 33

Expert Comment

by:shalomc
ID: 37734767
And don't forget that some of theses datacenters are used for cloud based storage - they rent out storage to anyone who wants it. Check out Amazon, Google, Microsoft Azure, RackSpace, Gogrid, Xerox, and others....

That's how many projects who need tons of storage start, when the risk does not justify building their own datacenters.
0
 
LVL 3

Assisted Solution

by:IMIronMan
IMIronMan earned 170 total points
ID: 37734885
And we keep getting bigger drive technology....Check out this news story:

A data repository almost 10 times bigger than any made before is being built by researchers at IBM's Almaden, California, research lab. The 120 petabyte "drive"—that's 120 million gigabytes—is made up of 200,000 conventional hard disk drives working together. The giant data container is expected to store around one trillion files and should provide the space needed to allow more powerful simulations of complex systems, like those used to model weather and climate.

A 120 petabyte drive could hold 24 billion typical five-megabyte MP3 files or comfortably swallow 60 copies of the biggest backup of the Web, the 150 billion pages that make up the Internet Archive's WayBack Machine.
0
 
LVL 8

Author Closing Comment

by:Tymetwister
ID: 37735318
All very interesting facts. I feel like I have a better understanding of how it works now. Thanks all.
0

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

The business world is becoming increasingly integrated with tech. It’s not just for a select few anymore — but what about if you have a small business? It may be easier than you think to integrate technology into your small business, and it’s likely…
Many businesses neglect disaster recovery and treat it as an after-thought. I can tell you first hand that data will be lost, hard drives die, servers will be hacked, and careless (or malicious) employees can ruin your data.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

749 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question