Solved

Where do big websites store all their uploads?

Posted on 2012-03-17
7
359 Views
Last Modified: 2012-03-18
Probably kind of a newb question here, but I guess I don't really know the answer.

Sites like YouTube, Facebook, which probably have tens of thousands of uploads to them per day... do they just continually add hard drives to increase storage space for all this? Seems kind of impractical.  I remember back in the day, you were only allowed to upload a certain amount of data per user account. How do they handle the mass amounts of data constantly being uploaded to those sites? It must be thousands of gigs if not terabytes per day. Where's it all being stored?
0
Comment
Question by:Tymetwister
  • 2
  • 2
  • 2
  • +1
7 Comments
 
LVL 3

Accepted Solution

by:
IMIronMan earned 170 total points
ID: 37733423
If I tried to explain it....you wouldn't believe me, so I let you take a look a how Google does it.  It's AMAZING!

http://www.youtube.com/watch?v=zRwPSFpLX8I
0
 
LVL 82

Expert Comment

by:Dave Baldwin
ID: 37733457
That's a cool video.  Here's more info: http://www.google.com/about/datacenters/  If you click on the 'Locations' link, you'll see that Google has at least 10 locations for data centers.  Microsoft, Yahoo, Facebook, Apple, Godaddy and other major internet sites have similar setups.  And there are data centers for people you never heard of.  One data center in Colorado has 1,000,000 sq ft of floor space and room for 262,000 servers and of course, all that support equipment to run them.
0
 
LVL 8

Author Comment

by:Tymetwister
ID: 37733537
I guess I'm still not absorbing the heart of what I was asking about. So... it pretty much is just adding an unthinkable amount of HDD's and storage space to house all of it in datacenters?
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 82

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 130 total points
ID: 37733640
Yep.  That Google video said it held 45,000 servers and each one would be a new machine in I think they said 2009.  That probably means 500GB or larger drives in each server.  And that is only 1 of 10 locations that they have around the world.  So some quick arithmetic comes to at least 450,000 hard drives with at least 22,000 Terabytes... and that is Just Google (Youtube is part of Google now).  Granted, it takes a while to install all those servers so you could say it just keeps on growing.  I wonder how long their servers last and how often they replace them with new machines.

You can Google info on the data centers for most of the large organizations.
0
 
LVL 32

Expert Comment

by:shalomc
ID: 37734767
And don't forget that some of theses datacenters are used for cloud based storage - they rent out storage to anyone who wants it. Check out Amazon, Google, Microsoft Azure, RackSpace, Gogrid, Xerox, and others....

That's how many projects who need tons of storage start, when the risk does not justify building their own datacenters.
0
 
LVL 3

Assisted Solution

by:IMIronMan
IMIronMan earned 170 total points
ID: 37734885
And we keep getting bigger drive technology....Check out this news story:

A data repository almost 10 times bigger than any made before is being built by researchers at IBM's Almaden, California, research lab. The 120 petabyte "drive"—that's 120 million gigabytes—is made up of 200,000 conventional hard disk drives working together. The giant data container is expected to store around one trillion files and should provide the space needed to allow more powerful simulations of complex systems, like those used to model weather and climate.

A 120 petabyte drive could hold 24 billion typical five-megabyte MP3 files or comfortably swallow 60 copies of the biggest backup of the Web, the 150 billion pages that make up the Internet Archive's WayBack Machine.
0
 
LVL 8

Author Closing Comment

by:Tymetwister
ID: 37735318
All very interesting facts. I feel like I have a better understanding of how it works now. Thanks all.
0

Featured Post

ScreenConnect 6.0 Free Trial

Want empowering updates? You're in the right place! Discover new features in ScreenConnect 6.0, based on partner feedback, to keep you business operating smoothly and optimally (the way it should be). Explore all of the extras and enhancements for yourself!

Join & Write a Comment

Microservice architecture adoption brings many advantages, but can add intricacy. Selecting the right orchestration tool is most important for business specific needs.
Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
The purpose of this video is to demonstrate how to set up the WordPress backend so that each page automatically generates a Mailchimp signup form in the sidebar. This will be demonstrated using a Windows 8 PC. Tools Used are Photoshop, Awesome…
Connecting to an Amazon Linux EC2 Instance from Windows Using PuTTY.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now