how does facebook / twitter store data?

What are some of the technologies that facebook / twitter use to store and search their massive data graphs?
I'm looking for something that will have capabilities to search massive amounts of data that will be growing constantly.
Who is Participating?
Russell FoxConnect With a Mentor Database DeveloperCommented:
That's...a big question. The most common solution is a "NoSql" database: graph, document, wide column, object, etc. Take a look a for a list of NoSQL databases out there, each with plusses and minuses. I'm currently learning about MongoDB, a JSON document store, for a project at work. The basic idea is that instead of having to buy better and better (i.e. more expensive) servers to run your relational database, you have a cluster of cheap(er) servers, all connected and talking to each other, with multiple copies of the same data spread all over the cluster. If one machine catches on fire, the rest just keep working and no data is lost. If your startup suddenly has a 500% increase in customers, you just add more machines and they automatically divvy up the data and processing needs. It's kind of a rabbit hole right now because dozens of vendors all want to corner the market.
SquadlessAuthor Commented:
customer % increase will not be significant, but i will be holding transactional data for years.  There will be a need to - at any point in time - search the entire tree for a specific transaction or detail of it, there are about 50k transactions per day currently but that will be going up exponentially.
Russell FoxConnect With a Mentor Database DeveloperCommented:
Another consideration then is data compression: I know Mongo's latest version has very good compression to keep the documents small on disk, which not only saves space but reduces i/o because the drive heads don't have to move as much. The trade-off is that the CPU has to work harder to compress/decompress the data. I'm sure most of the other solutions have similar capabilities. You'll also want to be very conscientious about how you structure your data - whenever you're thinking about how to store it, first consider how you're going to be retrieving it. That will help direct your indexing strategy.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.