how does facebook / twitter store data?

What are some of the technologies that facebook / twitter use to store and search their massive data graphs?
I'm looking for something that will have capabilities to search massive amounts of data that will be growing constantly.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Russell FoxDatabase DeveloperCommented:
That's...a big question. The most common solution is a "NoSql" database: graph, document, wide column, object, etc. Take a look a for a list of NoSQL databases out there, each with plusses and minuses. I'm currently learning about MongoDB, a JSON document store, for a project at work. The basic idea is that instead of having to buy better and better (i.e. more expensive) servers to run your relational database, you have a cluster of cheap(er) servers, all connected and talking to each other, with multiple copies of the same data spread all over the cluster. If one machine catches on fire, the rest just keep working and no data is lost. If your startup suddenly has a 500% increase in customers, you just add more machines and they automatically divvy up the data and processing needs. It's kind of a rabbit hole right now because dozens of vendors all want to corner the market.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
SquadlessAuthor Commented:
customer % increase will not be significant, but i will be holding transactional data for years.  There will be a need to - at any point in time - search the entire tree for a specific transaction or detail of it, there are about 50k transactions per day currently but that will be going up exponentially.
Russell FoxDatabase DeveloperCommented:
Another consideration then is data compression: I know Mongo's latest version has very good compression to keep the documents small on disk, which not only saves space but reduces i/o because the drive heads don't have to move as much. The trade-off is that the CPU has to work harder to compress/decompress the data. I'm sure most of the other solutions have similar capabilities. You'll also want to be very conscientious about how you structure your data - whenever you're thinking about how to store it, first consider how you're going to be retrieving it. That will help direct your indexing strategy.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.