Selecting database for key value pairs of size at least 32 GB

Hi,
I am planning to build a service like shortening url for learning purpose..I was working on it theoritically.
And realized that i would be needing to store approximately 32 GB of data in the long run...
It will be mostly key value pair data  
eg -
1) www.bity.com/abcdefg  |  www.klasjlkajsf.comalskdfjlaksdjflkasjdfl
2) www.bity.com/abcdefg  | www.laksdjflkasjdflkasd.aldskjflkasdjflkasdf//alsdkfjalsdfj

So i was wondering which database will be better.
Dont know how Mysql will behave when total data reaches 32 GB ?
How long will it take to query it..
Is there some theoritical way to find it out. I think putting an index on the first column should be really fast kind of binary search ?
What are the alternatives  or best way ?
I read about redis that seems to be an in memory database so should be really fast. But it probably cannot store more data than the amount of RAM.
is redis feasible when the amount of data is so big like 32 GB ?
Do we need 32 GB of RAM or there are alternatives ??


Thanks
Rohit BajajAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

David FavorLinux/LXD/WordPress/Hosting SavantCommented:
1) So i was wondering which database will be better.

You can use a keystore like Redis + well tuned MariaDB (working MySQL) is sufficient.

2) Dont know how Mysql will behave when total data reaches 32 GB ?

Er... MariaDB (community/free version) can handle 64TB/table, so 32GB size has no effect on operations.

So long as you use InnoDB storage engine (optimized for mostly reads) + use mysqltuner + fix any messages produced.

3) How long will it take to query it..

If you do #2 + use proper indexes, you'll measure SELECT time in milliseconds.

4) Is there some theoretical way to find it out. I think putting an index on the first column should be really fast kind of binary search ?

Guessing == waste of time.

Just setup a database + do a lookup test, then you'll know for sure.

Note: Query time will be very sensitive to your hosting also. If you host on a dedicated server, where you know all other processes running + can control your entire process ecosystem, you throughput (SELECT time) will be far more consistent, then if you run on a shared host with 1000s of other sites... over which you have no control.

5) What are the alternatives  or best way ?

Many alternatives. You just pick one. Many will work well.

Note: If I were developing this as a salable service, I'd stick with MariaDB + I'd also run this service in an LXD container, so entire system can move between machines in a few minutes + work 100% of them time after a move.

6) I read about redis that seems to be an in memory database so should be really fast. But it probably cannot store more data than the amount of RAM.

Redis has drawbacks. The developers were smoking their lawn when they wrote the network stack.

Their networking breaks whenever network namespaces are used, which means Redis breaks anytime you try to run it in a container - LXD, Docker, etc...

If you're planning on this project having a long life cycle, stick with MariaDB.

7) is redis feasible when the amount of data is so big like 32 GB ?

Avoid Redis.

8) Do we need 32 GB of RAM or there are alternatives ??

I think you may have a misconception about memory usage + databases.

If you stick with item #2,  then most used data will live in memory all the time.

If you stick with item #4, using a dedicated server will ensure your disk i/o won't have 1000s of other sites competing for a slice, so you can ignore the memory issue...

Primary memory consideration...

Enough memory to tune using mysqltuner, with no warnings about memory overruns + enough memory to run without swapping.

The way you know these numbers is to run your system in real time, knowing over time these numbers will change + if you're using LXD, you'll just spin up a new machine with more memory + move your LXD container to the new machine + retire the old machine.

Note: If you'll be running this project to pay your bills, likely best to hire an old dog (someone who's worked on systems like this for years) to do your design.
1
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
redis

From novice to tech pro — start learning today.