It all depends. Is your data easily sharded? Do you expect to have lots of small files? Do you have expect to have your data as lots of large text files or blobs? Do you need secondary indexes or will a simple key-value store be enough? Which languages do you need to be able to access your data using? Do you need low-latency access or will this be used for large-scale data analysis via batch jobs?
Don't forget that unfashionable as it is, you still have the option of using replicated mysql with a memcached layer in front of it.
If you have enough time then grab a reasonably realistic sample of your data and try out the leading contenders. If you define your metrics in advance you can see which of these score highest on all the metrics you care about.
I want to store hundreds of millions of key–value pairs in a scalable fashion. Keys would typically be no longer than 100 bytes, values a few kilobytes. I want to look up values by their keys very fast. No advanced queries need.
I want to be able to add machines if I need to make more frequent lookups, and ideally it should work in a multi-site setup, though I guess the latter might be pushing it.
Mostly I'm trying to get a feeling for what's out there. Don't want to reinvent the wheel. :)
@aehn The above sounds like a description of a distributed hashtable which uses consistent hashing to dynamically re-balance the shards and serves most requests from main memory.
That suggests MemCache or MemCacheDb (even though I think building a database on top of a cache is an abstraction-invertion that will cause problems in the future) might be your best solutions although I would still suggest looking at the possibility of using Project Voldemoort: http://project-voldemort.com/design.php with an in-memory configuration and Scalaris: http://code.google.com/p/scalaris/
The disadvantages of this approach would be: spending lots of money on memory, starting up the system is going to take ages as you transfer the data from disk to ram, choosing the replica count is going to an interesting trade-off between redundancy/availability/cost, this is likely to push you towards having big+expensive machines. Whatever you choose you're going to need a test bed in which to try out various configurations so you might as well use that test bed to verify the various distributed hash tables. Don't forget to test how these systems work with different usage patterns and different sized datasets.
18 comments so far
+1 for CouchDB
9 months, 1 week ago by flosoft
Why?
9 months, 1 week ago by aehn
This sums it up: http://pylab.blogspot.com/2009/01/ten-reasons-why-couchdb-is-better-than.html
9 months, 1 week ago by flosoft
"Uses Erlang, which means it has magical powers", apparently. Marvellous. Nothing like a good bit of hype.
9 months, 1 week ago by dwd
Well, that compares CouchDB to an RDBMS (MySQL), not to other non-relational databases. :)
9 months, 1 week ago by aehn
@dwd: Yeah. :)
9 months, 1 week ago by aehn
What are you trying to do? Is a non-SQL database the best fit? Have you overgrown MySQL and Postgres?
9 months, 1 week ago by melo
I am 100% in favor of anything called MongoDB.
9 months, 1 week ago by sarnesjo
MemcacheDB?
9 months, 1 week ago by osks
Tokyo Cabinet (just heard of it, don't know much)? http://tokyocabinet.sourceforge.net/
9 months, 1 week ago by osks
Also: Anti-RDBMS: A list of distributed key-value stores
9 months, 1 week ago by osks
Maybe http://hadoop.apache.org/hbase/ ? If RDBMs are of intereset, maybe a IBM DB2 on Amazon EC2.
9 months, 1 week ago by Clas
It all depends. Is your data easily sharded? Do you expect to have lots of small files? Do you have expect to have your data as lots of large text files or blobs? Do you need secondary indexes or will a simple key-value store be enough? Which languages do you need to be able to access your data using? Do you need low-latency access or will this be used for large-scale data analysis via batch jobs?
Don't forget that unfashionable as it is, you still have the option of using replicated mysql with a memcached layer in front of it.
If you have enough time then grab a reasonably realistic sample of your data and try out the leading contenders. If you define your metrics in advance you can see which of these score highest on all the metrics you care about.
9 months, 1 week ago by adewale
I want to store hundreds of millions of key–value pairs in a scalable fashion. Keys would typically be no longer than 100 bytes, values a few kilobytes. I want to look up values by their keys very fast. No advanced queries need.
I want to be able to add machines if I need to make more frequent lookups, and ideally it should work in a multi-site setup, though I guess the latter might be pushing it.
Mostly I'm trying to get a feeling for what's out there. Don't want to reinvent the wheel. :)
9 months, 1 week ago by aehn
Maybe MonetDB? Maybe MySQL and HiveDB? Maybe look at what Tailsweep are doing for their log-analysis http://dev.tailsweep.com/
9 months, 1 week ago by Clas
@aehn The above sounds like a description of a distributed hashtable which uses consistent hashing to dynamically re-balance the shards and serves most requests from main memory.
That suggests MemCache or MemCacheDb (even though I think building a database on top of a cache is an abstraction-invertion that will cause problems in the future) might be your best solutions although I would still suggest looking at the possibility of using Project Voldemoort: http://project-voldemort.com/design.php with an in-memory configuration and Scalaris: http://code.google.com/p/scalaris/
The disadvantages of this approach would be: spending lots of money on memory, starting up the system is going to take ages as you transfer the data from disk to ram, choosing the replica count is going to an interesting trade-off between redundancy/availability/cost, this is likely to push you towards having big+expensive machines. Whatever you choose you're going to need a test bed in which to try out various configurations so you might as well use that test bed to verify the various distributed hash tables. Don't forget to test how these systems work with different usage patterns and different sized datasets.
The Facebook article about their usage is also pretty good: http://www.facebook.com/note.php?note_id=39391378919&id=9445547199&index=1
9 months, 1 week ago by adewale
Thanks for all the great input in this thread!
9 months, 1 week ago by aehn
@aehn: You're welcome.
9 months, 1 week ago by sarnesjo