basho-labs / hanoidb Goto Github PK

Although Snappy compression has been integrated sometimes compression fails in the snappy erlang integration libraries we use (https://github.com/fdmanana/snappy-erlang-nif).

Right now the merge of the first level after the nursery is triggered when the nursery fills up. This is generally okay, but when you have a very fast IO system (SSDs for instance) you can stall operations on the nursery as you do that work to begin merging.

A better approach (design idea, but feel free to innovate in other ways) would be to trigger merges in the first level after the nursery when you reach some percentage full in the nursery (this could even be adaptive). This opens up space to later merge the nursery into the first level once it fills up. Ideally you never have the situation where you're waiting for the first post-nursery level to merge with the next level when the nursery fills up.

Just like bitcask, HanoiDB should have a mechanism to evict KVs after a given time period.

What would be really sweet was a 'atomic' update function that combined get and put into one call without the risk of having something happen in between.

Right now, HanoiDB uses almost exclusively KV counts as measure of progress and for controlling "enough progress". If the bottleneck of HanoiDB is IO (as it should be) then it should make things more smooth to base progress control on byte counts.

Up until now, performance measurements have been based on basho_bench runs that have equal key and value sizes. If KV sizes vary greatly, then KV count is not a good measure for how much progress the incremental merge needs to do in order to keep up with an "even" response time.

Perhaps if we have numbers for average KV sizes, we can impose on the caller that an insert of a large KV requires a larger incremental step, than insert of a smaller KV.

Hi,

I'm looking possibly for a pure erlang solution for an application I want to build and I wonder how hanoidb actually compares to eleveldb. Is it usable in production? What is missing?

Hello,

I'm somewhat cross posting this issue from krestenkrab/hanoidb (issue krestenkrab#20 there).

I'd like to read some feedback about use cases of Hanoidb backed Riak clusters.
Is it still experimental ? Or has it been stable enough for month to call it production ready ?
How does it do with Riak features ? KV, MR, 2i, ops friendly...
How much does it achieve about performance ? cluster size, node specs, concurrency patterns, iops, latency, flowers ?

To sum up : Are there some success stories to be shared ? Which use cases, which numbers ?

BTW thanks for the backend alternativeness.

When a database is aggressively removing data (expiry or delete) the predictable strategy for merge step size is not enough to prevent stalls. More information and a repeatable basho_bench config in this gist: https://gist.github.com/92c25633d61abdac8fec