vitorenesduarte / ldb Goto Github PK

View Code? Open in Web Editor NEW

38.0 38.0 1.0 2 MB

Replication of CRDTs

License: Apache License 2.0

Makefile 2.33% Erlang 97.27% Shell 0.40%

crdt replication

ldb's People

Contributors

Stargazers

Watchers

Forkers

castellanprime

ldb's Issues

In state-based replication, don't transmit unless there was a mutation since last transmission

Problems in pure-op-based mode

$ LDB_MODE=pure_op_based make shell

If I do this and wait long enough, I see:

[warning] Unhandled info message: check_resend

Ignoring that, I can't use a gcounter:

1> ldb:create(a, gcounter).
ok
2> 2> ldb:update(a, increment).
17:35:15.028 [error] gen_server ldb_pure_op_based_backend terminated with reason: no function clause matching ldb_pure_op_based_backend:encode_op(increment) line 415
17:35:15.041 [error] CRASH REPORT Process ldb_pure_op_based_backend with 1 neighbours exited with reason: no function clause matching ldb_pure_op_based_backend:encode_op(increment) line 415 in gen_server:terminate/7 line 826
** exception exit: {{function_clause,
                        [{ldb_pure_op_based_backend,encode_op,
                             [increment],
                             [{file,
                                  "/home/user/Downloads/ldb/_build/default/lib/ldb/src/ldb_pure_op_based_backend.erl"},
                              {line,415}]},
                         {ldb_pure_op_based_backend,handle_call,3,
                             [{file,
                                  "/home/user/Downloads/ldb/_build/default/lib/ldb/src/ldb_pure_op_based_backend.erl"},
                              {line,154}]},
                         {gen_server,try_handle_call,4,
                             [{file,"gen_server.erl"},{line,629}]},
                         {gen_server,handle_msg,5,
                             [{file,"gen_server.erl"},{line,661}]},
                         {proc_lib,init_p_do_apply,3,
                             [{file,"proc_lib.erl"},{line,240}]}]},
                    {gen_server,call,
                        [ldb_pure_op_based_backend,
                         {update,a,increment},
                         infinity]}}
     in function  gen_server:call/3 (gen_server.erl, line 212)

Changing encode_op and decode_op seems to solve it.

%% @private
encode_op({add, E}) ->
    {1, E};
encode_op(Op) ->
    Op.

%% @private
decode_op({1, E}) ->
    {add, E};
decode_op(Op) ->
    Op.

About the check_send, it seems that its handler is handle_cast and it should be handle_info.
However, doing that change, seems to break the ldb:update.

Implement DCOS barrier to make sure every tasks starts at the same time

Implement delta buffer shrinking

We should provide two shrinking methods:

only remove entries acknowledged by all peers
clear the buffer (provoking new sync for every pair of peers)

Shrink buffer when the set of neighbors changes in delta-based backend

Before #44 this was done periodically,
but since now we shrink the buffer on every ack,
it does not make sense to still shrink it periodically
(only when the set of neighbors changes)

Implement memory report for pure op based backend

Refactor Scuttlebutt backends

Refactor scuttlebutt and vanilla_ scuttlebutt into a single backend and have garbage collection be a performance optimization as BP and RR.
Also, the delta-buffer in this backend could have its own module, as the buffer in the delta-based backend.

If delta-buffer grows indefinitely, evict peer(s)

What's a good peer eviction strategy?

In pure op based, implement a causal stability mechanism

this mechanism should run in the bg
and periodically asks ishikawa for the stable vector
then folds all the store (per key i.e. POLog) and for each op in the POLog: 1) check if stable 2) if yes stabilize, else continue

any reason not to use partisan?

I was wondering why you were not using partisan for the peer service. Any reason for it?

vitorenesduarte / ldb Goto Github PK

ldb's People

Contributors

Stargazers

Watchers

Forkers

ldb's Issues

In state-based replication, don't transmit unless there was a mutation since last transmission

Problems in pure-op-based mode

Implement DCOS barrier to make sure every tasks starts at the same time

Implement delta buffer shrinking

Shrink buffer when the set of neighbors changes in delta-based backend

Implement memory report for pure op based backend

Refactor Scuttlebutt backends

If delta-buffer grows indefinitely, evict peer(s)

In pure op based, implement a causal stability mechanism

any reason not to use partisan?

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent