mrcroxx / runkv Goto Github PK
View Code? Open in Web Editor NEW[WIP] An experimental cloud-native distributed KV engine for OLTP workload.
License: MIT License
[WIP] An experimental cloud-native distributed KV engine for OLTP workload.
License: MIT License
Now rpc client connects to the server when building, which requires to build each components in order in integration test. The ideal way is avoid connecting when building, and connects when bootstrap.
As titled.
As in #74 , create or clone Bytes
involves atomic operation, whose costs cannot be ignored if performs frequently (e.g. in iterators). Unnecessary Bytes
should be replaced with Vec<u8>
or &[u8]
.
Since the QPS dropped sharply after integrating raft WAL, we need to find the reason for the decrease.
RunKV uses object store as part of its storage backend. S3 object store and memory object store (for tests) are needed.
ObjectStore
trait.MemoryObjectStore
.S3ObjectStore
.A not-exist sst is tend to be deleted after comapction.
2022-05-16T15:13:15.277677Z ERROR runkv_rudder::worker::compaction_detector: trigger compaction l0 error: other: compaction error: [Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697679 not exists, diff: VersionDiff { id: 696, sstable_diffs: [SstableDiff { id: 68719476789, level: 0, op: Delete, data_size: 68762 }, SstableDiff { id: 68719476790, level: 0, op: Delete, data_size: 67887 }, SstableDiff { id: 68719476791, level: 0, op: Delete, data_size: 69175 }, SstableDiff { id: 68719476792, level: 0, op: Delete, data_size: 68799 }, SstableDiff { id: 433791697679, level: 1, op: Delete, data_size: 33857 }, SstableDiff { id: 433791697684, level: 1, op: Delete, data_size: 33266 }, SstableDiff { id: 433791697689, level: 1, op: Delete, data_size: 34152 }, SstableDiff { id: 433791697695, level: 1, op: Delete, data_size: 33991 }, SstableDiff { id: 433791697699, level: 1, op: Delete, data_size: 33625 }, SstableDiff { id: 433791697704, level: 1, op: Delete, data_size: 34298 }, SstableDiff { id: 433791697708, level: 1, op: Delete, data_size: 3211975 }, SstableDiff { id: 433791697750, level: 1, op: Insert, data_size: 36418 }, SstableDiff { id: 433791697757, level: 1, op: Insert, data_size: 35829 }, SstableDiff { id: 433791697764, level: 1, op: Insert, data_size: 36713 }, SstableDiff { id: 433791697768, level: 1, op: Insert, data_size: 36553 }, SstableDiff { id: 433791697772, level: 1, op: Insert, data_size: 36059 }, SstableDiff { id: 433791697776, level: 1, op: Insert, data_size: 36856 }, SstableDiff { id: 433791697779, level: 1, op: Insert, data_size: 3458003 }] }")))), Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697685 not exists, diff: VersionDiff { id: 695, sstable_diffs: [SstableDiff { id: 107374182453, level: 0, op: Delete, data_size: 68025 }, SstableDiff { id: 107374182454, level: 0, op: Delete, data_size: 68593 }, SstableDiff { id: 107374182455, level: 0, op: Delete, data_size: 68662 }, SstableDiff { id: 107374182456, level: 0, op: Delete, data_size: 68806 }, SstableDiff { id: 433791697680, level: 1, op: Delete, data_size: 34008 }, SstableDiff { id: 433791697685, level: 1, op: Delete, data_size: 34511 }, SstableDiff { id: 433791697690, level: 1, op: Delete, data_size: 34004 }, SstableDiff { id: 433791697694, level: 1, op: Delete, data_size: 34295 }, SstableDiff { id: 433791697701, level: 1, op: Delete, data_size: 33997 }, SstableDiff { id: 433791697705, level: 1, op: Delete, data_size: 34002 }, SstableDiff { id: 433791697709, level: 1, op: Delete, data_size: 34281 }, SstableDiff { id: 433791697713, level: 1, op: Delete, data_size: 33783 }, SstableDiff { id: 433791697715, level: 1, op: Delete, data_size: 34362 }, SstableDiff { id: 433791697717, level: 1, op: Delete, data_size: 3109571 }, SstableDiff { id: 433791697752, level: 1, op: Insert, data_size: 36841 }, SstableDiff { id: 433791697759, level: 1, op: Insert, data_size: 37365 }, SstableDiff { id: 433791697762, level: 1, op: Insert, data_size: 36705 }, SstableDiff { id: 433791697767, level: 1, op: Insert, data_size: 36928 }, SstableDiff { id: 433791697770, level: 1, op: Insert, data_size: 36710 }, SstableDiff { id: 433791697775, level: 1, op: Insert, data_size: 36565 }, SstableDiff { id: 433791697778, level: 1, op: Insert, data_size: 36848 }, SstableDiff { id: 433791697781, level: 1, op: Insert, data_size: 36345 }, SstableDiff { id: 433791697782, level: 1, op: Insert, data_size: 36855 }, SstableDiff { id: 433791697784, level: 1, op: Insert, data_size: 3346709 }] }")))), Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697692 not exists, diff: VersionDiff { id: 695, sstable_diffs: [SstableDiff { id: 120259084341, level: 0, op: Delete, data_size: 68268 }, SstableDiff { id: 120259084342, level: 0, op: Delete, data_size: 68832 }, SstableDiff { id: 120259084343, level: 0, op: Delete, data_size: 68050 }, SstableDiff { id: 120259084344, level: 0, op: Delete, data_size: 69146 }, SstableDiff { id: 433791697681, level: 1, op: Delete, data_size: 33993 }, SstableDiff { id: 433791697688, level: 1, op: Delete, data_size: 33634 }, SstableDiff { id: 433791697692, level: 1, op: Delete, data_size: 34287 }, SstableDiff { id: 433791697696, level: 1, op: Delete, data_size: 33994 }, SstableDiff { id: 433791697700, level: 1, op: Delete, data_size: 34297 }, SstableDiff { id: 433791697703, level: 1, op: Delete, data_size: 33773 }, SstableDiff { id: 433791697707, level: 1, op: Delete, data_size: 33855 }, SstableDiff { id: 433791697710, level: 1, op: Delete, data_size: 34293 }, SstableDiff { id: 433791697712, level: 1, op: Delete, data_size: 33852 }, SstableDiff { id: 433791697714, level: 1, op: Delete, data_size: 34500 }, SstableDiff { id: 433791697716, level: 1, op: Delete, data_size: 3075627 }, SstableDiff { id: 433791697754, level: 1, op: Insert, data_size: 36557 }, SstableDiff { id: 433791697758, level: 1, op: Insert, data_size: 36345 }, SstableDiff { id: 433791697763, level: 1, op: Insert, data_size: 36849 }, SstableDiff { id: 433791697766, level: 1, op: Insert, data_size: 36421 }, SstableDiff { id: 433791697771, level: 1, op: Insert, data_size: 36855 }, SstableDiff { id: 433791697774, level: 1, op: Insert, data_size: 36415 }, SstableDiff { id: 433791697777, level: 1, op: Insert, data_size: 36416 }, SstableDiff { id: 433791697780, level: 1, op: Insert, data_size: 36926 }, SstableDiff { id: 433791697783, level: 1, op: Insert, data_size: 36562 }, SstableDiff { id: 433791697785, level: 1, op: Insert, data_size: 37073 }, SstableDiff { id: 433791697786, level: 1, op: Insert, data_size: 3311271 }] }"))))]
#149 may be related to this bug.
We need an action to check if the title and commit log of each PR obey some rules.
SSTable id should be globally unique. To simplify the sstable id generation logic, each generate sstable id based on its node id and a persisted sequential number.
group
in raft log store means log group, which is actually raft node id in the concept of Raft, not raft group
in multi-raft.
So it's better to rename group
in raft log store with another name (e.g. collection, set, ..?) to avoid misleading.
A PR template is needed to offer to contributors to help them write good PR messages.
Tracking:
SstableStore
and ObjectStore
).To balance the large total data volume, the low capacity of memory and the high latency of S3, RunKV can use the remaining space of the disk as secondary cache, or File Cache.
There are some considerations about File Cache:
Ref: #44 .
Tracking:
When running test_multi_raft_group_concurrent_put_get
with 100 loops for each group, sometimes assert failed.
e.g.
thread 'test_multi_raft_group_concurrent_put_get::test_multi_raft_group_concurrent_put_get' panicked at 'assertion failed: `(left == right)`
left: `[118, 52, 52, 52, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 48]`,
right: `[]`', tests/integrations/test_multi_raft_group_concurrent_put_get.rs:214:21
L214 asserts the result of get
after put
RunKV/tests/integrations/test_multi_raft_group_concurrent_put_get.rs
Lines 184 to 225 in b0d596c
which should be None
but got unexpected result.
When digging into the trace log, I found that the delete
was not applied:
[2022-05-07T06:16:17Z TRACE runkv_wheel::components::lsm_tree] put; self=ObjectStoreLsmTree { raft_node: 52 } key=b"k444 0" value=b"v444 0" sequence=36656013143 apply_index=6530
[2022-05-07T06:16:18Z TRACE runkv_wheel::components::lsm_tree] get; self=ObjectStoreLsmTree { raft_node: 52 } key=b"k444 0" sequence=36656013147
Ref: #30
The expected behavior of SeekRandom
is move to the exact given key or the "next" key. "next" here actually means "prev" if the iterator is expected to be move backward.
Especially for MergeIterator
. It's useful for changing the direction of it correctly.
As titled, see cargo-binstall .
Tracking
The write model of raft log store is simple now. Currently, it cannot utilize the benefits of batching and pipelining. A better write model like RocksDB or something is needed.
Tracking:
For each raft group, log entries before compact index
can be safely deleted. Although log entires are continuously written in most cases, but log entries of various groups are cross written.
e.g.
file 1: | group 1, 0 - 100 | group 2, 0 - 50 | group 1, 101 - 200 | group 2, 51 - 100 |
As a result, with the system running, some parts of some log file can be purged but the other parts can not (e.g. 90% can be purged in file 1, 50% can be purged in file 2, we call the rate that can be purged of a file utilization rate).
So we can track the (approximate) utilization rate of each file, when the utilization rate of some files drops below the threshold
, we need to rewrite the file (append the still needed part to the current active log file and update the memory indices of them) and remove the old log file. The rewrite operation is safe because we recorded term
of the raft log and we can easily distinguish if log entries with the same index can overwrite the current one.
The rewrite procedure is asynchronous, and the follow aspects should be taken into consideration:
As titled. Drunk.
Sometimes, fn read()
call of Log
leads to early EOF error. Maybe caused by data race when rotating active log file.
thread 'tokio-runtime-worker' panicked at 'unexpected error: Store(Other(StorageError(IoError(Custom { kind: UnexpectedEof, error: "early eof" })))), raft_id: 2, raft node: 2, group: 1, namespace: raft', /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raft_log.rs:624:26
stack backtrace:
0: rust_begin_unwind
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panicking.rs:142:14
2: raft::raft_log::RaftLog<T>::slice::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raft_log.rs:624:26
3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
4: raft::raft_log::RaftLog<T>::next_entries_since::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raft_log.rs:445:18
5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
6: raft::raw_node::RawNode<T>::gen_light_ready::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raw_node.rs:458:67
7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
8: raft::raw_node::RawNode<T>::ready::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raw_node.rs:553:42
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
10: runkv_wheel::worker::raft::RaftWorker<RN,F>::handle_ready::{{closure}}::{{closure}}
at ./wheel/src/worker/raft.rs:503:42
11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
12: runkv_wheel::worker::raft::RaftWorker<RN,F>::handle_ready::{{closure}}
at ./wheel/src/worker/raft.rs:499:5
13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
14: runkv_wheel::worker::raft::RaftWorker<RN,F>::run_inner::{{closure}}
at ./wheel/src/worker/raft.rs:421:36
15: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
16: <runkv_wheel::worker::raft::RaftWorker<RN,F> as runkv_common::Worker>::run::{{closure}}
at ./wheel/src/worker/raft.rs:289:35
17: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
18: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/future.rs:124:9
19: runkv_wheel::components::raft_manager::RaftManager::create_raft_node::{{closure}}::{{closure}}
at ./wheel/src/components/raft_manager.rs:176:43
20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
21: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
22: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
23: tokio::runtime::task::core::CoreStage<T>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
24: tokio::runtime::task::harness::poll_future::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
25: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panic/unwind_safe.rs:271:9
26: std::panicking::try::do_call
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:492:40
27: __rust_try
28: std::panicking::try
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:456:19
29: std::panic::catch_unwind
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panic.rs:137:14
30: tokio::runtime::task::harness::poll_future
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
31: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
32: tokio::runtime::task::harness::Harness<T,S>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
33: tokio::runtime::task::raw::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
34: tokio::runtime::task::raw::RawTask::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
35: tokio::runtime::task::LocalNotified<S>::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:347:9
36: tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:425:13
37: tokio::coop::with_budget::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
38: std::thread::local::LocalKey<T>::try_with
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/thread/local.rs:442:16
39: std::thread::local::LocalKey<T>::with
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/thread/local.rs:418:9
40: tokio::coop::with_budget
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
41: tokio::coop::budget
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
42: tokio::runtime::thread_pool::worker::Context::run_task
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:424:9
43: tokio::runtime::thread_pool::worker::Context::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:391:24
44: tokio::runtime::thread_pool::worker::run::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:376:17
45: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
46: tokio::runtime::thread_pool::worker::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:373:5
47: tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:352:45
48: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/task.rs:42:21
49: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
50: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
51: tokio::runtime::task::core::CoreStage<T>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
52: tokio::runtime::task::harness::poll_future::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
53: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panic/unwind_safe.rs:271:9
54: std::panicking::try::do_call
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:492:40
55: __rust_try
56: std::panicking::try
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:456:19
57: std::panic::catch_unwind
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panic.rs:137:14
58: tokio::runtime::task::harness::poll_future
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
59: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
60: tokio::runtime::task::harness::Harness<T,S>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
61: tokio::runtime::task::raw::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
62: tokio::runtime::task::raw::RawTask::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
63: tokio::runtime::task::UnownedTask<S>::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:384:9
64: tokio::runtime::blocking::pool::Task::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:91:9
65: tokio::runtime::blocking::pool::Inner::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:308:17
66: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:288:17
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
As in #65 , components should establish RPC connections when bootstrapping instead of building. And connection to the same node should be aggregated together. A connection pool can handle the requirements gracefully.
For a fast MVP, there are lots of unnecessary buffer copies in raft log store. Refactor needed.
When porting block and block iterator from RunKV to RisingWave, found some unnecessary slice copy influenced the performance of block iterator a lot. We can also avoid them in RunKV.
For more detailes, see risingwave#1138.
Ref: #37
When running bench_kv
in release mode, found the following panic:
left: `[]`,
right: `[118, 52, 51, 57, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 48]`', bench/bench_kv/main.rs:206:21
stack backtrace:
thread 'tokio-runtime-worker' panicked at 'assertion failed: `(left == right)`
left: `2`,
right: `1`: result: [863288426616, 863288426695]', wheel/src/components/lsm_tree.rs:143:21
Looks like multi versions of one key are separated into two sstables in the same non-overlapping level.
Currently ObjectLsmTree
simply hold a RwLock
for read/write memory tables. Find-grained concurrent control like LevelDB/RocksDB does can be used to optimized it.
As #71 , the comparison method of user key and full key can be abused easily. We should distinguish them with type or trait bounds. And overhead should not be introduced.
Sometimes concurrent put & get found None but Some expected.
Need further investigation.
Currently, in UserKeyIterator
, if a key is delete by a tombstone, it will not be invisible anytime, that leads a query go through it and returned a already deleted version.
Tracking:
bench_kv
sticks. Maybe caused by deadlock, incorrect wait condition or incorrect CAS operations.
Can debug with:
RUSTFLAGS="--cfg tokio_unstable" RUNKV_METRICS=true RUST_BACKTRACE=1 cargo run --features console,deadlock --release --package runkv-bench --bin bench_kv -- --loop 1000
Currently we simply rebuild the heap when changing the direction of MergeIterator
, which is not correct.
Consider this case:
After seeking for 6
, the inner iterators of MergeIterator
will be like:
Direction::Forward
1 - 5 - 9
^
2 - 6 - 10
^
^
3 - 7 - 11
^
If we want to get the prev
key, MergeIterator
will set its direction as Direction::Backward
, and rebuild its heap (move all valid iterators to max heap). This will not modify the states of inner iterators, which is wrong.
Direction::Backward
1 - 5 - 9
^
^
2 - 6 - 10
^
3 - 7 - 11
^
Then move to the previous key of the current state:
Direction::Backward
1 - 5 - 9
^
2 - 6 - 10
^
3 - 7 - 11
^
^
The output would be 7
.
To fix this, simply performing seek
is not enough. The core problem is the syntax of current Seek::Random
is only for forward moving, more details in #25 . With a bi-way random seek, we can easily handle the bug with seek
.
Ref: #25
As titled. Refs: risingwave#1138, risingwave#1468.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.