[WIP] An experimental cloud-native distributed KV engine for OLTP workload.
License: MIT License
Found some abnormal jitters in metrics:
After fio on the disk, fsync never reached second-level latency.
Ref: #30
Tracking:
As titled, see cargo-binstall .
Tracking:
SstableStore and ObjectStore).Currently ObjectLsmTree simply hold a RwLock for read/write memory tables. Find-grained concurrent control like LevelDB/RocksDB does can be used to optimized it.
A PR template is needed to offer to contributors to help them write good PR messages.
We need an action to check if the title and commit log of each PR obey some rules.
The expected behavior of SeekRandom is move to the exact given key or the "next" key. "next" here actually means "prev" if the iterator is expected to be move backward.
Especially for MergeIterator. It's useful for changing the direction of it correctly.
For each raft group, log entries before compact index can be safely deleted. Although log entires are continuously written in most cases, but log entries of various groups are cross written.
e.g.
file 1: | group 1, 0 - 100 | group 2, 0 - 50 | group 1, 101 - 200 | group 2, 51 - 100 |
As a result, with the system running, some parts of some log file can be purged but the other parts can not (e.g. 90% can be purged in file 1, 50% can be purged in file 2, we call the rate that can be purged of a file utilization rate).
So we can track the (approximate) utilization rate of each file, when the utilization rate of some files drops below the threshold, we need to rewrite the file (append the still needed part to the current active log file and update the memory indices of them) and remove the old log file. The rewrite operation is safe because we recorded term of the raft log and we can easily distinguish if log entries with the same index can overwrite the current one.
The rewrite procedure is asynchronous, and the follow aspects should be taken into consideration:
As titled. Drunk.
Tracking
The write model of raft log store is simple now. Currently, it cannot utilize the benefits of batching and pipelining. A better write model like RocksDB or something is needed.
SSTable id should be globally unique. To simplify the sstable id generation logic, each generate sstable id based on its node id and a persisted sequential number.
As titled. Refs: risingwave#1138, risingwave#1468.
As #71 , the comparison method of user key and full key can be abused easily. We should distinguish them with type or trait bounds. And overhead should not be introduced.
Currently we simply rebuild the heap when changing the direction of MergeIterator, which is not correct.
Consider this case:
After seeking for 6, the inner iterators of MergeIterator will be like:
Direction::Forward
1 - 5 - 9
^
2 - 6 - 10
^
^
3 - 7 - 11
^
If we want to get the prev key, MergeIterator will set its direction as Direction::Backward, and rebuild its heap (move all valid iterators to max heap). This will not modify the states of inner iterators, which is wrong.
Direction::Backward
1 - 5 - 9
^
^
2 - 6 - 10
^
3 - 7 - 11
^
Then move to the previous key of the current state:
Direction::Backward
1 - 5 - 9
^
2 - 6 - 10
^
3 - 7 - 11
^
^
The output would be 7.
To fix this, simply performing seek is not enough. The core problem is the syntax of current Seek::Random is only for forward moving, more details in #25 . With a bi-way random seek, we can easily handle the bug with seek.
Ref: #25
When running test_multi_raft_group_concurrent_put_get with 100 loops for each group, sometimes assert failed.
e.g.
thread 'test_multi_raft_group_concurrent_put_get::test_multi_raft_group_concurrent_put_get' panicked at 'assertion failed: `(left == right)`
left: `[118, 52, 52, 52, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 48]`,
right: `[]`', tests/integrations/test_multi_raft_group_concurrent_put_get.rs:214:21
L214 asserts the result of get after put
RunKV/tests/integrations/test_multi_raft_group_concurrent_put_get.rs
Lines 184 to 225 in b0d596c
which should be None but got unexpected result.
When digging into the trace log, I found that the delete was not applied:
[2022-05-07T06:16:17Z TRACE runkv_wheel::components::lsm_tree] put; self=ObjectStoreLsmTree { raft_node: 52 } key=b"k444 0" value=b"v444 0" sequence=36656013143 apply_index=6530
[2022-05-07T06:16:18Z TRACE runkv_wheel::components::lsm_tree] get; self=ObjectStoreLsmTree { raft_node: 52 } key=b"k444 0" sequence=36656013147
Actually nothing to celebrate, but life always needs something to celebrate, right?
As in #74 , create or clone Bytes involves atomic operation, whose costs cannot be ignored if performs frequently (e.g. in iterators). Unnecessary Bytes should be replaced with Vec<u8> or &[u8].
Sometimes concurrent put & get found None but Some expected.
Need further investigation.
Now rpc client connects to the server when building, which requires to build each components in order in integration test. The ideal way is avoid connecting when building, and connects when bootstrap.
As in #65 , components should establish RPC connections when bootstrapping instead of building. And connection to the same node should be aggregated together. A connection pool can handle the requirements gracefully.
For a fast MVP, there are lots of unnecessary buffer copies in raft log store. Refactor needed.
group in raft log store means log group, which is actually raft node id in the concept of Raft, not raft group in multi-raft.
So it's better to rename group in raft log store with another name (e.g. collection, set, ..?) to avoid misleading.
To balance the large total data volume, the low capacity of memory and the high latency of S3, RunKV can use the remaining space of the disk as secondary cache, or File Cache.
There are some considerations about File Cache:
Ref: #44 .
bench_kv sticks. Maybe caused by deadlock, incorrect wait condition or incorrect CAS operations.
Can debug with:
RUSTFLAGS="--cfg tokio_unstable" RUNKV_METRICS=true RUST_BACKTRACE=1 cargo run --features console,deadlock --release --package runkv-bench --bin bench_kv -- --loop 1000When porting block and block iterator from RunKV to RisingWave, found some unnecessary slice copy influenced the performance of block iterator a lot. We can also avoid them in RunKV.
For more detailes, see risingwave#1138.
Ref: #37
Since the QPS dropped sharply after integrating raft WAL, we need to find the reason for the decrease.
As titled.
When running bench_kv in release mode, found the following panic:
left: `[]`,
right: `[118, 52, 51, 57, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 48]`', bench/bench_kv/main.rs:206:21
stack backtrace:
thread 'tokio-runtime-worker' panicked at 'assertion failed: `(left == right)`
left: `2`,
right: `1`: result: [863288426616, 863288426695]', wheel/src/components/lsm_tree.rs:143:21
Looks like multi versions of one key are separated into two sstables in the same non-overlapping level.
Tracking:
Tracking:
Sometimes, fn read() call of Log leads to early EOF error. Maybe caused by data race when rotating active log file.
thread 'tokio-runtime-worker' panicked at 'unexpected error: Store(Other(StorageError(IoError(Custom { kind: UnexpectedEof, error: "early eof" })))), raft_id: 2, raft node: 2, group: 1, namespace: raft', /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raft_log.rs:624:26
stack backtrace:
0: rust_begin_unwind
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:584:5
1: core::panicking::panic_fmt
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panicking.rs:142:14
2: raft::raft_log::RaftLog<T>::slice::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raft_log.rs:624:26
3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
4: raft::raft_log::RaftLog<T>::next_entries_since::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raft_log.rs:445:18
5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
6: raft::raw_node::RawNode<T>::gen_light_ready::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raw_node.rs:458:67
7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
8: raft::raw_node::RawNode<T>::ready::{{closure}}
at /home/mrcroxx/.cargo/git/checkouts/raft-rs-097263935fea03be/710b3a9/src/raw_node.rs:553:42
9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
10: runkv_wheel::worker::raft::RaftWorker<RN,F>::handle_ready::{{closure}}::{{closure}}
at ./wheel/src/worker/raft.rs:503:42
11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
12: runkv_wheel::worker::raft::RaftWorker<RN,F>::handle_ready::{{closure}}
at ./wheel/src/worker/raft.rs:499:5
13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
14: runkv_wheel::worker::raft::RaftWorker<RN,F>::run_inner::{{closure}}
at ./wheel/src/worker/raft.rs:421:36
15: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
16: <runkv_wheel::worker::raft::RaftWorker<RN,F> as runkv_common::Worker>::run::{{closure}}
at ./wheel/src/worker/raft.rs:289:35
17: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
18: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/future.rs:124:9
19: runkv_wheel::components::raft_manager::RaftManager::create_raft_node::{{closure}}::{{closure}}
at ./wheel/src/components/raft_manager.rs:176:43
20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/future/mod.rs:91:19
21: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
22: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
23: tokio::runtime::task::core::CoreStage<T>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
24: tokio::runtime::task::harness::poll_future::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
25: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panic/unwind_safe.rs:271:9
26: std::panicking::try::do_call
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:492:40
27: __rust_try
28: std::panicking::try
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:456:19
29: std::panic::catch_unwind
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panic.rs:137:14
30: tokio::runtime::task::harness::poll_future
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
31: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
32: tokio::runtime::task::harness::Harness<T,S>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
33: tokio::runtime::task::raw::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
34: tokio::runtime::task::raw::RawTask::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
35: tokio::runtime::task::LocalNotified<S>::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:347:9
36: tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:425:13
37: tokio::coop::with_budget::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
38: std::thread::local::LocalKey<T>::try_with
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/thread/local.rs:442:16
39: std::thread::local::LocalKey<T>::with
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/thread/local.rs:418:9
40: tokio::coop::with_budget
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
41: tokio::coop::budget
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
42: tokio::runtime::thread_pool::worker::Context::run_task
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:424:9
43: tokio::runtime::thread_pool::worker::Context::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:391:24
44: tokio::runtime::thread_pool::worker::run::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:376:17
45: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
46: tokio::runtime::thread_pool::worker::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:373:5
47: tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:352:45
48: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/task.rs:42:21
49: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
50: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
51: tokio::runtime::task::core::CoreStage<T>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
52: tokio::runtime::task::harness::poll_future::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
53: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/core/src/panic/unwind_safe.rs:271:9
54: std::panicking::try::do_call
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:492:40
55: __rust_try
56: std::panicking::try
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panicking.rs:456:19
57: std::panic::catch_unwind
at /rustc/f4a7ce997a1d7546d2b737f8b87d36907bcea2ad/library/std/src/panic.rs:137:14
58: tokio::runtime::task::harness::poll_future
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
59: tokio::runtime::task::harness::Harness<T,S>::poll_inner
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
60: tokio::runtime::task::harness::Harness<T,S>::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
61: tokio::runtime::task::raw::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
62: tokio::runtime::task::raw::RawTask::poll
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
63: tokio::runtime::task::UnownedTask<S>::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:384:9
64: tokio::runtime::blocking::pool::Task::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:91:9
65: tokio::runtime::blocking::pool::Inner::run
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:308:17
66: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
at /home/mrcroxx/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:288:17
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Currently, in UserKeyIterator, if a key is delete by a tombstone, it will not be invisible anytime, that leads a query go through it and returned a already deleted version.
A not-exist sst is tend to be deleted after comapction.
2022-05-16T15:13:15.277677Z ERROR runkv_rudder::worker::compaction_detector: trigger compaction l0 error: other: compaction error: [Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697679 not exists, diff: VersionDiff { id: 696, sstable_diffs: [SstableDiff { id: 68719476789, level: 0, op: Delete, data_size: 68762 }, SstableDiff { id: 68719476790, level: 0, op: Delete, data_size: 67887 }, SstableDiff { id: 68719476791, level: 0, op: Delete, data_size: 69175 }, SstableDiff { id: 68719476792, level: 0, op: Delete, data_size: 68799 }, SstableDiff { id: 433791697679, level: 1, op: Delete, data_size: 33857 }, SstableDiff { id: 433791697684, level: 1, op: Delete, data_size: 33266 }, SstableDiff { id: 433791697689, level: 1, op: Delete, data_size: 34152 }, SstableDiff { id: 433791697695, level: 1, op: Delete, data_size: 33991 }, SstableDiff { id: 433791697699, level: 1, op: Delete, data_size: 33625 }, SstableDiff { id: 433791697704, level: 1, op: Delete, data_size: 34298 }, SstableDiff { id: 433791697708, level: 1, op: Delete, data_size: 3211975 }, SstableDiff { id: 433791697750, level: 1, op: Insert, data_size: 36418 }, SstableDiff { id: 433791697757, level: 1, op: Insert, data_size: 35829 }, SstableDiff { id: 433791697764, level: 1, op: Insert, data_size: 36713 }, SstableDiff { id: 433791697768, level: 1, op: Insert, data_size: 36553 }, SstableDiff { id: 433791697772, level: 1, op: Insert, data_size: 36059 }, SstableDiff { id: 433791697776, level: 1, op: Insert, data_size: 36856 }, SstableDiff { id: 433791697779, level: 1, op: Insert, data_size: 3458003 }] }")))), Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697685 not exists, diff: VersionDiff { id: 695, sstable_diffs: [SstableDiff { id: 107374182453, level: 0, op: Delete, data_size: 68025 }, SstableDiff { id: 107374182454, level: 0, op: Delete, data_size: 68593 }, SstableDiff { id: 107374182455, level: 0, op: Delete, data_size: 68662 }, SstableDiff { id: 107374182456, level: 0, op: Delete, data_size: 68806 }, SstableDiff { id: 433791697680, level: 1, op: Delete, data_size: 34008 }, SstableDiff { id: 433791697685, level: 1, op: Delete, data_size: 34511 }, SstableDiff { id: 433791697690, level: 1, op: Delete, data_size: 34004 }, SstableDiff { id: 433791697694, level: 1, op: Delete, data_size: 34295 }, SstableDiff { id: 433791697701, level: 1, op: Delete, data_size: 33997 }, SstableDiff { id: 433791697705, level: 1, op: Delete, data_size: 34002 }, SstableDiff { id: 433791697709, level: 1, op: Delete, data_size: 34281 }, SstableDiff { id: 433791697713, level: 1, op: Delete, data_size: 33783 }, SstableDiff { id: 433791697715, level: 1, op: Delete, data_size: 34362 }, SstableDiff { id: 433791697717, level: 1, op: Delete, data_size: 3109571 }, SstableDiff { id: 433791697752, level: 1, op: Insert, data_size: 36841 }, SstableDiff { id: 433791697759, level: 1, op: Insert, data_size: 37365 }, SstableDiff { id: 433791697762, level: 1, op: Insert, data_size: 36705 }, SstableDiff { id: 433791697767, level: 1, op: Insert, data_size: 36928 }, SstableDiff { id: 433791697770, level: 1, op: Insert, data_size: 36710 }, SstableDiff { id: 433791697775, level: 1, op: Insert, data_size: 36565 }, SstableDiff { id: 433791697778, level: 1, op: Insert, data_size: 36848 }, SstableDiff { id: 433791697781, level: 1, op: Insert, data_size: 36345 }, SstableDiff { id: 433791697782, level: 1, op: Insert, data_size: 36855 }, SstableDiff { id: 433791697784, level: 1, op: Insert, data_size: 3346709 }] }")))), Err(StorageError(ManifestError(InvalidVersionDiff("node 10000 sst L1-433791697692 not exists, diff: VersionDiff { id: 695, sstable_diffs: [SstableDiff { id: 120259084341, level: 0, op: Delete, data_size: 68268 }, SstableDiff { id: 120259084342, level: 0, op: Delete, data_size: 68832 }, SstableDiff { id: 120259084343, level: 0, op: Delete, data_size: 68050 }, SstableDiff { id: 120259084344, level: 0, op: Delete, data_size: 69146 }, SstableDiff { id: 433791697681, level: 1, op: Delete, data_size: 33993 }, SstableDiff { id: 433791697688, level: 1, op: Delete, data_size: 33634 }, SstableDiff { id: 433791697692, level: 1, op: Delete, data_size: 34287 }, SstableDiff { id: 433791697696, level: 1, op: Delete, data_size: 33994 }, SstableDiff { id: 433791697700, level: 1, op: Delete, data_size: 34297 }, SstableDiff { id: 433791697703, level: 1, op: Delete, data_size: 33773 }, SstableDiff { id: 433791697707, level: 1, op: Delete, data_size: 33855 }, SstableDiff { id: 433791697710, level: 1, op: Delete, data_size: 34293 }, SstableDiff { id: 433791697712, level: 1, op: Delete, data_size: 33852 }, SstableDiff { id: 433791697714, level: 1, op: Delete, data_size: 34500 }, SstableDiff { id: 433791697716, level: 1, op: Delete, data_size: 3075627 }, SstableDiff { id: 433791697754, level: 1, op: Insert, data_size: 36557 }, SstableDiff { id: 433791697758, level: 1, op: Insert, data_size: 36345 }, SstableDiff { id: 433791697763, level: 1, op: Insert, data_size: 36849 }, SstableDiff { id: 433791697766, level: 1, op: Insert, data_size: 36421 }, SstableDiff { id: 433791697771, level: 1, op: Insert, data_size: 36855 }, SstableDiff { id: 433791697774, level: 1, op: Insert, data_size: 36415 }, SstableDiff { id: 433791697777, level: 1, op: Insert, data_size: 36416 }, SstableDiff { id: 433791697780, level: 1, op: Insert, data_size: 36926 }, SstableDiff { id: 433791697783, level: 1, op: Insert, data_size: 36562 }, SstableDiff { id: 433791697785, level: 1, op: Insert, data_size: 37073 }, SstableDiff { id: 433791697786, level: 1, op: Insert, data_size: 3311271 }] }"))))]
#149 may be related to this bug.
The idle time is extremely long while apply duration looks normal. Consider another way to notify apply.
RunKV uses object store as part of its storage backend. S3 object store and memory object store (for tests) are needed.
ObjectStore trait.MemoryObjectStore.S3ObjectStore.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.