Comments (2)
MrCroxx
commented on May 24, 2024
Search key in level 2, which is a non-overlap level:
[2022-04-01T07:04:45Z TRACE runkv_wheel::storage::lsm_tree] find key b"k0000000000000000000000000000000000000000000000000000000000004176" in ssts:
[[], [], [863288426753, 863288426748], []]
thread 'test_concurrent_put_get' panicked at 'assertion failed: `(left == right)`
left: `None`,
right: `Some(b"v0000000000000000000000000000000000000000000000000000000000004176")`', tests/integrations/mod.rs:104:17
But actually, the two sstables overlapped in level 2:
[2022-04-01T07:04:42Z TRACE runkv_exhauster::service] build sst: Sstable {
id: 863288426748,
meta: SstableMeta {
block_metas: [
BlockMeta {
offset: 0,
len: 4156,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004125\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004141\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 4156,
len: 4161,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004141\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004158\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 8317,
len: 4148,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004158\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004174\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 12465,
len: 4153,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004174\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004190\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 16618,
len: 4149,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004191\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004206\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 20767,
len: 4212,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004206\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004222\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 24979,
len: 4147,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004222\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004238\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 29126,
len: 4234,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004238\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004255\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 33360,
len: 4147,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004256\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004271\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 37507,
len: 4213,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004271\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004287\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 41720,
len: 4141,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004287\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004303\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 45861,
len: 4139,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004303\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004318\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 50000,
len: 4149,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004319\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004335\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 54149,
len: 4234,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004335\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004352\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 58383,
len: 4145,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004353\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004369\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 62528,
len: 4160,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004370\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004387\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 66688,
len: 160,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004387\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004387\xff\xff\xff\xff\xff\xff\xff\xfe",
},
],
bloom_filter_bytes: [...],
data_size: 66852,
},
}
[2022-04-01T07:04:42Z DEBUG runkv_exhauster::service] sst 863288426748 uploaded
[2022-04-01T07:04:44Z TRACE runkv_exhauster::service] build sst: Sstable {
id: 863288426753,
meta: SstableMeta {
block_metas: [
BlockMeta {
offset: 0,
len: 4157,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003615\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003631\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 4157,
len: 4241,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003631\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003648\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 8398,
len: 4140,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003649\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003664\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 12538,
len: 4099,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003665\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003681\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 16637,
len: 4148,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003681\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003697\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 20785,
len: 4142,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003697\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003713\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 24927,
len: 4099,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003714\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003730\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 29026,
len: 4147,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003730\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003746\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 33173,
len: 4149,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003746\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003762\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 37322,
len: 4148,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003762\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003779\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 41470,
len: 4139,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003779\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003795\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 45609,
len: 4098,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003795\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003812\xff\xff\xff\xff\xff\xff\xff\xfa",
},
BlockMeta {
offset: 49707,
len: 4155,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003812\xff\xff\xff\xff\xff\xff\xff\xfe",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003829\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 53862,
len: 4139,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003829\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003844\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 58001,
len: 4148,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003845\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004656\xff\xff\xff\xff\xff\xff\xff\xfe",
},
BlockMeta {
offset: 62149,
len: 4157,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004657\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004673\xff\xff\xff\xff\xff\xff\xff\xf6",
},
BlockMeta {
offset: 66306,
len: 160,
first_key: b"k0000000000000000000000000000000000000000000000000000000000004673\xff\xff\xff\xff\xff\xff\xff\xfa",
last_key: b"k0000000000000000000000000000000000000000000000000000000000004673\xff\xff\xff\xff\xff\xff\xff\xfe",
},
],
bloom_filter_bytes: [...],
data_size: 66470,
},
}
[2022-04-01T07:04:44Z DEBUG runkv_exhauster::service] sst 863288426753 uploaded
from runkv.
MrCroxx
commented on May 24, 2024
Reason:
Previously, during compaction, next level overlapping sstables are picked file by file with the base level sstables, which may miss some sstables that fit the gap between two sstables.
e.g.
[2022-04-01T10:17:34Z TRACE runkv_storage::lsm_tree::manifest::version] updated levels: [
[],
[863288426682, 863288426715, 863288426718, 863288426719, 863288426720, 863288426721, 863288426722, 863288426723, 863288426726, 863288426727, 863288426728, 863288426729, 863288426732, 863288426733, 863288426734, 863288426735, 863288426736, 863288426737, 863288426738, 863288426739, 863288426740],
[863288426678, 863288426679, 863288426748, 863288426749, 863288426750, 863288426527, 863288426741, 863288426742, 863288426743, 863288426744, 863288426747, 863288426632, 863288426710, 863288426711, 863288426712, 863288426713, 863288426572, 863288426754, 863288426755, 863288426756, 863288426757, 863288426598, 863288426599, 863288426528, 863288426529],
[863288426714, 863288426751, 863288426752, 863288426680, 863288426708, 863288426709, 863288426606, 863288426753]
]
863288426752
first_key: b"k0000000000000000000000000000000000000000000000000000000000003096\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003613\xff\xff\xff\xff\xff\xff\xff\xfe",
[2022-04-01T10:17:34Z TRACE runkv_rudder::worker::compaction_detector] compaction version diff:
VersionDiff {
id: 0,
sstable_diffs: [
SstableDiff {
id: 863288426744,
level: 2,
op: Delete,
data_size: 66598,
first_key: b"k0000000000000000000000000000000000000000000000000000000000002834\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003094\xff\xff\xff\xff\xff\xff\xff\xfe",
},
SstableDiff {
id: 863288426747,
level: 2,
op: Delete,
data_size: 62994,
first_key: b"k0000000000000000000000000000000000000000000000000000000000003614\xff\xff\xff\xff\xff\xff\xff\xf6",
last_key: b"k0000000000000000000000000000000000000000000000000000000000003899\xff\xff\xff\xff\xff\xff\xff\xf6",
},
SstableDiff {
id: 863288426680,
level: 3,
op: Delete,
data_size: 67,
},
SstableDiff {
id: 863288426708,
level: 3,
op: Delete,
data_size: 67,
},
SstableDiff {
id: 863288426709,
level: 3,
op: Delete,
data_size: 7776,
},
SstableDiff {
id: 863288426758,
level: 3,
op: Insert,
data_size: 80,
},
SstableDiff {
id: 863288426759,
level: 3,
op: Insert,
data_size: 22222,
},
],
}
from runkv.
Related Issues (20)
- optimize concurrent writes for ObjectLsmTree memory states
- purge raft log files with low utilization rate HOT 1
- introduce tracing with opentelemetry and jaeger HOT 1
- refactor: rename group in raft log store
- wait apply notify latency is extremely high HOT 7
- bug: some requests are not applied or applied out of order without error HOT 1
- bug: abnormal fsync latency jitter HOT 3
- bug: multi version of one key are separated into two sstables in the same level HOT 1
- bug: bench_kv sticks HOT 4
- bug: tend to delete not-exist sstable after comaction
- bug: raft log store read data race HOT 1
- chore: Celebrate my 100th commit for RunKV HOT 1
- File Cache as Secondary Cache HOT 1
- introduce connection pool
- replace unnecessary Bytes with Vec<u8> and &[u8] in storage module HOT 2
- Tracking: raft WAL
- enhance: avoid unnecessary buffer copy in raft log store
- enhance: better write model for batching and pipeline
- bench: port block iterator micro bench HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from runkv.