kipdata / kipdb Goto Github PK

Feature Request

Is your feature request related to a problem? Please describe:

Describe the feature you'd like:
Need to be able to clean up disk and memory data methods, such as clear
Tips: MVCC needs to be taken into account when clearing, and the latest Version should be used as the cleanup object

• 现有策略根据数据条数进行触发Compaction判定
• 原有用数据量的bytes大小来判定
• 可以参考 levedb 实现多种触发策略同时存在

Question

Direct IO, usally means O_DIRECT [1] and related IO, which could bypass some IO stack [2]. Usally it needs to align the size of io, and can bypass some page buffer/cache

Should we rename this to Local IO or other?

Besides, read using current reader is limited to single thread, since read is only support single thread. should we support a readAt (pread [3]) for possible multi-thread read?

impl Read for DirectIoReader {
    fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
        self.fs.read(buf)
    }
}

[1] https://stackoverflow.com/questions/41257656/what-does-o-direct-really-mean
[2] https://www.thomas-krenn.com/en/wiki/Linux_Storage_Stack_Diagram
[3] https://man7.org/linux/man-pages/man2/pwrite.2.html

Bug Report

as title, after triggering swap during Comparison, the trigger should be reset to avoid frequent triggering

根据LSMStore的各个组件原理编写相应描述文档

• VersionEdit快照
• WAL
• 迭代器

这是我rustup show的信息

Default host: x86_64-unknown-linux-gnu
rustup home:  /root/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu (default)
nightly-x86_64-unknown-linux-gnu

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (overridden by '/root/rs_code/KipDB/rust-toolchain')
rustc 1.72.0-nightly (e6d4725c7 2023-06-05)

当我在ubuntu上运行cargo run --bin server，他会报两个错

error: expected one of `)`, `,`, `.`, `?`, or an operator, found `:`
   --> src/kernel/lsm/compactor.rs:145:71
    |
145 |                         (future::try_join_all(ss_table_futures).await?: Vec<(SSTable, Scope)>)
    |                                                                       ^ expected one of `)`, `,`, `.`, `?`, or an operator


error: calls to `std::mem::drop` with a value that implements `Copy` does nothing
   --> src/kernel/utils/lru_cache.rs:389:17
    |
389 |                 drop(node.as_ptr());
    |                 ^^^^^-------------^
    |                      |
    |                      argument has type `*mut Node<K, V>`
    |
    = note: use `let _ = ...` to ignore the expression or result

我做了如下修改后，可以运行起server

let (vec_new_ss_table, vec_new_scope): (Vec<SSTable>, Vec<Scope>) =
                        future::try_join_all(ss_table_futures).await?
                    .into_iter()
                    .unzip();

 let _ = node.as_ptr();

请问，这是我使用方式不对，还是在开发过程中后面会进一步解决的问题

异步化网络层提高吞吐，并吃Tokio的buffer加成

参考：
https://docs.rs/tokio-util/latest/tokio_util/codec/index.html

Bug Report

Repeatedly inserting a small amount of data and then closing it will cause too many sstables(The sstable that should be at level 0 is deleted.)

Feature Request

Is your feature request related to a problem? Please describe:
Support for flushing background (no async/await)

Describe the feature you'd like:
Trigger Flush asynchronously

Feature Request

目前SSTable简化，仅仅使用MetaBlock存储BloomFilter
参考到BF的后继优化以及减少额外依赖，因此进行复现重构

Benchmark中明显较慢的区域
尽可能提高Block编码速度

目前使用clap作为简单的测试客户端，若是能支持命令式则是极好的

Feature Request

Bug Report

After storing data, flush is performed, and then it can still be read after deleting data

#[tokio::main]
async fn main() -> Result<(), KernelError> {
    let temp_dir = TempDir::new().expect("unable to create temporary working directory");
    let config = Config::new(temp_dir.into_path()).enable_level_0_memorization();
    let kip_storage = KipStorage::open_with_config(config).await?;

    println!("Set KeyValue -> (apple, banana)");
    kip_storage
        .set(b"apple", Bytes::copy_from_slice(b"banana"))
        .await?;

    println!(
        "Get Key: apple -> Value: {:?}",
        kip_storage.get(b"apple").await?
    );
    println!("SizeOfDisk: {}", kip_storage.size_of_disk().await?);
    println!("Len: {}", kip_storage.len().await?);
    println!("IsEmpty: {}", kip_storage.is_empty().await);

    kip_storage.flush().await?;

    let join_cmd_1 = vec![
        CommandData::set(b"moon".to_vec(), b"star".to_vec()),
        CommandData::remove(b"apple".to_vec()),
    ];
    println!(
        "Join 1: {:?} -> {:?}",
        join_cmd_1.clone(),
        kip_storage.join(join_cmd_1).await?
    );
    let join_cmd_2 = vec![
        CommandData::get(b"moon".to_vec()),
        CommandData::get(b"apple".to_vec()),
    ];
    println!(
        "Join 2: {:?} -> {:?}",
        join_cmd_2.clone(),
        kip_storage.join(join_cmd_2).await?
    );

    Ok(())
}

Show:

Set KeyValue -> (apple, banana)
Get Key: apple -> Value: Some(b"banana")
SizeOfDisk: 0
Len: 1
IsEmpty: false
Join 1: [Set { key: [109, 111, 111, 110], value: [115, 116, 97, 114] }] -> [None]
Join 2: [Get { key: [109, 111, 111, 110] }, Get { key: [97, 112, 112, 108, 101] }] -> [Some([115, 116, 97, 114]), Some([98, 97, 110, 97, 110, 97])]

目前VersionStatus根据VersionLog进行状态回归的时候，根据VersionEdit进行对应SSTable加载获取其统计信息

此处可以减少SSTable加载以优化恢复性能：
例：Version::NewFile新增参数SSTableMeta等，统计时使用该参数而避免加载SSTable

文档/Wiki 翻译支持多种语言

• English
• 日本語

Bug Report

What did you see instead?
After the database service is restarted abnormally, the SSTable generated in the previous operation will not be cleaned up

Bug Report

Benchmark中Sled与KipDB会交错运行，进行对比性能数据参考，但T::open中未对文件格式名区分，会导致两个数据库之间的测试数据混杂在一个文件夹下

可以参考BenchMark测试名的形式使用T::name插入T::open中format的实例文件夹命名

Feature Request

Refactoring to a tonic based RPC server