dermesser / leveldb-rs Goto Github PK

View Code? Open in Web Editor NEW

496.0 8.0 58.0 1.86 MB

A reimplementation of LevelDB in Rust (no bindings).

License: Other

Shell 0.08% Rust 99.92%

leveldb keyvalue keyvaluestore rust

leveldb-rs's People

Contributors

Stargazers

Watchers

leveldb-rs's Issues

| |_^ `(dyn KeyValueDb + 'static)` cannot be shared between threads safely

|
12 | / lazy_static! {
13 | | pub static ref StorageInstanceRef:RwLock = RwLock::new(StorageFactory::default());
14 | | }
| |_^ (dyn KeyValueDb + 'static) cannot be shared between threads safely
|
::: C:\Users\Song.Shang.cargo\registry\src\mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd\lazy_static-1.4.0\src\inline_lazy.rs:19:20
|
19 | pub struct Lazy<T: Sync>(Cell<Option>, Once);
| ---- required by this bound in lazy_static::lazy::Lazy
|
= help: the trait Sync is not implemented for (dyn KeyValueDb + 'static)
= note: required because of the requirements on the impl of Sync for parking_lot::lock_api::RwLock<parking_lot::RawRwLock, (dyn KeyValueDb + 'static)>
= note: required because of the requirements on the impl of Send for Arc<parking_lot::lock_api::RwLock<parking_lot::RawRwLock, (dyn KeyValueDb + 'static)>>
= note: required because it appears within the type StorageFactory
= note: required because of the requirements on the impl of Sync for parking_lot::lock_api::RwLock<parking_lot::RawRwLock, StorageFactory>
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to 2 previous errors; 2 warnings emitted

For more information about this error, try rustc --explain E0277.
error: could not compile storage

DB uses excessive disk space

It seems to me that I need to call flush explicitly` to prevent the database disk use from blowing up.

Cf. Neptune-Crypto/neptune-core#14

Exception 0xc00000fd encountered at address 0x7ff792ced999: Stack overflow

Exception 0xc00000fd encountered at address 0x7ff792ced999: Stack overflow,

Chrome/Electron localStorage is custom encoded

Update

After many hours of trying to decode JSON from LevelDB from Chrome/Electron, I figured out the encoding

use std::{fs::File, io::Write};

use rusty_leveldb::{CompressionType, Options, DB};

fn main() -> anyhow::Result<()> {
    println!("Hello, world!");

    let mut db = DB::open(
        "leveldb",
        Options {
            compression_type: CompressionType::CompressionSnappy,
            create_if_missing: false,
            paranoid_checks: true,
            ..Default::default()
        },
    )?;

    let Some(data) = db.get(b"_file://\x00\x01persist:root") else {
        anyhow::bail!("None - Data not found");
    };

    let text = decode(&data);
    println!("Data Length: {}", data.len());
    println!("Text Length: {}", text.len());

    if let Ok(mut file) = File::create("data.json") {
        file.write_all(text.as_bytes())?;
    };

    println!("Goodbye, world!");

    Ok(())
}

fn decode(data: &[u8]) -> String {
    let mut decoded_string = String::new();
    let mut index = 0;

    while index < data.len() {
        if data[index] == 0 {
            let start_index = index + 1;

            while index + 1 < data.len() && (data[index + 1] != 0 || index % 2 != 0) {
                index += 1;
            }

            let end_index = index;
            let encoded_slice = &data[start_index..=end_index];

            if encoded_slice.len() >= 2 && encoded_slice.len() % 2 == 0 {
                let decoded_bytes: Vec<u16> = encoded_slice
                    .chunks_exact(2)
                    .map(|chunk| u16::from_le_bytes([chunk[0], chunk[1]]))
                    .collect();

                if let Ok(decoded) = String::from_utf16(&decoded_bytes) {
                    decoded_string.push_str(&decoded);
                }
            }
        }
        
        index += 1;
    }

    decoded_string
}

I found this blog post that helped point me in the right direction, but unfortunately, my data was not compressed with lz-string like Slack.

"attempt to subtract with overflow" panic in `DB::open`

thread 'main' panicked at 'attempt to subtract with overflow', /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/log.rs:151:16
stack backtrace:
   0: rust_begin_unwind
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:114:5
   3: rusty_leveldb::log::LogReader<R>::read
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/log.rs:151:16
   4: rusty_leveldb::version_set::VersionSet::recover
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/version_set.rs:585:34
   5: rusty_leveldb::db_impl::DB::recover
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/db_impl.rs:180:33
   6: rusty_leveldb::db_impl::DB::open
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/db_impl.rs:115:29

when multi-thread support?

When will multi-threaded DB be coming... the wait for lock release is such a huge flaw .... that really needs fixed asap especially for a code release claiming to be at version 3....

I spent days integrating completely over looking that tiny little note.... that seems to still be coming soon.... according to the docs.

So I am stuck looking for other implementations but this seems to be the only real active and working implimenation of leveldb in rust....

Please oh please release this... soon....

How to ensure the LRU cache is thread safe?

The LRU cache implemented in cache.rs is not thread safe. I'd like to know how you ensure that it is thread-safe

Investigate Skipmap memory leaks

There seems to be a bug in the implementation of SkipMap leading to memory leaks. I found this while investigating #11 which refers to a different area though.

Instrumenting the Drop implementation:

impl Drop for Node {
    fn drop(&mut self) {
        // large object should drop
        if let Some(mut next) = self.next.take() {
            println!("destroyed {:?}", next.key);
            while let Some(child) = next.next.take() {
                next = child;
                println!("destroyed {:?}", next.key);
            }
        }
            println!("destroyed {:?}", self.key);
        unsafe {
            for skip in self.skips.iter_mut() {
                if let Some(mut next) = skip.take() {
                    while let Some(child) = (*next).next.take() {
                        next = Box::into_raw(child);
                    }
                }
            }
        }
    }
}

gives

running 1 test
destroyed [97, 98, 97]
destroyed [97, 98, 97]
destroyed [97, 98, 98]
destroyed [97, 98, 98]
destroyed []
test skipmap::tests::test_skipmap_iterator_init ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 142 filtered out; finished in 0.00s

for a skipmap constructed as

    pub fn make_skipmap() -> SkipMap {
        let mut skm = SkipMap::new(options::for_test().cmp);
        let keys = vec![
            "aba", "abb", "abc", "abd", "abe", "abf", "abg", "abh", "abi", "abj", "abk", "abl",
            "abm", "abn", "abo", "abp", "abq", "abr", "abs", "abt", "abu", "abv", "abw", "abx",
            "aby", "abz",
        ];

        for k in keys {
            skm.insert(k.as_bytes().to_vec(), "def".as_bytes().to_vec());
        }
        skm
    }

panicked at 'called `Option::unwrap()` on a `None` value

thread 'tokio-runtime-worker' panicked at 'called Option::unwrap() on a None value', /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rusty-leveldb-1.0.7/src/cache.rs:90:75
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
double free or corruption (fasttop)

The program will report an error after a period of time, I only have two operations in the program, before saving to obtain whether the value exists, no then save

License question

Our project uses leveldb-rs and wants to confirm about the license. Is leveldb-rs original from [1]? Given it is a copy of the original it should be under the same license as the original, but [1] is BSD license while leveldb-rs is MIT licensed. Am I wrong about the original?

[1] https://github.com/google/leveldb

skipmap: the drop operation of skipmap may cause stackoverflow

The drop operation of the skipmap may cause stack overflow as the tail recursion optimization could not work in this case, for example, writing many keys into the skipmap.

thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
Aborted (core dumped)

From the stack

#0  0x00005555555c89ed in core::alloc::layout::Layout::max_size_for_align () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/alloc/layout.rs:93
#1  core::alloc::layout::Layout::array::inner (element_size=16, align=..., n=4) at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/alloc/layout.rs:438
#2  0x00005555555c1890 in core::alloc::layout::Layout::array () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/alloc/layout.rs:428
#3  alloc::raw_vec::RawVec<T,A>::current_memory (self=0x7ffff02127d0) at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/raw_vec.rs:247
#4  0x000055555558deac in <alloc::raw_vec::RawVec<T,A> as core::ops::drop::Drop>::drop (self=0x7ffff02127d0) at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/raw_vec.rs:478
#5  0x000055555558844b in core::ptr::drop_in_place<alloc::raw_vec::RawVec<core::option::Option<*mut rusty_leveldb::skipmap::Node>>> ()
    at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#6  0x0000555555587fe4 in core::ptr::drop_in_place<alloc::vec::Vec<core::option::Option<*mut rusty_leveldb::skipmap::Node>>> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#7  0x0000555555589407 in core::ptr::drop_in_place<rusty_leveldb::skipmap::Node> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#8  0x000055555558ae2a in core::ptr::drop_in_place<alloc::boxed::Box<rusty_leveldb::skipmap::Node>> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#9  0x0000555555587ca3 in core::ptr::drop_in_place<core::option::Option<alloc::boxed::Box<rusty_leveldb::skipmap::Node>>> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#10 0x000055555558943f in core::ptr::drop_in_place<rusty_leveldb::skipmap::Node> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#11 0x000055555558ae2a in core::ptr::drop_in_place<alloc::boxed::Box<rusty_leveldb::skipmap::Node>> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#12 0x0000555555587ca3 in core::ptr::drop_in_place<core::option::Option<alloc::boxed::Box<rusty_leveldb::skipmap::Node>>> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#13 0x000055555558943f in core::ptr::drop_in_place<rusty_leveldb::skipmap::Node> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#14 0x000055555558ae2a in core::ptr::drop_in_place<alloc::boxed::Box<rusty_leveldb::skipmap::Node>> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#15 0x0000555555587ca3 in core::ptr::drop_in_place<core::option::Option<alloc::boxed::Box<rusty_leveldb::skipmap::Node>>> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490
#16 0x000055555558943f in core::ptr::drop_in_place<rusty_leveldb::skipmap::Node> () at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ptr/mod.rs:490

This seems to be a common issue easily encountered in the rust list implementation, a possible optimization is to implement the drop trait for the list explicitly to avoid function calls.

Bloated crate size

Seems like data for testing got into crates.io release, bloating it up to 5 MB.

It is shown as number one wasteful crate here: https://the-lean-crate.github.io/waste/rusty-leveldb/

tooldb may be extra.

Panic: thread 'main' panicked at assertion failed: `(left == right)`

When I run the database for relatively large amount of data, sometimes the db panics. The following code can reproduce the problem.

use rand::{Rng, SeedableRng};
use rand::rngs::StdRng;
use rusty_leveldb::{Options, DB};

fn main() {
    let mut rng = StdRng::seed_from_u64(42);

    for t in 0..3 {
        println!("start t = {t}");
        let opt = Options::default();
        let mut db = DB::open("mydatabase", opt).unwrap();    

        for i in 0..1500 {
            println!("-- {i}");
            let key = format!("ABCDEF{}", i);

            let num_bytes = rng.gen_range(1..1024 * 1024);
            let long_vec: Vec<u8> = (0..num_bytes).map(|_| rng.gen_range(0..255)).collect();

            db.put(key.as_bytes(), &long_vec).unwrap();
        }

        db.flush().unwrap();
        println!("end t = {t}");
    }
}

the output is as follow

start t = 0
-- 0
-- 1
...
-- 1499
end t = 0
start t = 1
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `Equal`,
 right: `Less`', .../leveldb-rs/src/version_set.rs:797:17
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
   1: core::panicking::panic_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:228:5
   4: rusty_leveldb::version_set::Builder::maybe_add_file
             at .../leveldb-rs/src/version_set.rs:797:17
   5: rusty_leveldb::version_set::Builder::save_to
             at .../leveldb-rs/src/version_set.rs:827:17
   6: rusty_leveldb::version_set::VersionSet::log_and_apply
             at .../leveldb-rs/src/version_set.rs:499:13
   7: rusty_leveldb::db_impl::DB::install_compaction_results
             at .../leveldb-rs/src/db_impl.rs:945:9
   8: rusty_leveldb::db_impl::DB::start_compaction
             at .../leveldb-rs/src/db_impl.rs:698:13
   9: rusty_leveldb::db_impl::DB::maybe_do_compaction
             at .../leveldb-rs/src/db_impl.rs:601:17
  10: rusty_leveldb::db_impl::DB::open
             at .../leveldb-rs/src/db_impl.rs:135:9
  11: try_leveldb::main
             at ./src/main.rs:12:22
  12: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

It seems that somehow the non-overlap rule broke. Any ideas?

stack overflow when recover database

stack overflow when recover database.

Check the call stack and find that it has been recursively infinite from Display::fmt -> to_string.

leveldb-rs/src/error.rs

Lines 48 to 52 in abf543b

 impl Display for Status { 

 fn fmt(&self, fmt: &mut Formatter) -> result::Result<(), fmt::Error> { 

 fmt.write_str(&self.to_string()) 

 } 

 }

PANIC: could not remove_last(); bug!

Not sure how useful is this report, but anyway...

I was playing with example write-a-lot, adjusting size of the data and got this error: could not remove_last(); bug!

Now the application won't start. Fails with:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[71, 219, 128, 139, 36, 117, 71, 219]`,
 right: `[87, 251, 128, 139, 36, 117, 71, 219]`',  .../rusty-leveldb-0.3.0/src/table_builder.rs:50:9

Current code:


use rand::Rng;
use rusty_leveldb::CompressionType;
use rusty_leveldb::Options;
use rusty_leveldb::DB;

use std::error::Error;
use std::iter::FromIterator;

const KEY_LEN: usize = 16;
const VAL_LEN: usize = 400_000;

fn gen_string(len: usize) -> String {
    let mut rng = rand::thread_rng();
    String::from_iter(rng.gen_ascii_chars().take(len))
}
fn gen_val(len: usize) -> String {
    let mut rng = rand::thread_rng();
    let s = String::from_iter(rng.gen_ascii_chars().take(10));
    s.repeat(len / 10)
}

fn fill_db(db: &mut DB, entries: usize) -> Result<(), Box<Error>> {
    for i in 0..entries {
        println!("{}", i);
        let (k, v) = (gen_string(KEY_LEN), gen_val(VAL_LEN));
        db.put(k.as_bytes(), v.as_bytes())?;


        if i % 100 == 0 {
            db.flush()?;
        }
    }
    Ok(())
}

fn main() {
    println!("Start");
    let mut opt = Options::default();
    opt.compression_type = CompressionType::CompressionSnappy;
    let mut db = DB::open("/tmp/leveldb-test1", opt).unwrap();
    println!("DB opened");


    fill_db(&mut db, 32768).unwrap();
}

Rust version: rustc 1.41.0 (5e1a79984 2020-01-27)

DB state: https://send.firefox.com/download/7281446ca733dbcd/#lVmzPpIyJVg1JrmGt7hMCw

Panic in `DB::open`

Got this panic and stack trace when I called DB::open. Error occurs in recover_log_file -> insert_into_memtable.

I'm running level rusty-leveldb 1.0.6.

Stack trace

thread 'main' panicked at 'range end index 335261 out of range for slice of length 328603', /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/write_batch.rs:136:22
stack backtrace:
   0: rust_begin_unwind
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
   2: core::slice::index::slice_end_index_len_fail_rt
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/slice/index.rs:77:5
   3: core::slice::index::slice_end_index_len_fail
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/slice/index.rs:69:9
   4: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/slice/index.rs:409:13
   5: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/slice/index.rs:18:9
   6: <alloc::vec::Vec<T,A> as core::ops::index::Index<I>>::index
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/alloc/src/vec/mod.rs:2703:9
   7: <rusty_leveldb::write_batch::WriteBatchIter as core::iter::traits::iterator::Iterator>::next
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/write_batch.rs:136:22
   8: rusty_leveldb::write_batch::WriteBatch::insert_into_memtable
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/write_batch.rs:97:23
   9: rusty_leveldb::db_impl::DB::recover_log_file
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/db_impl.rs:268:13
  10: rusty_leveldb::db_impl::DB::recover
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/db_impl.rs:210:17
  11: rusty_leveldb::db_impl::DB::open
             at /home/thv/.cargo/registry/src/github.com-1ecc6299db9ec823/rusty-leveldb-1.0.6/src/db_impl.rs:115:29

thread 'main' has overflowed its stack

I got panic "thread 'main' has overflowed its stack" when try to run example/wirte-a-lot.

It happened when skipmap dropped.

The default drop of Node is recursive, it has a giant liked-list of nodes. The call back is tool large .

The following works.

impl Drop for Node {
    fn drop(&mut self) {
        // large object should drop
        if let Some(mut next) = self.next.take() {
            while let Some(child) = next.next.take() {
                next = child;
            }
        }
        unsafe {
            for skip in self.skips.iter_mut() {
                if let Some(mut next) = skip.take() {
                    while let Some(child) = (*next).next.take() {
                        next = Box::into_raw(child);
                    }
                }
            }
        }
    }
}

Tauri not compatible with LevelDB-RS

I have the problem that my tauri app is not compatible with this leveldb implementation. Tauri restarts a couple of times in dev mode and seems to confuse leveldb-rs with Locks. It says the process could not access thefile, because another process has locked a part of this file.

Fix the mcpe example

The mcpe example is broken and doesn't work anymore.
Some things have changed in the mcpe dbs and are not the same as they were. It would be lovely if this awesome example would be updated.

When running I get the error:

called `Result::unwrap()` on an `Err` value: Status { code: NotSupported, err: "invalid compression id `4`" }

Feature Request: Custom Compression Method

In current version is hardcode in rusty_leveldb::CompressionType.

I propose a Compressor trait

pub trait Compressor {
    fn encode(&self, block: Vec<u8>) -> rusty_leveldb::Result<Vec<u8>>;

    fn decode(&self, block: Vec<u8>) -> rusty_leveldb::Result<Vec<u8>>;
}

then change rusty_leveldb::Options to this

pub struct Options {
    ...
    compressor: Box<dyn Compressor>,
}

this should fit in here and here

Fails to create a new iterator

This is the error Error: Status { code: IOError, err: "IOError: snappy: corrupt input (expected stream header but got unexpected chunk type byte 200)" }

Possible memory leak on large data

I found strange behaviour with this method https://github.com/dermesser/leveldb-rs/blob/master/src/table_reader.rs#L42
Probably ref cycle?

Code sample on repr:

 let mut block_index = Vec::with_capacity(800000);
    let mut db = DB::open(path, Options::default())?;
    let mut iter = db.new_iter()?;
    let (mut k, mut v) = (Vec::with_capacity(800000), Vec::with_capacity(800000));

    while iter.advance() {
        iter.current(&mut k, &mut v);
        if is_block_index_record(&k) {
            let record = BlockIndexRecord::from(&k[1..], &v)?;
            if record.status & (BLOCK_VALID_CHAIN | BLOCK_HAVE_DATA | BLOCK_VALID_CHAIN) > 0 {
                block_index.push(record);
            }
        }
    }
    
 Ok(block_index)

Related valgrind output:

8,409,399 bytes in 2,018 blocks are possibly lost in loss record 1,309 of 1,311
==2643397==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2643397==    by 0x20FAB7: rusty_leveldb::table_block::read_table_block 
==2643397==    by 0x2129EE: rusty_leveldb::table_reader::TableIterator::load_block 
==2643397==    by 0x212722: rusty_leveldb::table_reader::TableIterator::skip_to_next_entry 
==2643397==    by 0x2125C0: <rusty_leveldb::table_reader::TableIterator as rusty_leveldb::types::LdbIterator>::advance 
==2643397==    by 0x2158B2: <rusty_leveldb::version::VersionIter as rusty_leveldb::types::LdbIterator>::advance 
==2643397==    by 0x20C64B: <rusty_leveldb::merging_iter::MergingIter as rusty_leveldb::types::LdbIterator>::advance 
==2643397==    by 0x23A84B: rusty_leveldb::db_iter::DBIterator::find_next_user_entry
==2643397==    by 0x15BE54: rusty_blockparser::blockchain::parser::index::get_block_index 
==2643397==    by 0x1564C9: rusty_blockparser::blockchain::parser::chain::ChainStorage::new 
==2643397==    by 0x176E6B: rusty_blockparser::main 
==2643397==    by 0x18AF62: std::sys_common::backtrace::__rust_begin_short_backtrace 
==2643397== 
==2643397== 16,801,974 bytes in 4,032 blocks are possibly lost in loss record 1,310 of 1,311
==2643397==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==2643397==    by 0x20FAB7: rusty_leveldb::table_block::read_table_block (
==2643397==    by 0x2129EE: rusty_leveldb::table_reader::TableIterator::load_block 
==2643397==    by 0x212722: rusty_leveldb::table_reader::TableIterator::skip_to_next_entry 
==2643397==    by 0x2125C0: <rusty_leveldb::table_reader::TableIterator as rusty_leveldb::types::LdbIterator>::advance 
==2643397==    by 0x2158B2: <rusty_leveldb::version::VersionIter as rusty_leveldb::types::LdbIterator>::advance
==2643397==    by 0x20C64B: <rusty_leveldb::merging_iter::MergingIter as rusty_leveldb::types::LdbIterator>::advance 
==2643397==    by 0x23A84B: rusty_leveldb::db_iter::DBIterator::find_next_user_entry
==2643397==    by 0x15BE54: rusty_blockparser::blockchain::parser::index::get_block_index 
==2643397==    by 0x17AF6D: rusty_blockparser::main 
==2643397==    by 0x18AF62: std::sys_common::backtrace::__rust_begin_short_backtrace (
==2643397==    by 0x18E9A0: main

Any thoughts?

Implement `Clone` for `AsyncDB`

Problem

I think that this is a common use case, when you are using AsyncDB you want to pass it to each subservice by cloning, for example.

Consideration

As AsyncDB is just a wrapper around tokio::mpsc channel, there should be no problem with just deriving Clone. Expect JoinHandle, for that we can just wrap it around Arc and it will drop when last reference will be dropped

Solution

Derive Clone for AsyncDB, and add Arc around JoinHandle (see #40)

integer-encoding dependency: breaking change

probably need to change dependency to

integer-encoding = "~1.0"

since they introduced breaking change in 1.2

How can I use prefix query

How can I use prefix query？

Data vanishes when using `word-analyze` and `leveldb-tool` with Snappy enabled.

[lbo@lboyoga leveldb-rs]$ \time target/debug/word-analyze LICENSE 
0.08user 0.02system 0:00.11elapsed 95%CPU (0avgtext+0avgdata 3584maxresident)k
0inputs+24outputs (0major+827minor)pagefaults 0swaps
[lbo@lboyoga leveldb-rs]$ \time target/debug/word-analyze LICENSE 
0.01user 0.00system 0:00.02elapsed 79%CPU (0avgtext+0avgdata 3328maxresident)k
0inputs+16outputs (0major+176minor)pagefaults 0swaps
[lbo@lboyoga leveldb-rs]$ \time target/debug/word-analyze LICENSE 
0.02user 0.00system 0:00.03elapsed 93%CPU (0avgtext+0avgdata 3456maxresident)k
0inputs+16outputs (0major+186minor)pagefaults 0swaps
[lbo@lboyoga leveldb-rs]$ cat tooldb/LOG
Recovered manifest with next_file=118 manifest_num=117 log_num=116 prev_log_num=0 last_seq=7916
reusing manifest "wordsdb/MANIFEST-000114"
Recovering log file "/home/lbo/dev/rust/leveldb-rs/wordsdb/000116.log"
reusing log file "/home/lbo/dev/rust/leveldb-rs/wordsdb/000116.log"
[lbo@lboyoga leveldb-rs]$ target/debug/leveldb-tool iter | grep and
and => 12
andor => 3
[lbo@lboyoga leveldb-rs]$ cat tooldb/LOG
Recovered manifest with next_file=118 manifest_num=117 log_num=116 prev_log_num=0 last_seq=7916
Recovering log file "/home/lbo/dev/rust/leveldb-rs/wordsdb/000116.log"
Start write of L0 table 000118
L0 table 000118 has 5557 bytes
Deleting file type=Log num=116
Deleting file type=Descriptor num=114
Compacting @0 [1, 103, 32, 0, 0, 0, 0, 0] .. [119, 105, 116, 104, 111, 117, 116, 1, 174, 29, 0, 0, 0, 0, 0]
Compacting 4 files at L0 and 1 files at L1
New table num=120: keys=105 size=1632
Compacted 4 L0 files + 1 L1 files => 1632B
Compaction finished: level 1: 1 files, 1632 bytes ([(120, 1632)]); 
Deleting file type=Table num=103
Deleting file type=Table num=109
Deleting file type=Table num=112
Deleting file type=Table num=115
Deleting file type=Table num=118
[lbo@lboyoga leveldb-rs]$ \time target/debug/word-analyze LICENSE 
0.05user 0.01system 0:00.07elapsed 94%CPU (0avgtext+0avgdata 3456maxresident)k
0inputs+24outputs (0major+312minor)pagefaults 0swaps
[lbo@lboyoga leveldb-rs]$ cat tooldb/LOG
Recovered manifest with next_file=122 manifest_num=121 log_num=119 prev_log_num=0 last_seq=8468
reusing manifest "wordsdb/MANIFEST-000117"
Recovering log file "/home/lbo/dev/rust/leveldb-rs/wordsdb/000119.log"
reusing log file "/home/lbo/dev/rust/leveldb-rs/wordsdb/000119.log"
[lbo@lboyoga leveldb-rs]$ target/debug/leveldb-tool iter | grep and
and => 4
andor => 1

Expected result is that and appears as 16 and andor as 4. This works when compression is disabled. It's currently unclear why this occurs.

Fix various memory leaks

There is a whole bundle of more or less difficult memory leaks found by memcheck. This is a tracking issue for some of them.

How to support multi-thread safety,

seek is not working?

let mut db = LevelDB::default();

    let key = vec![0, 1, 2];

    {
        db.set(&vec![10, 11], 1u32.to_le_bytes().as_slice())
            .unwrap();
    }

    {
        let mut key1 = key.clone();
        key1.extend_from_slice(&[1, 2]);
        db.set(key1.as_slice(), 1u32.to_le_bytes().as_slice())
            .unwrap();
    }

    {
        let mut key2 = key.clone();
        key2.extend_from_slice(&[3, 4]);
        db.set(key2.as_slice(), 2u32.to_le_bytes().as_slice())
            .unwrap();
    }

    {
        let mut key3 = key.clone();
        key3.extend_from_slice(&[5, 6]);
        db.set(key3.as_slice(), 3u32.to_le_bytes().as_slice())
            .unwrap();
    }

    db.iter_all(key.as_slice(), |v| {
        let v = u32::from_le_bytes(v.as_slice().try_into().unwrap());
        print!("{} ", v);
        v
    })
    .unwrap();

Can't use `next()` when database contains the empty byte slice `&[]` as a key

The function truncate_to_userkey panics if the empty byte vector is used as key. This can be fixed by changing the below > to a >=. The function truncate_to_userkey is called when you have a DBIterator where next() is called.

pub fn truncate_to_userkey(ikey: &mut Vec<u8>) {
    let len = ikey.len();
    assert!(len > 8);
    ikey.truncate(len - 8);
}

Suggested fix: Change src/key_types.rs line 212 to: assert!(len >= 8);

After read/write too many data, one key-value lost.

I don't know why this error happened. Is it possible?

Also i don't know whether other key-values lost.

The scenario:

1. insert key1 value1
2. insert a lot of keys and values
3. key1 and value1 lost

I guess this error happened with creating new ldb file.

New version

I would love a new version that contains the fix in b0c22e4

Can we get a 1.05?

	impl Display for Status {
	fn fmt(&self, fmt: &mut Formatter) -> result::Result<(), fmt::Error> {
	fmt.write_str(&self.to_string())
	}
	}