Git Product home page Git Product logo

photondb's Introduction

PhotonDB

crates docs

A high-performance storage engine for modern hardware and platforms.

PhotonDB is designed from scratch to leverage the power of modern multi-core chips, storage devices, operating systems, and programming languages.

Features:

  • Latch-free data structures, scale to many cores.
  • Log-structured persistent stores, optimized for flash storage.
  • Asynchronous APIs and efficient file IO, powered by io_uring on Linux.

Design

Architecture

Progress

We have published the photondb crate v0.0.4. You can try some examples to see what it can do so far. It is important to note that the current version is still too young to be used for anything serious.

Use the synchronous APIs:

[dependencies]
photondb = "0.0.4"
use photondb::{std::Table, Result, TableOptions};

fn main() -> Result<()> {
    let table = Table::open("/tmp/photondb", TableOptions::default())?;
    let key = vec![1];
    let val1 = vec![2];
    let val2 = vec![3];
    // Simple CRUD operations.
    table.put(&key, 1, &val1)?;
    table.delete(&key, 2)?;
    table.put(&key, 3, &val2)?;
    assert_eq!(table.get(&key, 1)?, Some(val1));
    assert_eq!(table.get(&key, 2)?, None);
    assert_eq!(table.get(&key, 3)?, Some(val2.clone()));
    let guard = table.pin();
    // Get the value without copy.
    assert_eq!(guard.get(&key, 3)?, Some(val2.as_slice()));
    // Iterate the tree page by page.
    let mut pages = guard.pages();
    while let Some(page) = pages.next()? {
        for (k, v) in page {
            println!("{:?} {:?}", k, v);
        }
    }
    Ok(())
}

Use the asynchronous APIs:

[dependencies]
photondb = "0.0.4"
photonio = "0.0.5"
use photondb::{Result, Table, TableOptions};

#[photonio::main]
async fn main() -> Result<()> {
    let table = Table::open("/tmp/photondb", TableOptions::default()).await?;
    let key = vec![1];
    let val1 = vec![2];
    let val2 = vec![3];
    // Simple CRUD operations.
    table.put(&key, 1, &val1).await?;
    table.delete(&key, 2).await?;
    table.put(&key, 3, &val2).await?;
    assert_eq!(table.get(&key, 1).await?, Some(val1.clone()));
    assert_eq!(table.get(&key, 2).await?, None);
    assert_eq!(table.get(&key, 3).await?, Some(val2.clone()));
    let guard = table.pin();
    // Get the value without copy.
    assert_eq!(guard.get(&key, 3).await?, Some(val2.as_slice()));
    // Iterate the tree page by page.
    let mut pages = guard.pages();
    while let Some(page) = pages.next().await? {
        for (k, v) in page {
            println!("{:?} {:?}", k, v);
        }
    }
    Ok(())
}

Run the example with:

cargo +nightly-2022-10-01 run

photondb's People

Contributors

felixonmars avatar huachaohuang avatar ming535 avatar w41ter avatar zojw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

photondb's Issues

Roadmap v0.0.1

I am trying to cut the first patch version of the storage engine this month.

This version will include an in-memory Bw-tree implementation that can be used as a concurrent tree map. Most features are done so far, except for the merge SMO. I will postpone the merge SMO until we have implemented the persistent part.

Here are some tasks before the release:

  • Refactor the iterator interface
  • Do some benchmark
  • Improve README and other materials

Distinguish garbage collection and space reclamation

In a storage engine that supports MVCC, there are usually two kinds of "garbage collection". The first kind is to drop obsoleted key-value versions. The second kind is to reclaim disk space. In order to distinguish these two concepts, from now on, we use the concept of garbage collection for the former and the concept of space reclamation for the latter.

PageTxn update page validation

In former PR: #85 (comment), the condition, old_addr should less than new_addr, is not guarrantted by caller. We need to find a more appropriate way to verify and tell the caller to try again.

Can't make the loom test work

running 1 test
Process 75752 stopped
* thread #2, name = 'tree::pagetable::tests::loom_test', stop reason = EXC_BAD_ACCESS (code=2, address=0x1011ea6b8)
    frame #0: 0x00000001004ddee7 photondb_engine-afcc1143f8647152`__rust_probestack + 23
photondb_engine-afcc1143f8647152`__rust_probestack:
->  0x1004ddee7 <+23>: testq  %rsp, 0x8(%rsp)
    0x1004ddeec <+28>: subq   $0x1000, %r11             ; imm = 0x1000 
    0x1004ddef3 <+35>: cmpq   $0x1000, %r11             ; imm = 0x1000 
    0x1004ddefa <+42>: ja     0x1004ddee0               ; <+16>
Target 0: (photondb_engine-afcc1143f8647152) stopped.
(lldb) bt
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
warning: could not execute support code to read Objective-C class data in the process. This may reduce the quality of type information available.
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #2, name = 'tree::pagetable::tests::loom_test', stop reason = EXC_BAD_ACCESS (code=2, address=0x1011ea6b8)
  * frame #0: 0x00000001004ddee7 photondb_engine-afcc1143f8647152`__rust_probestack + 23
    frame #1: 0x0000000100016fae photondb_engine-afcc1143f8647152`_$LT$photondb_engine..tree..pagetable..L0$LT$_$GT$$u20$as$u20$core..default..Default$GT$::default::h770627e018345a19 at pagetable.rs:131
    frame #2: 0x000000010002ae9b photondb_engine-afcc1143f8647152`_$LT$alloc..boxed..Box$LT$T$GT$$u20$as$u20$core..default..Default$GT$::default::h39c04354260c9a48 at boxed.rs:1259:18
    frame #3: 0x0000000100040f75 photondb_engine-afcc1143f8647152`_$LT$photondb_engine..tree..pagetable..Inner$u20$as$u20$core..default..Default$GT$::default::h71cfdbafd85f959b at pagetable.rs:66:17
    frame #4: 0x000000010001d595 photondb_engine-afcc1143f8647152`_$LT$loom..sync..arc..Arc$LT$T$GT$$u20$as$u20$core..default..Default$GT$::default::h210725ee7d10f769 at arc.rs:260:18
    frame #5: 0x00000001000413e4 photondb_engine-afcc1143f8647152`_$LT$photondb_engine..tree..pagetable..PageTable$u20$as$u20$core..default..Default$GT$::default::h3fd7c054b3063417 at pagetable.rs:16:5
    frame #6: 0x000000010004ff94 photondb_engine-afcc1143f8647152`photondb_engine::tree::pagetable::tests::loom_test::_$u7b$$u7b$closure$u7d$$u7d$::h78cdb847f839fb89((null)=0x0000000101306710) at pagetable.rs:251:25
    frame #7: 0x0000000100033ad0 photondb_engine-afcc1143f8647152`loom::model::Builder::check::_$u7b$$u7b$closure$u7d$$u7d$::h68f685cedde533f2 at model.rs:185:17

Tracking issues for develop PageStore

Overview

The goals of the current milestone:

Set up an initial version runnable PageStore.

Tasks

Scheduled

Archived

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.