@huachaohuang thanks for your reply.

Sure. Please help to add the CODE_OF_CONDUCT.

I've read the CoC of Rust Community and really like its section "Moderation". I'll create a dedicated issue for continuing the work of CoC - it deserves a dedicated thread.

Originally posted by @tisonkun in #19 (comment)

A kernel is responsible for the persistent part of an engine. It provides the ability to store events, objects, metadata, etc.

engula using some Rust tools, like cargo-udeps, cargo-audit, maybe we should reduce CI time cost to improve contributor experience.

Mostly, maybe we should download binary rather than runing cargo install for every job.

As the project grows, I answer similar questions again and again. Our current CONTRIBUTING.md somehow lacks information and doesn't specialize in our project. Thus, I propose to elaborate the contributing guide a bit to cover the following topics:

• Get started
  • Prepare the development environment
  • Run examples
• Communications (why communications are important, how we organize discussions)
  • Discussion forum
  • Zulip chat room
• Contributions
  • Principles (reference)
  • Report issues
  • Review patches (it should be intuitive but many (potential) contributors ask me about this topic)
  • Contribute code (it should be intuitive also, with satisfying our CI tasks)
  • Licenses (many contributors confuse where a license header should be placed)

I'm going to prepare a PR this week to catch up on the exposure of v0.2 release. The points above are insights. If things go complex (I hope not yet), it can be factored out to the docs/ folder.

Reference:

• rustc dev guide
• flink how to contributibute
• python dev guide

cc @huachaohuang

cc @Xuanwo @zojw @DCjanus any information you think there should be?

Provide a simple hash engine that supports get/set/delete key-value pairs.
The engine should be able to use different kinds of kernels.

• #168

#143 only supports get/set, we should support delete too. This is not as easy as it might sound. We need to add tombstones for deleted records.

Non-goals:

• Data persistence
• Leader election and failover
• Resource isolation and management

Tasks:

• Design document
• Commands
  • Start a node
  • Bootstrap a universe
  • Join a node to a universe

A file kernel integrates a file journal, storage, manifest, etc.

There are a few papers about applying machine learning to construct indexes on sorted records.
These learned indexes can reduce memory usage significantly.
Maybe we can explore the possibilities to build a new file format with these learned indexes.

Specifically, the PGM index looks very interesting.
However, it seems only applicable to integers instead of arbitrary byte arrays.
Maybe we can apply some order-preserving minimal perfect hash functions here.

Just some immature thoughts here, welcome for discussion.

Reference:

• The Case for Learned Index Structures
• The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds
• Order-preserving minimal perfect hash functions and information retrieval

Tasks:

• Design document
• Background traits
• A local background implementation
• A remote background implementation
  • Background client
  • Background service
• Integrate the background service with microunit

There is a Cargo.toml style guide here. We can consider following it.

Although it does do some works and write under /tmp/engula_test/hello, however, a user friendly message helps improve the journey.

For example, tell the user he/she can find the output under the directory or just list them out. It seems when failed, the framework itself can give error message.

Discussed in #141

@huachaohuang I see this comment but don't know what is "in some cases". Could you please use a TODO comment as well as comment under this issue with an error stack?

It seems from the doc of install it already states that:

An error will be returned if there's an issue with creating the HTTP server or with installing the recorder as the global recorder.

We have three traits for the storage component now: Storage, Bucket, Object. This brings two problems:

• The trait bound is a bit complicated, like Storage<Object, Bucket>
• There are some corner cases dealing with individual buckets, for example, #86 (comment)

A simpler solution is to merge Bucket into Storage, so Storage will provide the following interfaces:

• list buckets
• create bucket
• delete bucket
• list objects in a bucket
• upload an object to a bucket
• delete an object from a bucket

We can still keep Object and ObjectUploader as usually. Note that only Object is performance-critical here, as users may read from an object frequently. Other storage operations don't need to care about performance very much.

@huachaohuang @ZhongliGao I don't know why you guys change the branch name, but if it's the case and we have only interest branches on the upstream repository (i.e., no personal branch), we can trigger the workflow on every "push" and "pull request" event.

See also https://docs.github.com/en/actions/reference/events-that-trigger-workflows#configuring-workflow-events .

Otherwise tests aren't covered.

ref #99 (comment)

need some method to automatic test with AWS service like s3 in CI without leaking access&secret key, IMHO, maybe we can do it in two directions:

• unit test: mock AWS SDK, there are exists an issue indicates SDK support TestConnection to mock/record request-response and we take some tries, but it seems we need more encapsulation works to make it easy for use.
• integration-test: deploy a temp S3-like service(for example https://min.io/) in CI pipeline and make test case using temp S3-like service.

Welcome to give more advice or contribution code 😄

Tasks:

• Design document
• Journal abstraction
• A memory implementation
• #128
• #117

What do you mean by "make it generic"? Does it mean:

pub struct Event<TS> {
    pub ts: TS,
    pub data: Vec<u8>,
}

or Timestamp<T>? Is there similar stuff in other projects?

cc @huachaohuang

Originally posted by @tisonkun in #82 (comment)

The Event<TS> one, but should be more complicated than that. The rationale is to let users customize the type of the timestamp. Depending on how users implement their databases, the usage of timestamp can be very different. For example, it can be a simple increasing counter, a logical lock, or a hybrid logical lock, which may need different representations.

Originally posted by @huachaohuang in #82 (comment)

A Filter trait has been added without an implementation.
We can add a BloomFilter like LevelDB.

Reference: https://github.com/google/leveldb/blob/master/util/bloom.cc

We need to run several commands to check tests, style, license, unused dependencies before sending a PR, which is tedious. Maybe we can add a simple way to check everything at once locally.

A Cache trait has been added with no concrete implementation yet.
We can add a simple LRU cache like RocksDB.
For example, an LRUCache can be partition into multiple LRUShard, each of which can be a simple HashMap protected by a RwLock.

Reference: https://github.com/facebook/rocksdb/blob/main/cache/lru_cache.h

First of all, I must clarify that this project seems still in an early stage so this issue is a suggestion to consider along with the project grows.

According to APL 2.0, it is a common practice to apply explicitly the license by:

To apply the Apache License to specific files in your work, attach the following boilerplate declaration, replacing the fields enclosed by brackets "[]" with your own identifying information. (Don't include the brackets!) Enclose the text in the appropriate comment syntax for the file format. We also recommend that you include a file or class name and description of purpose on the same "printed page" as the copyright notice for easier identification within third-party archives.

Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

If we decide to stick to APL 2.0, we'll meet this proposal sooner or later. Contributors probably complain that a template is too heavy to carry when we're still in an early stage - we don't care about how others make use of the demo. But when you think your work has its shape, it is a signal that you should consider this proposal.

Also, thinking about the name of "copyright owner" is essential for the project.

Last, without audit tools the license header can easily miss or with typo, etc. skywalking-eyes will be a great language-agnostic solution integrated with GitHub Actions.

We have some APIs that need to return a list of values. For example, get a list of object names from a bucket. The simplest way is to return a Vec<T>. But if the list goes too long, it is not a good idea to return it all at once. A common solution for this kind of problem is paging. Rust provides a concept called Stream, which is like an asynchronous iterator that allows the caller to get one value at a time.

I think the most mature way to do that is to use Stream from the futures crate. But it is not a good idea for our public APIs to depend on a third-party trait, which is not stable either. Fortunately, Rust is working on a built-in Stream trait (nightly), so we may consider using that in our public APIs when it is stablized.

Rust crates:

• engula: the integral library and binary
  • engula-engine: contains different kinds of storage engines
  • engula-kernel: contains the kernel abstraction and some built-in implementations
  • engula-journal: contains the journal abstraction and some built-in implementations
  • engula-storage: contains the storage abstraction and some build-in implementations

TODO:

• Improve descriptions in Cargo.toml
  • engula
  • engula-engine
  • engula-kernel
  • engula-journal
  • engula-storage

Kernel needs a metadata store for its metadata persistence. We have two choices:

• Add a general metadata abstraction in the same level with Journal and Storage
• Add a sub-module inside Kernel that only handles kernel specific metadata

For v0.2, we can choose the second solution for simplicity. We can still move to the first solution in the future if the second one is proofed to be inadequate.

Tasks:

• Design document
• Storage abstraction
• A mem storage implementation
• A file storage implementation
• A grpc storage implementation

Most tasks have been done, but we still need to refactor some implementations before releasing v0.2.

ref #99 (comment)

maybe we can add https://github.com/est31/cargo-udeps to git workflow, just like cargo audit

This is confused and too complex for now. We can use main only and see where we go later.

In #78 we simplify the project layout into:

├── Cargo.lock
├── Cargo.toml
├── src
│   ├── api
│   ├── background
│   ├── hello_unit.rs
│   ├── main.rs
│   ├── manifest
│   ├── microunit
│   ├── node.rs
│   └── storage

However, the root cargo config file define both the executable and library, while a consumer of the library doesn't want the executable part. We may consider releasing executable and library separately. @zojw suggested learn from how tokio does release.

cc @huachaohuang @PsiACE

We have two toml files style now. For example, this uses the style:

tokio = {version = "1.13", features = ["full"]}

But this use the style:

tokio = { version = "1.13", features = ["full"] }

Note the whitespace between {}. It would be nice to keep toml files consistent for all contributors.

We should complete some common fields like license, homepage, description before we publish crates to crates.io.

While async-trait works fine, it will result in a heap allocation per-function-call. So for performance-critical scenarios, we still need a better way to define async traits.

A grpc kernel integrates a grpc journal, storage, etc. It consists of a client and a server part. We need to provide a binary to start the kernel server.

The project is just started and is still in the demo stage now.
The primary goal in this stage is to explore the possibility of our designs.

We plan to release the first demo at the end of Sep 2021.
We are going to achieve the following objectives:

• A working demo with simple read/write APIs on AWS
• A demo report about what we did and the lessons learned

Tasks:

• Cache
• HybridStorage
• SstableStorage
• ParquetStorage
• RemoteJournal
• RemoteManifest
• RemoteCompaction
• Support S3 file system and SELECT
• A purpose-built benchmark tool

However, we are short of hands.
If you think we are behind schedule, please push us with a 👍

Update: this demo has ended, please check the report for more details.

Overview

Goals:

• Set up basic project management
• Present the fundamental ideas and usages of Engula

Non-goals:

• Reliability
• Performance

Project

• #59
• #144

Modules

• #73
• #145
• #65
• #68
• #66

Documents

• #132
• engula/engula.github.io#17
• engula/engula.github.io#16

Otherwise inner crates aren't covered.

In order not to disturb your prototyping and rapidly development, I'm glad to write this file and propose a PR, with your help on collecting necessary information.

The draft is looks like:

How to Contribute

I'm really glad you're reading this, because we need volunteer developers to help this project come to fruition.

If you haven't already, come find us on gitter. We want you working on things you're excited about.

Welcome to review our design or participant discussions about the roadmap!

Get Started

We develop Engula with rust stable toolchain.

You're able to get started with Engula with three steps:

1. Setup the environment with rustup.
2. Build Engula via cargo build.
3. Run the example via cargo run --example hello.

Report an Issue

If you think you have found an issue in Engula, you can report it to the issue tracker.

Before filing an issue report is to see whether the problem has already been reported. You can use the search bar to search existing issues. This doesn't always work, and sometimes it's hard to know what to search for, so consider this extra credit. We won't mind if you accidentally file a duplicate report. Don't blame yourself if your issue is closed as duplicated.

If the problem you're reporting is not already in the issue tracker, you can open a GitHub issue with your GitHub account.

Submitting a Pull Request

Please send a GitHub Pull Request to Engula with a clear list of what you've done (read more about pull requests). When you send a pull request, we're looking forward to an expressive description, clear commit messages, and more test coverage if it is code contribution.

Before submitting the pull request, please make sure all tests pass locally:

cargo build --release
cargo test
cargo clippy -- -D warnings
cargo fmt --all -- --check

Thank you for your participation!

The questions are:

1. Do we protect main and only modify it by PR?
2. What merge strategy do we use, specially, merge with commit, rebase and merge, or squash and merge? I highly recommend the latter two where merge with commit make history hard to read - however, I'm not participant your project deeply, so it's your choice.
3. Shall we adopt a code of conduct? If so, I'd suggest Contributor Covenant Code of Conduct and the project should provide a contact method.
4. Any other concern on the draft above?

We can wrap a gRPC Journal similar to the gRPC Storage. I think it is not very hard, so maybe someone can give it a try :)

From Zulip:

Hmm, maybe we can let mini-redis run on our hash engine. I think the job should not be hard since mini-redis stores everything in a HashMap. It will be an interesting way to demonstrate the usage of our hash engine in v0.2.
We can consider working on that once we get #143 landed

This is not required for v0.2. But if anyone is interested in this job, we can add it to the list 🤓

This should be the last closed issue before releasing v0.2.

• Add more crate-level documents

https://github.com/engula/engula/runs/4233986277?check_suite_focus=true

A memory kernel integrates a memory journal, storage, etc.

We chose Gitter for IM before. Gitter is simple to use but not powerful. I see a lot of projects using Discord now. Maybe we should migrate to Discord too.

It's redundant allocation.

Seems you don't use Clippy, it's a very useful linter.

We have a mem and grpc implementation now. But most applications will need persistent journal storage. A sophisticated implementation is not easy, we can start with a naive implementation for now. Check RocksDB LogWriter for an example.

With current main 027b59a cargo build generate a diff on Cargo.lock file.

To prevent leaving an unclean repo where users find there is an annoying diff soon after checking out the repo, I propose to check cargo build generates consistent lock file.

@huachaohuang @Xuanwo it seems we just cannot skip the install step even if cargo cache hit.

Cargo cache can already speed up the installation by cargo-install cache hit logic, we should always run cargo install to avoid false positive.

Remove if: steps.cache.outputs.cache-hit != 'true' lines should work. @Xuanwo we may later figure why current settings create false positive.

Originally posted by @tisonkun in #185 (comment)

We will provide an abstraction and three implementations:

• #149
• #146
• #147
• #148
• #164