Distributed logging for cloud-native databases about bunshin HOT 10 OPEN

How about providing support for configurable consistency level: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlConfigConsistency.html

It allows user to trade consistency for performance if they have large amount of data and each data point is not that critical to lose.

In quorum-write pattern, configurable consistency level can be easily implemented using client-side option ACK_QUORUM, which means client confirms write as soon as it has received given number of ACKs. Thta's exactly how cassandra and Kafka implement this feature.

from bunshin.

What about putting this into a design-docs directory?

from bunshin.

How about providing support for configurable consistency level: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlConfigConsistency.html

It allows user to trade consistency for performance if they have large amount of data and each data point is not that critical to lose.

from bunshin.

What about putting this into a design-docs directory?

Will move to design doc when it's finished.

from bunshin.

The design looks similar to bookkeeper. I'd recommend adding a reference or related works part.

Will move to design doc when it's finished.
Maybe you could open a pull request for this (like an RFC), which should be easier to review.

from bunshin.

A sequence of log entries in a segment that are written to the same write quorum is logically called a "chunk"

The naming of the chunk is a bit confusing with the segment.

from bunshin.

A sequence of log entries in a segment that are written to the same write quorum is logically called a "chunk"

The naming of the chunk is a bit confusing with the segment.

Agree. Maybe a graph to describe these concepts is better.

from bunshin.

The design looks similar to bookkeeper. I'd recommend adding a reference or related works part.

Will move to design doc when it's finished.
Maybe you could open a pull request for this (like an RFC), which should be easier to review.

Quorum systems almost look the same, and we have to somehow simplify the design of bookkeeper, since the operation and maintainence bring lot of pain in real-world production environment.

from bunshin.

If we build such a quorum read/write system on cloud, it will require stateful set to store multiple replications. That would cause a dependency to PV or similar things.

Instead, projects like AutoMQ builds WAL over single AZ EBS (perhaps Azure Storage supports cross-region replication out-of-the-box).

I doubt a BookKeeper alike system does not leverage cloud storage and introduces unnecessary complexity.

What if we read/write to a block service on cloud as a WAL? IIRC we can tolerate some data loss in WAL. And AWS EBS is 99.999% reliable; maybe even better than operating a multi-server quorum system with stateful set and PV.

WDYT?

from bunshin.

You're correct. That's why we are still revisiting the design.

That would cause a dependency to PV or similar things.

That's not the main concern. Persistent Volumes are just the abstraction of durable storage like EBS in K8s.

The problems of BookKeeper-like quorum systems are:

• First, it mixes cold and hot data in one single storage, in which case they may contend for IO and bandwidth. In fact, WAL systems have one write pattern (tail append) and two read patterns (tail read and catch-up read), while the latter (reading cold data) can tolerate higher latency.
• Second, quorum systems implicitly replicate data multiple times. In the quorum level, take Qw=3 for example, data has 3 replicas. While in each quorum node, data is stored on EBS, which also has 3 replicas internally. That sums to 9 replicas, with no region-failover capabiity.

Despite it's quorum or not, we still need a standalone WAL service. It serves not only for GreptimeDB components as WAL, but also as a channel that streams database events to faciliate functions like CDC and incremental materialized views.

from bunshin.