CN Doc

LakeSoul is a unified streaming and batch table storage for fast data processing built on top of the Apache Spark engine by the DMetaSoul team, and supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and streaming & batch unification.

LakeSoul implements incremental upserts for both row and column and allows concurrent updates on the same partition. LakeSoul uses LSM-Tree like structure to support updates on hash partitioning table with primary key, and achieve very high write throughput (30MB/s/core) on cloud object store like S3 while providing optimized merge on read performance. LakeSoul scales meta data management by using PostgreSQL DB.

More detailed features please refer to our wiki page: Wiki Home

Please find usage documentations in project's wiki: Usage Doc

使用文档

Follow the Quick Start to quickly set up a test env.

Checkout the CDC Ingestion with Debezium and Kafka example on how to sync LakeSoul table with OLTP dbs like MySQL in a realtime manner.

• Meta Management (#23)
  • Multiple Level Partitioning: Multiple range partition and at most one hash partition
  • Concurrent write with auto conflict resolution
  • MVCC with read isolation
  • Write transaction through Postgres Transaction
• Table operations
  • LSM-Tree style upsert for hash partitioned table
  • Merge on read for hash partition with upsert delta file
  • Copy on write update for non hash partitioned table
  • Compaction
• Spark Integration
  • Table/Dataframe API
  • SQL support with catalog except upsert
  • Query optimization
    • Shuffle/Join elimination for operations on primary key
  • Merge UDF (Merge operator)
  • Merge Into SQL support
    • Merge Into SQL with match on Primary Key (Merge on read)
    • Merge Into SQL with match on non-pk
    • Merge Into SQL with match condition and complex expression (Merge on read when match on PK) (depends on #66)
• Flink Integration (#57)
  • Table API
  • Flink CDC
    • Exactly Once Sink
    • Auto Schema Sync
    • Auto Table Creation (depends on #78)
• Hive Integration
  • Export to Hive partition after compaction
• Realtime Data Warehousing
  • CDC ingestion
  • Time Travel (Snapshot read)
  • Snapshot rollback
  • MPP Engine Integration (depends on #66)
    • Presto
    • Apache Doris
• Native IO (#66)
  • Object storage IO optimization
  • Native merge on read
• Cloud Native
  • Multi-layer storage classes support with data tiering

Community guidelines

Please feel free to open an issue or dicussion if you have any questions.

Join our slack user group

Email us at [email protected].

LakeSoul is opensourced under Apache License v2.0.