s7v7nislands / datafuse Goto Github PK

View Code? Open in Web Editor NEW

This project forked from datafuselabs/databend

0.0 1.0 0.0 2.99 MB

Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, written in Rust

Home Page: https://datafuse.dev

License: Apache License 2.0

Makefile 0.12% Rust 94.13% Dockerfile 0.14% Shell 1.61% Python 2.17% Smarty 0.21% JavaScript 1.33% CSS 0.30%

datafuse's Introduction

Datafuse

Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture

Datafuse is a Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture written in Rust, inspired by ClickHouse and powered by arrow-rs, built to make it easy to power the Data Cloud.

Principles

Fearless
- No data races, No unsafe, Minimize unhandled errors
High Performance
- Everything is Parallelism
High Scalability
- Everything is Distributed
High Reliability
- True Separation of Storage and Compute

Architecture

Performance

Memory SIMD-Vector processing performance only
Dataset: 100,000,000,000 (100 Billion)
Hardware: AMD Ryzen 7 PRO 4750U, 8 CPU Cores, 16 Threads
Rust: rustc 1.49.0 (e1884a8e3 2020-12-29)
Build with Link-time Optimization and Using CPU Specific Instructions
ClickHouse server version 21.2.1 revision 54447

Query	FuseQuery (v0.1)	ClickHouse (v21.2.1)
SELECT avg(number) FROM numbers_mt(100000000000)	(3.11 s.)	×3.14 slow, (9.77 s.) 10.24 billion rows/s., 81.92 GB/s.
SELECT sum(number) FROM numbers_mt(100000000000)	(2.96 s.)	×2.02 slow, (5.97 s.) 16.75 billion rows/s., 133.97 GB/s.
SELECT min(number) FROM numbers_mt(100000000000)	(3.57 s.)	×3.90 slow, (13.93 s.) 7.18 billion rows/s., 57.44 GB/s.
SELECT max(number) FROM numbers_mt(100000000000)	(3.59 s.)	×4.09 slow, (14.70 s.) 6.80 billion rows/s., 54.44 GB/s.
SELECT count(number) FROM numbers_mt(100000000000)	(1.76 s.)	×2.22 slow, (3.91 s.) 25.58 billion rows/s., 204.65 GB/s.
SELECT sum(number+number+number) FROM numbers_mt(100000000000)	(23.14 s.)	×5.47 slow, (126.67 s.) 789.47 million rows/s., 6.32 GB/s.
SELECT sum(number) / count(number) FROM numbers_mt(100000000000)	(3.09 s.)	×1.96 slow, (6.07 s.) 16.48 billion rows/s., 131.88 GB/s.
SELECT sum(number) / count(number), max(number), min(number) FROM numbers_mt(100000000000)	(6.73 s.)	×4.01 slow, (27.59 s.) 3.62 billion rows/s., 28.99 GB/s.
SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 1000	(6.91 s.)	×1.42 slow, (9.83 s.) 1.02 billion rows/s., 8.14 GB/s.
SELECT max(number),sum(number) FROM numbers_mt(1000000000) GROUP BY number % 3, number % 4, number % 5	(10.87 s.)	×1.95 fast, (5.58 s.) 179.23 million rows/s., 1.43 GB/s.