This repository contains list of distributed system projects with open source code in various programming languages which may be useful in order to better understand how to build distributed services.
- (Golang) Jocko - Kafka/distributed commit log service in Go. [Serf + Raft]
- (Golang) oklog - distributed and coordination-free log management system for big ol' clusters [Archived]
- (Golang) elasticell - a distributed HA Redis-compatible NoSQL database with strong consistency and reliability
- (Erlang) CouchDB - highly available, partition tolerant, eventually consistent document database . Supports master-master setups with automatic conflict detection.
- (Java) Apache Cassandra - a highly-scalable partitioned row store. Rows are organized into tables with a required primary key
- (Java) Apache HBase - Hadoop database, a distributed, scalable, big data store. Useful when random, realtime read/write access to big data needed
- (Golang) Tair - high-performance and high-availability distributed fast-access memory (MDB)/persistent (LDB) storage service
- (Golang) immudb - immutable database based on zero trust, Key/Value & SQL, tamperproof, data change history
- [Rust] toydb - distributed SQL database in Rust, written as a learning project
- (Rust) DB3 Network - open-source decentralized firebase firestore alternative
- (C) memcached - high performance multithreaded event-based key/value cache store intended to be used in a distributed system
- (C) redis - in-memory database with various value types that persists on disk
- (Rust) TiKV - distributed transactional key-value database, originally created to complement TiDB
- (C++) leveldb - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values
- (Golang) goleveldb - LevelDB implemented in Golang
- (Golang) summitdb - in-memory, NoSQL key/value database. It persists to disk, uses the Raft consensus algorithm, is ACID compliant, and built on a transactional and strongly-consistent model.It supports custom indexes, geospatial data, JSON documents, and user-defined JS scripting
- (Python) pupdb - a simple file-based key-value database
- (Python) pickledb - an open source key-value store using Python's json module
- (C++) KeyDB - a faster drop-in multithreaded alternative to Redis
- (C++) Dragonfly - is an in-memory data store fully compatible with Redis and Memcache and designed using modern algorithms
- (Golang) CockroachDB - distributed fault-tolerant SQL database built on a transactional and strongly-consistent key-value store
- (Golang) YugabyteDB - cloud native distributed SQL database for mission-critical applications
- (Golang) RQLite - lightweight, distributed relational database, which uses SQLite as its storage engine
- (Golang) Kingbus - a distributed MySQL binlog store based on raft [Raft]
- (C++) MongoDB - document database designed for ease of development and scaling
- (Golang) FerretDB - an open-source proxy, converting the MongoDB 6.0+ wire protocol queries to SQL - using PostgreSQL as a database engine
- (C#) LiteDB - NoSQL Document Store in a single data file
- (Python) tinydb - lightweight document oriented database written in pure Python
- (Java) neo4j - Graph Database
- (Python) edgedb - a graph-relational database
- (C++) nebula - a distributed, fast open-source graph database featuring horizontal scalability and high availability
- (Golang) VictoriaMetrics - fast, cost-effective monitoring solution and time series database
- (Golang) influxdb - scalable datastore for metrics, events, and real-time analytics
- (Java) trino - fast distributed SQL query engine for big data analytics
- (Java) Apache Doris - an easy-to-use, high performance and unified analytics database
- (Scala) FiloDB - Distributed, Prometheus-compatible, real-time, in-memory, massively scalable, multi-schema time series / event / operational database
- (Rust) ceresdb - high-performance, distributed, schema-less, cloud native time-series database that can handle both time-series and analytics workloads
- (Golang) etcd - distributed reliable key-value store for the most critical data of a distributed system [Raft + gRPC]
- (Java) Apache Zookeeper - highly reliable distributed coordination
- (Golang) chubby - A (very simplified) implementation of Chubby, Google's distributed lock service
- (Java) Kafka - distributed, highly scalable, elastic, fault-tolerant, and secure event streaming platform
- (Python) faust - a distributed stream processing library that ports the ideas from Kafka Streams to Python
- (Golang) Liftbridge - lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system
- (Golang) dkron - distributed, fault tolerant job scheduling system for cloud native environments
- (Python) Celery - distributed task queue
- (Python) Apache Airflow - a platform to programmatically author, schedule, and monitor workflows
- (Golang) nsq - realtime fault tolerant distributed messaging platform designed to operate at scale, handling billions of messages per day [Raft + gRPC]
- (Golang) Sandglass - distributed, horizontally scalable, persistent, time ordered message queue
- (Golang) dnpipes - distributed version of Unix named pipes comparable to AWS SQS
- (PHP) GatewayWorker - distributed realtime messaging framework based on workerman
- (C++) ZeroMQ - abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more
- (Java) Apache Pulsar - distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API
- (Java) Apache ActiveMQ - high performance Apache 2.0 licensed Message Broker
- (Java) ElasticSearch - distributed, RESTful search and analytics engine
- (Java) Apache Lucene - a high-performance, full featured text search engine library
- (Rust) MeiliSearch - Lightning Fast, Ultra Relevant, and Typo-Tolerant Search Engine
- (JS) FlexSearch - memory-flexible full-text search library
- (Golang) RiotSearch - distributed, Simple and efficient full text search engine
- (C++) Typesense - fast, typo tolerant, fuzzy search engine
- (Golang) JuiceFS - Hadoop-compatible AWS S3-compatible high-performance POSIX file system
- (Golang) SeaweedFS - a simple Hadoop-compatible AWS S3-compatible distributed highly scalable distributed file system
- (C) GlusterFS - distributed storage that can scale to several petabytes
- (C++) GlusterFS - highly reliable, scalable and efficient distributed file system. It spreads data over a number of physical servers, making it visible to an end user as a single file system.
- (Golang) etcd - framework for distributed systems development. Provides the core requirements for distributed systems development including RPC and Event driven communication
- (Golang) ergo - port of Erlang/OTP approaches in Golang
- (Golang) gosiris - an actor framework for Golang
- (Python) cotyledon - a framework for defining long-running services. It provides handling of Unix signals, spawning of workers, supervision of children processes, daemon reloading, sd-notify, rate limiting for worker spawning, and more.
- (Java) atomix - fully featured framework for building fault-tolerant distributed systems [REST + Raft]
- (Kotlin) orbit - virtual actor framework for building distributed systems
- (JS) hemera - A Node.js microservices toolkit for the NATS messaging system [RPC]
- (Python) Tooz - centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications
- (C++) Nebula - powerful framework for building highly concurrent, distributed, and resilient message-driven applications
- (GoLang) Service Weaver - A framework that allows to write applications as modular binary and deploy it as a set of microservices
- (Golang) Dragonboat - a high performance multi-group Raft consensus library in pure Go
- (Golang) Golimit - Uber ringpop based distributed and decentralized rate limiter
- (Python) Tenacity - general-purpose retrying library
- (Elixir) ex_hash_ring - pure Elixir consistent hash ring implementation based on the excellent C hash-ring lib
- (Elixir) raft - Raft consensus implementation
- (C++) NuRaft - Raft implementation derived from the cornerstone project
- (Python) Hyx - Lightweight fault tolerance primitives for your resilient and modern Python microservices
- awesome-scalability - Reading list for illustrating the patterns of scalable, reliable, and performant large-scale systems
- awesome-distributed-systems - curated list on awesome material on distributed systems
- awesome-database-learning - a list of learning materials to understand databases internals
- (C/C++)(Book) Build Your Own Redis with C/C++
- (C) (Article) Writing a sqlite clone from scratch in C
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent