ZanRedisDB is a distributed redis cluster with strong consistency.
- Compatible with redis protocol: most redis command is supported in ZanRedisDB, so you can replace Redis with ZanRedisDB and get a more powerful cluster with unlimited data.
- Namespace and set supported: You can use the namespace and set as prefix to isolate different data to make operation more easily.
- Strong consistence and persistence: Use raft and rocksdb to make sure all the data have the consistence replication and persistent on stable storage.
- Scalability: You can easily increase the cluster capacity just by adding more machines.
- Multi-DC support: multi data centers deployment can be supported with rackaware feature.
Install the compress library
yum install snappy-devel (for CentOS) apt-get install libsnappy1 libsnappy-dev (for Debian/Ubuntu) brew install snappy (for Mac)
Build the rocksdb with jemalloc
git clone https://github.com/absolute8511/rocksdb.git cd rocksdb git checkout v6.4.6-patched PORTABLE=1 USE_SSE=1 USE_PCLMUL=1 WITH_JEMALLOC_FLAG=1 JEMALLOC=1 make static_lib
Install the dependency (for old go version only, if using go1.13+, it will be done in go modules):
CGO_CFLAGS="-I/path/to/rocksdb/include" CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lsnappy -lrt -ljemalloc" go get github.com/youzan/gorocksdb CGO_CFLAGS="-I/path/to/rocksdb/include" CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lsnappy -ljemalloc" go get github.com/youzan/gorocksdb (for MacOS)
use the dep ensure to install other dependencies or use go modules for go1.13+
Build zankv and placedriver from the source (only support go version 1.10+, gcc 4.9+ or xcode-command-line-tools on Mac):
ROCKSDB=/path/to/rocksdb make
If you want package the binary release run the scripts
./pre-dist.sh ROCKSDB=/path/to/rocksdb ./dist.sh
- Deploy the rsync daemon which is needed on all server node to transfer the snapshot data for raft
- Deploy etcd cluster which is needed for the meta data for the namespaces
- Deploy the placedriver which is used for data placement:
placedriver -config=/path/to/config - Deploy the zankv for data storage server
zankv -config=/path/to/config - Init a namespace using the create the namespace API in placedriver
- Setting
vm.swappiness=0 - Setting
vm.min_free_kbytesto at least 1GB (8GB on larger memory system) - Disable NUMA zone reclaim with
vm.zone_reclaim_mode=0 - Disable THP(transparent huge pages)
- Avoid the tcp delay ack by
echo 4 > /proc/sys/net/ipv4/tcp_delack_min(for old OS only)
placedriver has several HTTP APIs to manager the namespace
- list the namespace:
GET /namespaces - list the data nodes:
GET /datanodes - list the placedriver nodes:
GET /listpd - query the namespace meta info:
GET /query/namespace_name - create the namespace (handle only by leader) :
POST /cluster/namespace/create?namespace=test_p16&partition_num=16&replicator=3 - delete the namespace (handle only by leader):
POST /cluster/namespace/delete?namespace=test_p16&partition=**
storage server HTTP APIs for stats:
- namespace stats :
GET /stats - namespace raft internal stats :
GET /raft/stats - optimize the data storage :
POST /kv/optimize - get the raft leader of the namespace partition:
GET /cluster/leader/namespace-partition
storage server also support the redis apis for read/write :
- KV:
- Hash Set:
- List:
- Sorted Set:
- ZSet:
Golang client SDK : client-sdk , a redis proxy can be deployed based on this golang sdk if you want use the redis client in other language.
- Redis data structures
- KV
- Hash
- List
- Set
- Sorted Set
- GeoHash
- Expires
- HyperLogLog
- JSON
- Distributed system
- Raft based replication
- Partitions
- Auto balance and migrate
- Support namespace
- High available
- Distributed scan on table
- Rackaware deployment used for across data center deployment
- Searchable and Indexing
- Secondary index support on Hash fields
- Secondary index support for json kv
- Full text search support
- Operation
- Backup and restore for cluster
- More stats for read/write performance and errors.
- Client
- High available for redis commands (Retry on fail)
- Extand redis commands to support index and search
- Extand redis commands for advance scan
- Others (maybe)
- Support configure for Column storage friendly for OLAP
- Pebble as the storage engine
- Support other memory storage engine
- Support export data to other systems
Many thanks for these great projects which make this project possible: etcd, RocksDB, ledisdb, pika.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent