Git Product home page Git Product logo

raftd's Introduction

raftd

raftd 基于raft算法,支持数据持久化(bolt)的分布式KV数据库,可以用于简单的服务发现,基于gin框架提供http服务

Usage

cd cmd/raftd

go build

./raftd -id node01 -haddr 127.0.0.1:8001 -raddr 127.0.0.1:8101 ~/.raftd01

./raftd -id node02 -haddr 127.0.0.1:8002 -raddr 127.0.0.1:8102 -join 127.0.0.1:8001 ~/.raftd02

./raftd -id node03 -haddr 127.0.0.1:8003 -raddr 127.0.0.1:8103 -join 127.0.0.1:8001 ~/.raftd03
curl -X PUT 127.0.0.1:8001/key/test -T ./test.png

curl 127.0.0.1:8001/key/test --output 1.png

步骤:

  1. 启动的时候只make() 然后 调用 tx.ForEach() 遍历整个 db 建立缓存
  2. 更新/删除 KV 的时候同步更新缓存
  3. 集群同步的时候 直接使用 Store.m 进行数据传输

节点同步的步骤:

  1. 先 fsm.db.Close() 然后 截断 data.db
  2. 重新建立 db bolt.Open(...)
  3. range snapshot.m 然后执行一个 Batch Update 将数据同步到 fsm.db 和 fsm.m 中
  4. TODO 失败的情况

租约系统(可以用于服务发现)

Usage

获取租约并续租

# 服务1
ID=$(curl -sL -XPOST 127.0.0.1:8001/lease/grant\?ttl=10\&name=/esq/node-1);while true;do curl -sL -XPOST 127.0.0.1:8001/lease/keepalive/$ID -d 'key=nodeinfo' -d 'data={"http_addr":"127.0.0.1:9001","tcp_addr":"127.0.0.1:9002","node_id":1,"weight":1}';sleep 3;done


# 服务2
ID=$(curl -sL -XPOST 127.0.0.1:8001/lease/grant\?ttl=10\&name=/esq/node-2);while true;do curl -sL -XPOST 127.0.0.1:8001/lease/keepalive/$ID -d 'key=nodeinfo1' -d 'data={"http_addr":"127.0.0.1:9003","tcp_addr":"127.0.0.1:9004","node_id":2,"weight":2}';sleep 3;done

获取键值对

curl -sL -XPOST 127.0.0.1:8001/lease/kv/nodeinfo -d 'prefix=/esq/node'

实现原理

  1. 创建一个DefaultLeaseBucketName的Bucket
  2. 用雪花算法生成一个LeaseId,这个Id作为一个子桶,里面存放共享这个租约的键值对,注意有一个Key是不允许覆写的,KeyName为 meta,存放当前租约的元数据,包括 LeaseTTL LeaseAliveCount LeaseStatus LeaseName
  3. 启动一个 goroutine 每秒检测当前租约的状态,一旦有客户端租用了当前租约,租约的状态就从 CREATE 转为 ALIVE,并且给租约中的所有键值对的TTL减去一,在操作完所有的键值对后,统计TTL不为0的AliveCount,如果AliveCount==0,撤销当前租约(退出当前goroutine,删除这个LeaseId的子桶)
  4. 续租:客户端在申请完租约后,会从服务端获取租约号,使用租约号可以续租,一旦续租,对应租约中的对应键值对的TTL会重新置为这个租约的TTL(从 meta 中获取)

TODO

  • 申请了却从不租用的租约该怎么处理?这些租约的状态为CREATE,如果每次同步状态机时全部重启loopCheck,会消耗一定的资源,如果同步的时候直接删除这些租约属实 鸭子睁眼——大可不必 了,目前有一个可能的解决方案是 PutInPool

  • TimeToLive API

  • 存放租约的树换成LSM-Tree,KV系统属于读多写少,租约系统属于读写都比较多,每秒会有多个协程更新整棵树,可以考虑为租约系统单独做一个存储引擎,准备参考https://github.com/dgraph-io/badger/tree/v1.0.0,这个库比较成熟,值得借鉴

  • 关于 LSM Tree 存放租约 key 应该设计为 LeaseID value 设计为 metaSize(4bytes)valueSize(4bytes)meta(struct_json_bytes)value(map_json_bytes) TODO:考虑使用某个更高性能的json库

raftd's People

Contributors

impact-eintr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

q739916537

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.