Git Product home page Git Product logo

sharkstore's Introduction

SHARKSTORE

sharkstore是一个分布式的持久化K-V存储系统,存储层依赖rocksdb,数据副本之间通过raft协议进行复制

系统主要包括
元数据管理模块:master-server(golang),
业务数据存储模块:data-server(c++),
访问代理层:gateway-server(golang),
管理端:console(golang)

目录结构:
. |-- README.md
|-- console web管理端
|-- data-server 业务数据存储服务
|-- glide.yaml
|-- master-server 元数据管理服务
|-- model
|-- pkg-go 公共模块
|-- proxy
| |-- gateway-server sql/http rest代理层

架构设计

详细查看arch.md

安装说明

详细查看INSTALL.md

测试数据:

压测表名:metric,3台dataserver物理机【NVMe盘】,
压测表metric有12个column,每个column都为int类型,
前4个column(salt,key,host,ts)组成联合索引做为key,
对metric表预分裂100个range,三个副本,raft复制

批量插入:

一次批量插入100行记录:测的最大TPS为80W笔/秒,平均响应时间为56ms,累计插入了610亿条数据

查询负载测试:

  1. 根据salt+key+host+ts查询单条数据:平均响应时间为5ms

  2. 根据salt+key+host进行查询(10-100条记录):平均响应时间为50ms,所查数据有了缓存之后,平均响应时间8ms。

features


  • SQL syntax and rest handle Users can access with sql client directly, and also by sdk with rest method api.

  • Dynamic table scheme Table columns which need be added or renamed is allowed.

  • Pre-sharding Table can be created pre-sharding.

  • Data global sorted User can global scan by primary key.

  • Strong consistency. Data replicated as raft group.

  • Online scalability, auto failover, auto rebalance. Auto schedule

  • NVMe+SPDK in testing

License

under the Apache 2.0 license. See the LICENSE file for details.

sharkstore's People

Contributors

dingjun84 avatar jonahgao avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.