Inspire by

• scaling-services-with-shard-manager
• MirrorMaker 2.0
• The Chubby lock service for loosely-coupled distributed systems

We build sm to handle the backend service which need partition data into several instance(k8s pod or actual machines, use container below), the data in each container may be seen as a job. sm developed base on etcd, to make sure that:

• When container changed, some shard moves will be made.
• When shard changed, some shard moves will be made.
• When shard or container load changed, shard moves(TODO).

• Getting Started
  • Installing
• Concept explanation
  • Container
• Example
• Question
  • Question

Install etcd, test with etcd-v3.4.13

Install go >=go1.17.1, and run:

go get https://github.com/entertainment-venue/sm/server

go mod tidy

go run main.go --config-file sample.yml

Container is the resource for shard to run in, and Container establish Session with etcd, the Session is used for the management of keys in etcd, management meaning that:

• Container goes down, heartbeat which from container to etcd server will be ended.
• Only the unchanged properties of app or shard will stay in etcd permanently, make data to be managed as clean as the moonlight.

Container also define common protocol which need sharded application who want to integrate with sm:

type ShardInterface interface {
	Add(id string, spec *ShardSpec) error
	Drop(id string) error
}

You can implement the ShardInterface and inject the implementation into the Container with WithShardImplementation, and also wrap common http api to interact with the sm server. The keep http path:

• /sm/admin/add-shard
• /sm/admin/drop-shard

For more information, you can read the source code for Container.

You can see the test code as tip to understand how to construct you own sharded application:

https://github.com/entertainment-venue/sm/blob/main/client/client_test.go#L13

If you want to no better how the sm server running, see the code below as an entry point:

https://github.com/entertainment-venue/sm/blob/main/server/main.go

Please feel free to report any issue or optimization suggestion in Issues