Handle housekeeping with sharded cluster about yorkie HOT 2 OPEN

The options that you've suggested look good to me.
I have listed some of the considerations for these options below.

1. Only the master server executes housekeeping in the cluster.

• Implementation: Using etcd with the raft leader election algorithm will ease this implementation. We can use K8s go client leader election package with etcd for this operation:
  • Package: https://pkg.go.dev/k8s.io/client-go/tools/leaderelection
  • Example: https://github.com/travisjeffery/kube-leaderelection.
• Pros: It looks easy to implement.
• Cons: As workload for housekeeping increases, housekeeping overhead on the master server will increase, which may lead to server panic or failure. 

2. Split housekeeping tasks per project and execute them on servers mapped by project.

• Implementation: We need to map the server with the project range to housekeeping in etcd and pull this information before housekeeping. Maybe we can reference Yorkie's etcd implementation in the sync package for this task
  • Yorkie sync package's Membership: https://github.com/yorkie-team/yorkie/blob/3022eecebad070db32d06ba3c762d54875743dc4/server/backend/sync/etcd/membermap.go
• Pros: We can distribute housekeeping workload just like document workload sharding, which will ease the overhead of servers to handle housekeeping.
• Cons: Implementation seems to be more difficult than the first option. 

These options seem like opposite options to me, but I think they both can be implemented by using etcd.
I'll keep searching for more information on these options.

from yorkie.

I have implemented leader election with K8s lease object, but this will make yorkie k8s dependent.

Therefore, I will change this implementation with MongoDB, where yorkie already have dependency on.

To be specific about the implementation, I will use the way of nodes try to update on the same document(same _id) by checking lease_expire_at field.

Also, to ensure that only one node becomes the leader when a lease has expired, I will use TTL indexes to remove document on expiry, preventing nodes to update the same document simultaneously.

from yorkie.