nats-io / graft Goto Github PK

Hi - our question is "is it possible to specify for a cluster option values for

• HEARTBEAT_INTERVAL
• MIN_ELECTION_TIMEOUT
• MAX_ELECTION_TIMEOUT

In place of the default hard coded values. We are looking at a use case where a slower heartbeat interval would suffice. Currently the only solution is to fork and update constants.

Is this on a feature roadmap or would you consider a PR to making the change ... and if so any advise on how you would suggest it is done?

Thanks.

Hey there,

Do you have any plans to include support for dynamic membership changes to the raft cluster?

Cheers

When a node switches state, it invokes go n.handler.StateChange(old, state) in switchState, this is wrong since it does not guarantees the order of the state changes. If in time a node goes from FOLLOWER to CANDIDATE and then CANDIDATE to LEADER, the state change channel may end-up having the two state changes reversed due to the execution of a go routine to report the change of the state.

The RAFT protocol defines how to handle configuration changes, we need to implement.

Currently the NewNatsRpc func creates a new NatsRpcDriver using a *nats.Options parameter. We already have a NATS connection in our app and were thinking about reusing it here instead of having the driver create a new connection. Is there a reason why we shouldn't reuse the connection and, if not, would you consider adding a new method to create a NatsRpcDriver from an existing *nc.Conn?

Use something more efficient.

the example usage is too simple.
can you give me a fully usage, I can run it

graft currently uses nats v1.9.2, but when it is included in a project that uses NATS v1.10.0, the cluster never elects a leader.

All members get stuck as candiates in the leader election process.

Steps to reproduce

1. git clone [email protected]:nats-io/graft.git
2. go get github.com/nats-io/nats.go@latest (v1.10.0)
3. go test ./...

Expected result

ok  	github.com/nats-io/graft	Xs
?   	github.com/nats-io/graft/pb	[no test files]

Actual result

--- FAIL: TestNatsLeaderElection (1.13s)
    test_helper.go:51: Cluster doesn't match expect state: expected 1 leaders, 4 followers, 0 candidates, actual 0 Leaders, 0 followers, 5 candidates

        1 - github.com/deixis/graft/test_helper.go:124
        2 - github.com/deixis/graft/nats_rpc_test.go:62
        3 - ...
FAIL
FAIL	github.com/nats-io/graft	Xs
?   	github.com/nats-io/graft/pb	[no test files]
FAIL

I am not sure whether that is a graft, a NATS problem, or if you even consider it as an issue.

Hi,

Thanks for providing graft. I like how simple graft is to use and understand. I was able to accomplish what I was looking for with graft so far, simple leader election. But then I realized that I might need a way to propose node state change. Are there any plans for it or is there an alternative pkg in nats-io that I can look at?

Thank you.

HI, we are seeing very frequent leadership changes (every ~5 minutes) while running on GKE (but have heard it happens on EKS as well). I believe this happens whether we run with the NATS Streaming bus or Jetstream bus. And at the time the bus is basically idle.

If I change the MIN_ELECTION_TIMEOUT value from 500 msec to 1 sec they almost entirely go away (running with the Jetstream bus). The heartbeats go over NATS itself, right?

Any advice on what could be causing this? I don't believe we have any special configuration settings on NATS that would be causing any sort of rate limiting.

there is not any chance to run this library without NATS server or I am wrong? I just copy the sample (in the first page) and run, got error "nats: no servers available for connection".

Can I use this raft library without nats server? any sample?

Thank you

We're using graft for our consumer deployment's leader election. I'm noticing something happening while testing in GKE that I've never seen while testing on my local laptop (using k3d for Kubernetes). Very frequently (every couple minutes or so) the leader is going down - I guess it must be getting some RPC message? Not sure why. Our code looks like:

    rpc, err := graft.NewNatsRpc(opts)
    if err != nil {
        log.Fatalw("failed to new Nats Rpc", zap.Error(err))
    }
    errChan := make(chan error)
    stateChangeChan := make(chan graft.StateChange)
    handler := graft.NewChanHandler(stateChangeChan, errChan)
    node, err := graft.New(ci, handler, rpc, "/tmp/graft.log")
    if err != nil {
        log.Fatalw("failed to new a node", zap.Error(err))
    }
    defer node.Close()

    cctx, cancel := context.WithCancel(ctx)
    defer cancel()

    if node.State() == graft.LEADER {
        log.Info("I'm the LEADER, starting ...")
        go callbacks.OnStartedLeading(cctx)
    } else {
        log.Info("Not the LEADER, stand by ...")
    }

    handleStateChange := func(sc graft.StateChange) {
        switch sc.To {
        case graft.LEADER:
            log.Info("I'm the LEADER, starting ...")
            go callbacks.OnStartedLeading(cctx)
        case graft.FOLLOWER, graft.CANDIDATE:
            log.Infof("Becoming a %v, stand by ...", sc.To)
            dat, err := os.ReadFile("/tmp/graft.log")
            if err != nil {
                log.Debugf("error reading /tmp/graft.log: %v", err)
            } else {
                log.Debugf("/tmp/graft.log contents: '%s'", dat)
            }
            if sc.From == graft.LEADER {
                cancel()
                callbacks.OnStoppedLeading()
                cctx, cancel = context.WithCancel(ctx)
            }
        case graft.CLOSED:
            if sc.From == graft.LEADER {
                cancel()
                callbacks.OnStoppedLeading()
            }
            log.Fatal("Leader elector connection was CLOSED")
        default:
            log.Fatalf("Unknown state: %s", sc.To)
        }
    }

And we are hitting that case graft.FOLLOWER, graft.CANDIDATE: line.

I added the log line to dump the /tmp/graft.log file. Not sure if that's interesting but this last time it looked like:
'{"SHA":"eyJDdXJyZW50VGVybSI6MjksIlZvdGVkRm9yIjoiYTM1NjI4NjI4ODI2YmQ0YWMzYzIyYWQ2OTIifdo5o+5ea0sNMlW/75VgGJCv2AcJ","Data":"eyJDdXJyZW50VGVybSI6MjksIlZvdGVkRm9yIjoiYTM1NjI4NjI4ODI2YmQ0YWMzYzIyYWQ2OTIifQ=="}'