Add pod label with allocation state about agones HOT 8 OPEN

Just today I wished there would be another thing we would update when we allocate: The safe-to-evict annotation. This is rather more annoying. For us Ready pods are safe to evict, but Allocated ones aren't.

@zmerlynn - Have we considered allowing Ready game servers to be evicted?

I think there is a race condition here:

1. The cluster autoscaler decide to remove a node and evict all pods
2. The game server gets allocated and Agones applies the annotation
3. The pod gets drained and the allocated game server is shutdown abruptly

With kubernetes being eventually consistent, there isn't a way to guarantee that the cluster autoscaler doesn't decide to evict and remove a node while game servers are being allocated.

from agones.

The only other concern I have is that if as it doesn't appear that we are currently updating the Pod resource when we allocate (source), then this would increase the load on the kubernetes control plane (by requiring an update to a second object in etcd). This could have performance implications on allocation throughput.

from agones.

Would that really be that bad?

Just today I wished there were another thing we could update when we allocate: The safe-to-evict annotation. This is rather more annoying. For us, Ready pods are safe to evict, but Allocated ones aren't.

So I guess according your statement, that is also not possible?

from agones.

Would that really be that bad?

It depends on your use case. When agones was first envisioned the common scenario was game sessions would last tens of minutes, so doing a bit of extra work on an allocation wasn't a big deal. However, we now also see use cases where game sessions last tens of seconds and having a high allocation rate is critical.

from agones.

So to the original comment on this post:

Would that really be that bad?

Yes 😄 any extra load affects performance.

That being said, if this is something you want, this is where I would strongly advocate for you building this as a third party controller you could add to your Agones cluster (and please open source it! - we'd be happy to take a listing on the website). If it becomes something that's utilised by a large majority of users then we can consider it for main contribution.

We love our ecosystem projects: https://agones.dev/site/docs/third-party-content/libraries-tools/

And this wouldn't be a complicated controller to write either.

Just today I wished there would be another thing we would update when we allocate: The safe-to-evict annotation. This is rather more annoying. For us Ready pods are safe to evict, but Allocated ones aren't.

@roberthbailey captured the potential race condition perfectly.

But if you want to control eviction, https://agones.dev/site/docs/advanced/controlling-disruption/ is the place to read.

from agones.

@towolf What's your session length? And this is GKE if I remember, right?

Adding safe-to-evict=false on allocation is infeasible due to the race that @roberthbailey mentioned - the best we could do in that situation is insist on a long terminationGracePeriodSeconds, but CA by default will ignore tGPS > 10m, so that doesn't entirely close the race. If you're on GKE we might have options to talk about, though - happy to chat on Slack.

from agones.

@zmerlynn We have a wide range of session lengths, some are only a minute, some go on for hours. We kill any gameservers that are still allocated after 24 hours. We have shared "lobby" servers where players meet and can hang out all day. So, I'm not sure if tGPS is suitable for us at all. We're considering somehow shortening the sessions, but they'll never be shorter than 10 minutes.

We are mostly on GKE now but our baremetal contingent is RKE2.

Here's the spectrum:

from agones.

As an FYI, we're working on #3680 to show how to write a small controller to drop into a cluster, to lower the barrier to entry there (the code is remarkably small - you can do it in one main.go).

from agones.