Agones Controller has OOMKilled in version 1.35.0 about agones HOT 4 CLOSED

If I look at the installed controller that I've installed via Helm (this is the dev version), we don't set any cpu or memory limits by default.

  agones-controller:
    Container ID:   containerd://d9507676a30507a6bbfb5f0e1ddbe24e089b59874e0b78193eb3b98dff2e9426
    Image:          us-docker.pkg.dev/agones-mark-dev/images/agones-controller:1.36.0-dev-b148d3b
    Image ID:       us-docker.pkg.dev/agones-mark-dev/images/agones-controller@sha256:16182889ecf89777799f2801ea73b183915044281a6765aeadc36482d8a8bc2b
    Ports:          8081/TCP, 8080/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 19 Oct 2023 13:19:23 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      ephemeral-storage:  10100Mi
    Requests:
      ephemeral-storage:  10100Mi

It looks like you've set the memory limit to 500Mi ? This may just be too low a memory limit for the Agones controller - it can be quite memory hungry at times.

What happens if you disable the memory limit?

from agones.

If I look at the installed controller that I've installed via Helm (this is the dev version), we don't set any cpu or memory limits by default.

  agones-controller:
    Container ID:   containerd://d9507676a30507a6bbfb5f0e1ddbe24e089b59874e0b78193eb3b98dff2e9426
    Image:          us-docker.pkg.dev/agones-mark-dev/images/agones-controller:1.36.0-dev-b148d3b
    Image ID:       us-docker.pkg.dev/agones-mark-dev/images/agones-controller@sha256:16182889ecf89777799f2801ea73b183915044281a6765aeadc36482d8a8bc2b
    Ports:          8081/TCP, 8080/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Thu, 19 Oct 2023 13:19:23 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      ephemeral-storage:  10100Mi
    Requests:
      ephemeral-storage:  10100Mi

It looks like you've set the memory limit to 500Mi ? This may just be too low a memory limit for the Agones controller - it can be quite memory hungry at times.

What happens if you disable the memory limit?

#3692
This PR has already fixed the issue.

The reason is that when a deadlock occurs, the k8s event gets blocked and cannot be processed, leading to a continuous increase in cache memory, which ultimately causes an OOM (Out of Memory) issue.

from agones.

The previous issue is here:
#3380

from agones.

Closing!

from agones.