Activegate is consuming 20 GB of memory on Openshift cluster about dynatrace-operator HOT 8 CLOSED

How big is your monitored cluster (nodes, workloads, namespaces)?

Ours has 79 nodes, ~2200 workloads and 392 namespaces and we currently use a memory limit of 1GiB which works fine most of the time.

However, we experienced one particular event where the memory consumption suddenly went through the roof with warnings like

WARNING [<collector>] [<platform>, MemoryTrackerImpl] Heap memory shortage detected: 91% memory usage in memory pool 'CMS Old Gen'

spamming the log. We were never able to determine the root cause of that despite involving support... The really bad thing is it went completely unnoticed for several days and caused complete K8s monitoring metric data loss during that period as there were no built-in alerts of any kind and neither the pod's liveness nor readiness probes seemed to be bothered by this so the pod didn't get auto-killed.... The workaround was simply to kill the AG pod manually.... Luckily this happened only once so far...

We keep seeing occasional warnings as the one above although without any noticeable negative effects so currently we keep sticking to a memory limit of 1GiB and instead created custom events for alerting on missing metric data and high heap memory consumption...

Operator version: 0.3.0
AG version: 1.225.137

from dynatrace-operator.

What will happen if you set the pod's memory limit to the "default" 1GiB?

Did you ever contact support about your issue?

Note that in our case support told us that a memory limit of 1GiB should usually be "more than enough"... so 20GB seems totally unreasonable and I wonder why you decided to set a memory limit that allowed for such high memory consumption in the first place,...

Also, I suppose increasing the memory limit for no actual reason will automatically cause the JVM to adapt its memory pool sizes accordingly and possibly grab all the memory it can even if it would be running fine (or even better) with just 1GiB...

from dynatrace-operator.

We are having 6 nodes , approx 500 pods running in 100 namespaces , still memory consumption is really high for dynakube-activegate agent. We need some idea how we can debug and and resolve this concern.

from dynatrace-operator.

Hi, if you are running the Activegate without any limits applied it can use up to 50% of the available resources, which is in your case as high as 20gb i guess. We're also aiming to provide sizing guidline and built-in defaults as soon as possible. In the meanwhile I would suggest to set memory limits on your own in order to decrease usage of memory. We suggest to giving 1GB to AG and see on sfm metrics how it behaves.

from dynatrace-operator.

any changes expected for this AG mem behaviour in 0.4.0 ?

from dynatrace-operator.

Not in 0.4.0 since it was released today, we are still working on providing sizing guideline and defaults as I said. But anyway, you can set limits on your own at anytime

from dynatrace-operator.

any update? same issue when using activegate on AWS EKS

from dynatrace-operator.

The conclusion is, that when facing this issue using the ActiveGate, memory limits should be set by setting the appropriate fields in the custom resource spec.

E.g.:

activeGate:
  resources:
    limits:
      memory: 1G

If you experience problems such as missing events or log spamming as mentioned above, please contact the ActiveGate support, as there is not much we can do from the Operator side.

from dynatrace-operator.