Comments (8)
How big is your monitored cluster (nodes, workloads, namespaces)?
Ours has 79 nodes, ~2200 workloads and 392 namespaces and we currently use a memory limit of 1GiB which works fine most of the time.
However, we experienced one particular event where the memory consumption suddenly went through the roof with warnings like
WARNING [<collector>] [<platform>, MemoryTrackerImpl] Heap memory shortage detected: 91% memory usage in memory pool 'CMS Old Gen'
spamming the log. We were never able to determine the root cause of that despite involving support... The really bad thing is it went completely unnoticed for several days and caused complete K8s monitoring metric data loss during that period as there were no built-in alerts of any kind and neither the pod's liveness nor readiness probes seemed to be bothered by this so the pod didn't get auto-killed.... The workaround was simply to kill the AG pod manually.... Luckily this happened only once so far...
We keep seeing occasional warnings as the one above although without any noticeable negative effects so currently we keep sticking to a memory limit of 1GiB and instead created custom events for alerting on missing metric data and high heap memory consumption...
Operator version: 0.3.0
AG version: 1.225.137
from dynatrace-operator.
What will happen if you set the pod's memory limit to the "default" 1GiB?
Did you ever contact support about your issue?
Note that in our case support told us that a memory limit of 1GiB should usually be "more than enough"... so 20GB seems totally unreasonable and I wonder why you decided to set a memory limit that allowed for such high memory consumption in the first place,...
Also, I suppose increasing the memory limit for no actual reason will automatically cause the JVM to adapt its memory pool sizes accordingly and possibly grab all the memory it can even if it would be running fine (or even better) with just 1GiB...
from dynatrace-operator.
We are having 6 nodes , approx 500 pods running in 100 namespaces , still memory consumption is really high for dynakube-activegate agent. We need some idea how we can debug and and resolve this concern.
from dynatrace-operator.
Hi, if you are running the Activegate without any limits applied it can use up to 50% of the available resources, which is in your case as high as 20gb i guess. We're also aiming to provide sizing guidline and built-in defaults as soon as possible. In the meanwhile I would suggest to set memory limits on your own in order to decrease usage of memory. We suggest to giving 1GB to AG and see on sfm metrics how it behaves.
from dynatrace-operator.
any changes expected for this AG mem behaviour in 0.4.0 ?
from dynatrace-operator.
Not in 0.4.0 since it was released today, we are still working on providing sizing guideline and defaults as I said. But anyway, you can set limits on your own at anytime
from dynatrace-operator.
any update? same issue when using activegate on AWS EKS
from dynatrace-operator.
The conclusion is, that when facing this issue using the ActiveGate, memory limits should be set by setting the appropriate fields in the custom resource spec.
E.g.:
activeGate:
resources:
limits:
memory: 1G
If you experience problems such as missing events or log spamming as mentioned above, please contact the ActiveGate support, as there is not much we can do from the Operator side.
from dynatrace-operator.
Related Issues (20)
- How to find be notified on which dynakube params to migrate? HOT 1
- customProperties settings for ActiveGate are not reconciled and changes are not deployed HOT 1
- [operator v0.12.1, classicFullStack] OneAgent pods deployed on EKS/Bottlerocket AMI not running HOT 1
- oneagent readiness probe fails and writes huge amount of logs HOT 1
- dynatrace-oneagent-csi-driver HOT 1
- Readiness and liveness probe failing with default chart config HOT 1
- Dynakube automatic kubernetes monitoring - Do not override environment settings HOT 3
- Missing cpu/memory request/limits from csi-init container HOT 2
- Dynatrace Operator - Not granting the Dynatrace Operator access to image pulling and managing causes a hard-crash in v0.12.x+ HOT 4
- GKE Autopilot Mutating Webhook Namespace Selector HOT 3
- Missing release notes for operator v0.14.0 HOT 1
- PodDisruptionBudget for DT ActiveGate
- SkipCertCheck doens't work on Operator 14.x HOT 1
- Operator should not depend on cluster.local domain HOT 1
- Add affinity support for DT oneAgent HOT 1
- dynatrace-oneagent-csi-driver provisioner gets OOMKilled HOT 2
- oneagent-csi-driver provisioner crashes with OOM, resource limit is set too low? HOT 3
- Injection of java agent failed in ARM64 EKS nodes HOT 1
- A potential risk of Dynatrace Operator makes a worker node get the token of any Service Account HOT 1
- Add ability to manage dashboards via Kubernetes manifests HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dynatrace-operator.