OSM Agent high memory usage about docker-provider HOT 43 CLOSED

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

up

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

Are there any updates? I'm having the same issues in a brand new cluster.

from docker-provider.

Looks like no. I have to write anything every 7 days here just to prevent this issue to be closed automatically.
And also it looks like nobody cares about this. For a half of a year this issue is opened already.

from docker-provider.

Hi, @EderRoger , @MasterKolka , can you please share your cluster configuration in terms of the VM SKU, k8s version, number of nodes in the cluster, workload info- number of pods/containers etc. along with the AKS cluster name to understand & investigate what contributing this memory usage? We have seen issue, where memory usage higher (keep increasing) if the cluster has network issues and we are rolling out the agent version which would address this specific issue. Without investigate the agent logs from your cluster, I am not sure whether the agent which we are rolling out would address or not, the memory usage issue which you guys are referring here. We have some work items to improve the memory usage with regards to turning off Prometheus sidecar container if its not used.

from docker-provider.

@ganga1980 I wrote all info there - #624
As for now, it is reproduceable on any new empty cluster. 300mb of ram per omsagent instance + 150mb of ram per omsagent-prometeus instance. Which takes from a 2-nodes kube cluster 300x2 (omsagent, ds) + 150x2 (omsagent-prometeus, ds) + 300 (omsagent, rs) = 1200mb of ram.

from docker-provider.

@ganga1980 I wrote all info there - #624 As for now, it is reproduceable on any new empty cluster. 300mb of ram per omsagent instance + 150mb of ram per omsagent-prometeus instance. Which takes from a 2-nodes kube cluster 300x2 (omsagent, ds) + 150x2 (omsagent-prometeus, ds) + 300 (omsagent, rs) = 1200mb of ram.

Thanks for the feedback @MasterKolka . We have work items to investigate & optimize the overall memory usage and as part of this optimization, first thing, we are planning to push the release to turn off all the processes in omsagent-Prometheus sidecar container if there is no scrape config. We can circle back as make progress on this, as its takes considerable time to bring down overall memory usage.

from docker-provider.

Hi @ganga1980 , my cluster infos is pretty similar.
Thanks for sharing @MasterKolka and thanks for looking into this @ganga1980

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

up

from docker-provider.

up

from docker-provider.

up

from docker-provider.

up

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

up

from docker-provider.

up

from docker-provider.

Hi, @MrImpossibru , As part of our next agent release, we are pushing the memory optimizations as described in this PR - #769

from docker-provider.

@ganga1980 Cool. At least it will roll back the situation with memory consumption like it was last October.

from docker-provider.

up

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

up

from docker-provider.

up

from docker-provider.

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from docker-provider.

up

from docker-provider.

Hi, @MrImpossibru , we already rolled out the optimizations which I mentioned earlier. That should have been rolled out to your clusters already.

from docker-provider.

@ganga1980 I'll check. If omsagent still uses 300mb per instance and replicaset exists in the cluster (nobody described it's purpose) I'll reopen this issue by creating another one.
This was an initial issue, Prometheus agent was added later and made it worse.

from docker-provider.