Memory and CPU leak in agent and service about indexer HOT 7 OPEN

Looks like it's leaking CPU cycles too:

from indexer.

Looks like indexer-agent and indexer-service still have a memory leak problem. I've been using a Kubernetes CronJob to automatically restart them every day to keep memory usage under control. Here's a gist for anyone that wants to do the same: https://gist.github.com/chriswessels/8271f82a0ae7342d7d0822ea1e796246

from indexer.

I'm also observing this issue on v0.18.6. However I've found a good way to accelerate it: send lots of GraphQL queries to the indexer-agent management endpoint. I've also noticed that the queries would take longer to execute as memory usage grows.

However I am not sure if certain types of queries are worse than others. Read below.

Context
As @fordN and @chriswessels already know, I'm currently running experiments that require constantly updating the Agora models. In this particular case, I am updating the model variables for each subgraph every 3 minutes.
In the plot below, every drop is the OOM killer killing the indexer-agent. The frequency increased around May 10th, when I deployed my experiment to the indexer.

from indexer.

Update: The problem is still present on v0.19.2.

from indexer.

Update: immediately preceding the OOM crash, queries begin failing. This causes a drop in quality of service.

from indexer.

Update: the problem is still present on v0.20.12

from indexer.

Figment appears to be experiencing this issue too (shared with me on March 14, 2024)

from indexer.