Loki crashes after ResourceExhausted exc about loki-k8s-operator HOT 4 CLOSED

There seems to be two issues here:

1. Queries larger than a certain size are blocked. This is clearly related to the linked bug and should be easy to fix if we are okay increasing the value. We can discus this within the team.
2. Loki crashes. This does not happen in the upstream issue. What resources are available to the Loki pod on your system? It might not be enough to handle such large queries. Do you have the cpu or memory config values set? How much memory does your k8s system have?

from loki-k8s-operator.

@dstathis We are using 1 Gi for memory and cpu config is unset.

In terms of memory:

$ kubectl top pod loki-0
NAME     CPU(cores)   MEMORY(bytes)   
loki-0   19m          711Mi

root@loki-0:/# cat /sys/fs/cgroup/memory.max 
1073741824

the container requests

    resources:
      limits:
        memory: "1073741824"
      requests:
        cpu: 250m
        memory: 200Mi

The particular k8s node has 32 GiB of total memory.

from loki-k8s-operator.

• Are you able to manually temporarily update the loki config with a higher grpc_server_max_send_msg_size? You can juju ssh, apt install vim, edit /etc/loki/loki-local-config.yaml and pebble restart loki. If this works then we could easily add it as a config option.
• Perhaps you could narrow down your LogQL queries so they return less data?

from loki-k8s-operator.

Thanks for your answers. I will close the ticket as the cause of the crash seems to be clearly memory related (137 exit code) and not due to the ResourceExhausted (I saw this also for other queries without the ResourceExhausted), so this probably indicates that we need to increase the memory (1 Gi seems to be too low, just tested with 2 Gi and at first glance it looks better). I wonder if the charm could log a memory-related warning in this case (if loki exits with 137) to avoid the operator having to look into the k8s logs for troubleshooting.

If necessary I will open another issue related to the grpc_server_max_send_msg_size config, I could not reproduce it so far this morning.

And of course narrowing the LogQL queries helps, but sometimes it is hard to narrow when you need to find the root cause of an issue first (so you need to know what to narrow for).

from loki-k8s-operator.