panic: close of closed channel about prometheus HOT 2 OPEN

Thank you for providing the logs!

This looks like a bug and not it Is not clear if it is a regression. I will try to reproduce and provide a fix.

from prometheus.

I just confirmed my hunch:

This panic can happen if the configuration for the rule groups is reloaded while prometheus is being shutdown:

• t1: Ruler is shutdown here after the process receive the TERM signal here. During shutdown, the groups are closed.
• t2: process receive the HUP signal here
  and configuration is reloaded. Please notice that this actor hasn't received the cancel from the channel yet because the shutdown of actors is processed sequentially. During the configuration reload, the groups are closed again, causing the panic.

One approach to solve this problem that I'm thinking of is to make the Reload method in the manager noop in case the manager has already stopped. It does not make sense to reload a closed manager.

Another alternative is to make the Reload function error in case it is already closed, but also include a flag that allow consumers of the manager to know its state so that they don't call Reload.

I'm inclined to implement the first approach.

from prometheus.