[feature] --reload about cloudprober HOT 9 CLOSED

@MrHohn got the reload working by adding an annotation on the Deployment based on the config's sha256 sum, this pattern is described by Helm on its docs
This ensures pods will roll whenever the contents of the configmap change.

from cloudprober.

FWIW, one potential use case would be running cloudprober with kubernetes cluster. With the configuration mounted via a configMap (like what #89 does), we can easily implement dynamic probing by editing the configMap in-flight.

from cloudprober.

Dynamic reloading of config in cloudprober is non-trivial to implement. Cloudprober forks off quite a few goroutines to run probes and other components like servers and surfacers. It also opens a bunch of listening sockets. We use contexts in most cases to keep track and shutdown these goroutines, but it's hard to guarantee that everything will clean up nicely and in time, in the case of a context close.

That said, there are a few options here:

1. We can provide an option for cloudprober to quit in case of a config change (and if config parses successfully), and let the process/container manager restart cloudprober. This will be cheap and clean to implement but it's disruptive.

2. Have a sidecar process watch the config and restart cloudprober with new config in case of a config change. Sidecar process could then monitor the main process after the restart and if it fails to come back up and become healthy, it can restart the main process with the old config.

3. Have cloudprober reload the config dynamically. Implement tests to make sure that cloudprober goroutines and listener sockets can be stopped cleanly.

from cloudprober.

Thinking more about it, I am not sure if I understand the requirement and use case clearly.

@MrHohn, regarding your comment about reloading config in response to configMap changes, I was wondering if that will not be an unsafe thing to do? There will be no way to rollout this change slowly as configMaps themselves don't seem to have a way to rollout slowly. Can you please comment if that's okay in your use case? Also, will it be sufficient for cloudprober to quit itself in response to config change?

from cloudprober.

This stackoverflow thread discusses the same issue:
https://stackoverflow.com/questions/37317003/restart-pods-when-configmap-updates-in-kubernetes

from cloudprober.

Thanks for getting back @manugarg. Now I think about this, it makes much sense that we handle this one layer above cloudprober. As you said, it is non-trivial to implement and hard to guarantee safety with cloudprober alone.

Simply making clourprober quit itself when config changes may work but sounds a bit too tangled. It also won't have a way to stop the rollout if new config is found broken.

from cloudprober.

@cilindrox That's great. Thanks for sharing.

from cloudprober.

If this helps, I would like the reload to be implemented as I want to hot reload without restarting just like many other daemons do it.
If being worried about breaking config-maps, cloudprober could have the approach of keeping the old config if the reloaded config fails. Just like prometheus does, and expose a metric last_succesful_config_reload that way the prober will keep running and you have a way to check if the config-map broke the config.
Off course this is only safe for syntax errors, doesn't cover errors where the logic is wrong but the syntax checks out.

from cloudprober.

@brat002, @fcastello, @MrHohn and other folks, Please see the announcement here: #679. Active development of this repository is being moved to github.com/cloudprober/cloudprober. I am closing this issue as I think the core of it is resolved by using config checksum in deployment annotations like this:
https://cloudprober.org/how-to/run-on-kubernetes/#deployment-map

This allows Kubernetes to rollout config changes, instead of Cloudprober rolling out changes by itself.

If you disagree, please feel free to file another issue at github.com/cloudprober/cloudprober.

Thanks,
Manu

from cloudprober.