Make Argo Workflows Deployment Restart on Config change using Helm ConfigMap annotation about argo-helm HOT 3 OPEN

I initially requested in the wrong repo argoproj/argo-workflows#12876

from argo-helm.

I agree with @agilgur5. I think this should be a toggle(s) in the helm chart values file. Either use the config watcher to hot-reload config changes, or use the helm checksum/config annotation convention to automatically restart both the server and controller deployments on config changes. Note that we always need to use the checksum/config annotation for the server until argoproj/argo-workflows#10970 has been resolved.

If the toggle is enabled:

• Set the WATCH_CONTROLLER_SEMAPHORE_CONFIGMAPS env var to false for the controller.
• Annotate both the server and controller deployments' pod templates with the checksum/config annotation to get them to automatically restart on config changes.

If the toggle is disabled (the default value):

• Omit the WATCH_CONTROLLER_SEMAPHORE_CONFIGMAPS env var for the controller, or explicitly set it to true.
• Annotate only the server deployment's pod template with the checksum/config annotation.

from argo-helm.

As I wrote in the upstream issue, this feature should be optional for the Controller as it is built into Workflows already. The value should also be explicitly documented as such.
Note that there is also currently no way to turn off the built-in feature without turning off other features, like the semaphore ConfigMap watcher, per argoproj/argo-workflows#12622.

This also would be good to have for the Server, which does not yet have a built-in watcher: argoproj/argo-workflows#10970

and much cheaper in terms of resource.

For this part, I don't know that this is necessarily correct. Disabling the watcher will certainly use less constant memory, but it may not be significant. In terms of CPU, disk I/O, and network I/O, restarting the entire deployment instead of an in-memory hot reload is certainly more expensive (and creates churn on the cluster).

I think it would be more appropriate to call it a trade-off when it comes to performance.

You could also call it an optional workaround for a current bug (and any future such bugs), which is caused by an upstream k8s issue, and which can cause performance degradation.

from argo-helm.