Add label according to node's pool about digitalocean-cloud-controller-manager HOT 23 CLOSED

I envision a case of scaling a cluster up and down to cater for base VS peak load. In this case it seems like a good idea to use different node pools to do so. It would allow to pin resources that have to run to ensure base functionality to a specific node pool. A generic transpose of tags as suggested in #135 into labels would also work as we could tag the droplets accordingly.

from digitalocean-cloud-controller-manager.

We have a dedicated repository by now to track generic feature requests and bug reports for DOKS. Further updates and discussions will go through digitalocean/DOKS#3.

from digitalocean-cloud-controller-manager.

So this is a bit tricky in my opinion. Labels in Kubernetes have more strict, and generally different, validation than do tags in DigitalOcean. For instance, by default DigitalOcean Kubernetes node pools are tagged with k8s:${cluster_id}. This tag would be invalid in Kubernetes since colons are not allowed in neither the label key, nor the value. Ignoring validation, in my opinion it would be nice if instead of this creating a digitalocean.com/droplet-tag/k8s:${cluster_id}="" label, it created digitalocean.com/droplet-tag/k8s=${cluster_id} (ignoring the fact that label key should really be more explicitly named such as digitalocean.com/droplet-tag/cluster-id=${cluster_id}). Not being able to achieve setting a value on labels feels like we're nerfing the power of Kubernetes.

I'm sure others have better ideas around this, but one naïve approach to achieve this functionality could be to treat the colon as an equals, thus pseudo-creating key-value pairs. This of course is limited by the fact that tags in DigitalOcean can have multiple colons, so it would need to be well documented that those tags would never be synced. If DigitalOcean could support explicit key-value tags delimited by an equals sign, that would serve as the easiest fix :)

from digitalocean-cloud-controller-manager.

@normanjoyner thanks for elaborating, that does make a lot of sense. I have created an internal ticket to track progress on this one.

from digitalocean-cloud-controller-manager.

For those interested, I noticed DOKS node pools now have the doks.digitalocean.com/node-pool-id label, with corresonding node pool UUID value. Note: this does not yet include support for arbitrary tag syncing. Thanks @timoreimann!

from digitalocean-cloud-controller-manager.

Hey @wnagele, we don't add labels per node pool, but we do have lables for droplet size (see here). Would that still accomplish what you need?

from digitalocean-cloud-controller-manager.

🤔 worker pools are tags on the droplet side.
Wondering if we should surface tags as labels on k8s nodes

from digitalocean-cloud-controller-manager.

related #135 :)

from digitalocean-cloud-controller-manager.

@wnagele thanks for opening this! We're interested in this exact use case for integrating the Cerebral autoscaler - see this related issue: https://github.com/containership/cerebral/issues/52. This is the only thing blocking us from having a fully functional autoscaling integration on DO Kubernetes.

@andrewsykim I did notice that #142 is ready (?) but sitting around going stale. Are you able to provide any insight on an ETA for this going in? Thanks!

from digitalocean-cloud-controller-manager.

Hi @mattkelly! We're currently discussing what the best label format should be as it will be difficult to change once it's deployed. Please let me know if you have any opinions on this, right now it's digitalocean.com/droplet-tag/<tag>. cc @joonas

from digitalocean-cloud-controller-manager.

Good catch on the label validation @normanjoyner

from digitalocean-cloud-controller-manager.

Going through the various requests received, what seems to be the primary use case expressed by most users is the ability to select node pools and subsequently apply certain actions on them, such as scaling or scheduling. This could already be accomplished by setting node pool identifiers as labels on the affected nodes (which has also been mentioned by @normanjoyner).

Support for a way to apply (and persist) arbitrary labels to DOKS-managed nodes is still on the radar; as others have pointed out as well, however, translating tags to labels isn't straight-forward and comes with certain limitations. That's why we want to spend more time thinking about a more flexible labeling mechanism (where one possibility could be to expose a label interface directly instead of going through DO tags) and focus on node pool identifiers first to serve the immediate needs.

Let me know what your thoughts are on the proposed path forward.

from digitalocean-cloud-controller-manager.

To me, this seems like a reasonable path forward. I would love to see the node pool ID as the label value, as opposed to part of the key with an empty value. The same goes for cluster ID (as outlined in my comment above).

from digitalocean-cloud-controller-manager.

Thanks for your feedback, @normanjoyner. I think canonical label values is what we are aiming for now.

Can I ask what you'd like to use the cluster ID for if it were attached as a label (ideally in proper key/value form)? I presume this is for discovery purposes which the DO API should also allow hopefully? Just trying to understand use cases better so that we can account for it. Thanks!

from digitalocean-cloud-controller-manager.

From the perspective of a Kubernetes application developer, the more “metadata” exposed through labels, the better. With well-defined labels such as digitalocean.com/cluster-id (as an example), I can rely on the label value to make API calls to the DigitalOcean API that affect the cluster my application is running on, without additional configuration from my end-user. Ultimately this makes my application more portable between DOKS clusters. Furthermore, when thinking about running and managing multiple DOKS clusters, I am able to rely on this label for more advanced scheduling and placement techniques.

from digitalocean-cloud-controller-manager.

@timoreimann any update on when this will be released?

The last DO Kubernetes platform update (12 Feb) caused all labels and taints to be removed from our nodes which brought down the entire cluster as deployments were not able to schedule as they relied on the labels and taints. Not looking forward to that happening ever again..

from digitalocean-cloud-controller-manager.

Thanks for sharing the news @normanjoyner, I missed to post an update to this issue.

from digitalocean-cloud-controller-manager.

@boedy apologies for the inconveniences this has caused; unfortunately, we have no news to share yet as to when we will be able to provide better support for labels (and taints).

The best advice I can give for now is to create a custom controller that reconciles labels/taints based on node pool labels. I know it's not ideal but might be a workaround for you and others.

from digitalocean-cloud-controller-manager.

I think the original issue opened by @wnagele is satisfied by the addition of the doks.digitalocean.com/node-pool-id label. Related issues, such as the syncing of arbitrary DigitalOcean tags as labels, have been raised in this thread (and work was started in #142, which was subsequently closed with good reason). I’m happy to encapsulate these points into a new issue to track moving forward if we feel this issue can be closed out. I think this is functionality that the wider DigitalOcean Kubernetes community will be interested in moving forward, so ensuring there is a public place to track updates and gather feedback would be encouraged from my point of view. @timoreimann @andrewsykim

from digitalocean-cloud-controller-manager.

@normanjoyner sorry for the delay. I agree that the feature request raised by this issue has been implemented, so we can close it.

The way it stands right now, CCM will likely not be involved in the implementation of a generic system to accept and persist user-provided labels as it makes more sense to build a dedicated component for it. And such, I am somewhat hesitant to track any progress in this repository which could create false expectations (the predominant one that CCM is the component users would have to wait for or could try to contribute to it in order to implement the feature).

I do understand the value in having a two-way communication channel though. For this purpose, DO runs https://ideas.digitalocean.com/ to accept ideas / feature requests and stay in touch with the community. Would you be fine with submitting a ticket / an idea over there and use it to solicit periodic updates / leave feedback?

Appreciate your and everybody else's feedback towards what we have built so far and what lies ahead. Thanks!

from digitalocean-cloud-controller-manager.

(Closing, but I'll continue to watch for post updates on this issue to wrap up the discussion.)

from digitalocean-cloud-controller-manager.

Thanks for the update @timoreimann ... I’ve created an idea for this if anyone else is interested in tracking / lending input.

from digitalocean-cloud-controller-manager.

Thank you @normanjoyner, appreciated.

from digitalocean-cloud-controller-manager.