Controller manual mode about digitalocean-cloud-controller-manager HOT 17 CLOSED

Perhaps this is a problem something similar to this #454 (comment) but I'm not sure

from digitalocean-cloud-controller-manager.

There is one more idea. If the name of the load balancer is specified, then work is done directly with it. At the moment, if a new service is created, it overwrites the data. I don't think this is the correct way for the controller to work.

from digitalocean-cloud-controller-manager.

The "one LB per Service" mapping is how CCM is designed upstream -- to my understanding, we do not and cannot change that approach without effectively diverging from the core upstream logic. Essentially, each CCM implementation is a set of hooks that follow a predefined workflow.

Your request isn't uncommon though. One recommendation to achieve that could be to use Ingress and demultiplex requests at the proxy level.

I'm not sure I understand your point about the LB name: it's another property of an LB that can be defined at any time through an annotation and causes the current LB configuration to be adjusted accordingly. I don't see why that would be out of line of how controllers should work -- let me know though if you feel like I'm missing something.

from digitalocean-cloud-controller-manager.

Let's discuss. We have a Load Balancer called k8s-loadbalancer on which some settings are already hung. I want to connect a service to a specific loadbalacer by specifying the loadbalacer name while doing so so that the settings are saved.
I do it like this:

service:
   annotations:
     service.beta.kubernetes.io/do-loadbalancer-name: k8s-loadbalancer
   type: LoadBalancer
   port: 3000

Expected behavior - the created loadbalacer will not be deleted, but will be modified. The entry_port & target_port are taken from the k8s services and these values are added to the loadbalancer digital ocean using the API

from digitalocean-cloud-controller-manager.

@feelan03 so you are saying that in your scenario, you see an existing load balancer getting deleted when you specify the name annotation? That should certainly not happen, and instead what you described should be carried out (existing load balancer should be modified/renamed).

Could you provide a series of reproduction steps to confirm the behavior and allow analyzing the issue further?

from digitalocean-cloud-controller-manager.

A helm chart victoria metrics was launched with this service block

service:
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-name: k8s-loadbalancer
    service.beta.kubernetes.io/do-loadbalancer-hostname: "grafana.do.domain"
  type: LoadBalancer
  port: 3000

loadbalancer k8s-loadbalancer created
Next, victoria metrics insert was launched with such a block of settings

  vminsert:
    replicaCount: 1
    subMetadata:
      labels:
        owner: infra
    affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              match Expressions:
              - key: "app.kubernetes.io/name"
                operator: In
                values:
                - "vmincert"
            topologyKey: "kubernetes.io/hostname"
    serviceSpec:
      metadata:
        annotations:
          service.beta.kubernetes.io/do-loadbalancer-name: k8s-loadbalancer
      spec:
        type: LoadBalancer

After that, another loadbalancer was created with the name k8s-loadbalancer, but they now have different ports

Here is the cloud-controller log

from digitalocean-cloud-controller-manager.

@feelan03 the likely reason the that VictoraMetrics controller(?) is creating a new LB is that the service annotation referencing the LB ID is missing within the serviceSpec so CCM has no way to associate the LB unambiguously. See also the docs we have in this regard.

from digitalocean-cloud-controller-manager.

@timoreimann It's very strange anyway. If you look at the yaml for this service, you can see the difference kubernetes.digitalocean.com/load-balancer-id. How could this happen, why didn't he specify an ID that already exists. Don't you think this is strange? Let me know if I'm doing something wrong, maybe I'm missing something.

from digitalocean-cloud-controller-manager.

But what if you try to force the id load balancer. maybe then he can earn?

from digitalocean-cloud-controller-manager.

The load balancer ID is only added to the Service object after the LB create request has returned (i.e., when a Service object of type LoadBalancer is reconciled by CCM for the first time). If the ID is missing (and CCM cannot derive it from the default, auto-generated LB name), then CCM will assume that a new LB must be created.

In your VictoriaMetrics manifest, you haven't provided the LB ID annotation so CCM cannot relate it to the existing LB. I think that if you include and copy the annotation from an existing Service / LB, then it could work. (No guarantees though since I don't know how that controller works in detail.)

from digitalocean-cloud-controller-manager.

Specifically, I don't know if the controller would possibly overwrite the Service object on each reconcile, which could lead to the LB ID annotation being removed again unless it was smart about retaining annotations it does not own (which is best practice for any controller).

from digitalocean-cloud-controller-manager.

Strange, I specified the name Load Balancer and type Load Balancer. For me, it's still surprising that CCM did not find this LB. I will definitely test the manifesto with the settings from the finished LB and the ladies feedback. Thank you.

from digitalocean-cloud-controller-manager.

I just confirmed in the code that, without an LB ID annotation available, CCM should still try to discover a pre-existing LB by looking up the custom and default (auto-generated) name. Given that and the fact that you specified the custom name annotation on the VictoriaMetrics manifest, I'd have expected an existing LB to be found.

I am more and more thinking that it's somehow related to the VM controller behavior. Might be worth for you to look up the actual implementation (and perhaps any logs) to see what it does exactly.

from digitalocean-cloud-controller-manager.

Perhaps, but judging by the screenshot of the log from the cloud controller, he actually found this load balancer, but could not do anything with it...

from digitalocean-cloud-controller-manager.

Sorry I don't speak what appears to be Russian. 🙂

from digitalocean-cloud-controller-manager.

I'm sorry, I forgot to translate into English

from digitalocean-cloud-controller-manager.

Now I'm seeing different behavior.
I removed absolutely all loadbalancer in DO. Removed all services that used them. And I decided to do it all over again, so that there would be no problems with the CCM.

1. I installed ingress-controller
   helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace=ingress-nginx
   When installing ingress-controler it created the loadbalancer itself
2. I copied the ID of the created loadbalancer to myself. Took it with k edit svc ingress-nginx-controller -n ingress-nginx -o yaml
3. I edited the values ​​of helm chart grafana. This is how he looks now

service:
  annotations:
    kubernetes.digitalocean.com/load-balancer-id: f7aaf1a4-6679-49ca-aa1a-c80708812d5c
    service.beta.kubernetes.io/do-loadbalancer-hostname: "grafana.do.domain"
  type: LoadBalancer
  port: 3000

4. Launched helm chart and started observing the loadbalancer settings. Action expected: adding port 3000 to the created load balancer. Reality: Deleted settings 80:30194/TCP,443:30490/TCP and created setting
   3000:30334/TCP

Could this be related to digital-ocean-manager being in separate namespaces?

from digitalocean-cloud-controller-manager.