Error fetching by providerID about digitalocean-cloud-controller-manager HOT 13 CLOSED

Hey @jaxxstorm it seems like node.Spec.ProviderID on your nodes is empty. Are your kubelets running with --cloud-provider=external?

from digitalocean-cloud-controller-manager.

They are indeed:

core@rancher-1 ~ $ docker inspect 182ef91887a6 | jq '.[].Args'
[
  "kubelet",
  "--cluster-domain=cluster.local",
  "--cluster-dns=10.43.0.10",
  "--root-dir=/opt/rke/var/lib/kubelet",
  "--cgroups-per-qos=True",
  "--cni-conf-dir=/etc/cni/net.d",
  "--cni-bin-dir=/opt/cni/bin",
  "--resolv-conf=/etc/resolv.conf",
  "--client-ca-file=/etc/kubernetes/ssl/kube-ca.pem",
  "--feature-gates=MountPropagation=false",
  "--cadvisor-port=0",
  "--read-only-port=0",
  "--network-plugin=cni",
  "--hostname-override=rancher-1.briggs.io",
  "--fail-swap-on=false",
  "--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
  "--v=2",
  "--address=0.0.0.0",
  "--pod-infra-container-image=rancher/pause-amd64:3.1",
  "--volume-plugin-dir=/var/lib/kubelet/volumeplugins",
  "--cloud-provider=external",
  "--kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml",
  "--enforce-node-allocatable=",
  "--allow-privileged=true",
  "--anonymous-auth=false"
]

It seems the cloud provider taint has been added

kubectl get nodes -o json | jq '.items[].spec.taints'
[
  {
    "effect": "NoSchedule",
    "key": "node.cloudprovider.kubernetes.io/uninitialized",
    "value": "true"
  }
]
[
  {
    "effect": "NoSchedule",
    "key": "node.cloudprovider.kubernetes.io/uninitialized",
    "value": "true"
  }
]
[
  {
    "effect": "NoSchedule",
    "key": "node.cloudprovider.kubernetes.io/uninitialized",
    "value": "true"
  }
]

but the ProviderId hasn't been set

from digitalocean-cloud-controller-manager.

Yeah, it seems like your nodes are being initialized by the cloud provider, otherwise, those taints would have been removed. Odd that we're still seeing this though

Successfully initialized node rancher-1.briggs.io with cloud provider

from digitalocean-cloud-controller-manager.

Can you show the entire log output for the cloud controller manager?

from digitalocean-cloud-controller-manager.

sure thing, here you go:

https://gist.github.com/jaxxstorm/8271b28c1bb060e2aac20593eec0f5d4

from digitalocean-cloud-controller-manager.

I see this related issue, but I don't believe this should be happening as I'm using 1.10 kubernetes/kubernetes#49836

I wonder if this is a rancher bug

from digitalocean-cloud-controller-manager.

E0531 14:26:42.918291       1 node_controller.go:321] failed to set node provider id: failed to get instance ID from cloud provider: instance not found

seems to be the problem.

One of the documented requirements is that your node names match the droplet name (or their IPv4 or IPv6 addresses) - see https://github.com/digitalocean/digitalocean-cloud-controller-manager#kubernetes-node-names-must-match-the-droplet-name-private-ipv4-ip-or-public-ipv4-ip. Otherwise cloud controller manager has no way of discovery your node.

Can you confirm that this is the case?

from digitalocean-cloud-controller-manager.

Yep, the nodes names and the droplet names match, I checked that.

from digitalocean-cloud-controller-manager.

For some reason it seems like your node can't be discovered from the API. Is the token you're using in the CCM owned by the same account running those droplets by any chance?

from digitalocean-cloud-controller-manager.

Yep, I even regenerated the token to make sure.

I really don't understand what's going on. I've tried:

• Ensure the cloudprovider=external is set correctly on the kubelet
• Having cloudprovider= and also cloudprovider=external on the API and controller manager

Whatever happens, the providerid is never specified. It doesn't make sense, and I can't find anything that might indicate an issue :(

from digitalocean-cloud-controller-manager.

Are you able to share output for

doctl compute droplet list

and

kubectl get node -o yaml

from digitalocean-cloud-controller-manager.

I don't believe it. There was a typo in my droplet name. I'm so sorry for wasting your time, thanks for your help

from digitalocean-cloud-controller-manager.

Not a problem! Glad it's working!

from digitalocean-cloud-controller-manager.