Git Product home page Git Product logo

Comments (12)

fatih avatar fatih commented on June 16, 2024 2

@lucasecdb alright. So first of all, having a cluster with a single node is a bad idea for kubernetes. The reason this node is tainted because it's acting as a master. Please see this issue: kubernetes/kubernetes#65473

Because there are no other nodes, k8s is preventing to schedule anything here. You can manually remove the taint via:

kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-

And everything should start work. Next time, make sure to understand these concepts and to use describe and get to get an understanding of the healthiness of your kubernetes cluster. Thanks!

from csi-digitalocean.

lucasecdb avatar lucasecdb commented on June 16, 2024 2

@fatih Thanks for taking the time to help me out!

from csi-digitalocean.

fatih avatar fatih commented on June 16, 2024

Thanks @lucasecdb

Can you please provide the following logs:

kubectl logs -l app=csi-provisioner-doplugin -c digitalocean-csi-plugin -n kube-system  
kubectl logs -l app=csi-attacher-doplugin -c digitalocean-csi-plugin -n kube-system
kubectl logs -l app=csi-doplugin -c csi-doplugin -n kube-system

Also, you didn't provide the full manifest of your setup. I wrote it here: https://github.com/digitalocean/csi-digitalocean/blob/master/.github/ISSUE_TEMPLATE.md but seems like it was neglected. So who uses this PV? If it's a deployment you should also provide it or you should provide something that is similar and still doesn't work so I can apply it to my cluster and see what's going on.

Thanks

from csi-digitalocean.

lucasecdb avatar lucasecdb commented on June 16, 2024

@fatih sorry about the manifests, I thought that only the pvc was sufficient, my full setup looks like this.

The image my deployment is using isn't exactly the postgres as in the gist, but a custom one that's on a private registry (GCR), but it's basically the same, diffing only in the DB schema that's contained in the image.

I can't access my droplet right now, but when I get home it'll be the first thing I'll do.

from csi-digitalocean.

lucasecdb avatar lucasecdb commented on June 16, 2024

@fatih sorry that it took so long to get back, but the logs didn't return anything. I'm pretty sure I ran the correct command to install the csi (and even updated it now, with v0.1.5) and still it doesn't seem to have anything in the logs.

from csi-digitalocean.

fatih avatar fatih commented on June 16, 2024

@lucasecdb that's interesting. What's:

kubectl get pods -n kube-system

gives you ?

from csi-digitalocean.

lucasecdb avatar lucasecdb commented on June 16, 2024

@fatih here's the output

NAME                              READY     STATUS    RESTARTS   AGE
calico-node-z8bct                 2/2       Running   2          3d
coredns-78fcdf6894-2kdml          1/1       Running   1          3d
coredns-78fcdf6894-kcg2m          1/1       Running   1          3d
csi-attacher-doplugin-0           0/2       Pending   0          3d
csi-provisioner-doplugin-0        0/2       Pending   0          3d
etcd-cramkle                      1/1       Running   1          3d
kube-apiserver-cramkle            1/1       Running   1          3d
kube-controller-manager-cramkle   1/1       Running   1          3d
kube-proxy-86jzz                  1/1       Running   1          3d
kube-scheduler-cramkle            1/1       Running   1          3d

from csi-digitalocean.

lucasecdb avatar lucasecdb commented on June 16, 2024

And here are the logs of both pods

$ kubectl logs csi-attacher-doplugin-0 -n kube-system
Error from server (BadRequest): a container name must be specified for pod csi-attacher-doplugin-0, choose one of: [csi-attacher digitalocean-csi-plugin]
$ kubectl logs csi-provisioner-doplugin-0 -n kube-system
Error from server (BadRequest): a container name must be specified for pod csi-provisioner-doplugin-0, choose one of: [csi-provisioner digitalocean-csi-plugin]

from csi-digitalocean.

fatih avatar fatih commented on June 16, 2024

@lucasecdb great, seems like the plugin is not running in your cluster. Which is why it prevent you to create volumes. Can you give me the following information as well:

kubectl describe pods csi-attacher-doplugin-0  -n kube-system
kubectl describe pods csi-provisioner-doplugin-0  -n kube-system

Also please try to give me this information

kubectl cluster-info dump > kubernetes-dump.log

and upload the file to somewhere I can read, such as https://gist.github.com/

from csi-digitalocean.

lucasecdb avatar lucasecdb commented on June 16, 2024

I'm getting an error when trying to dump the cluster info, I'll search for a fix. Here's the output for the first 2 commands:

$ kubectl describe pods csi-attacher-doplugin-0  -n kube-system
Name:               csi-attacher-doplugin-0
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=csi-attacher-doplugin
                    controller-revision-hash=csi-attacher-doplugin-5f6bb64659
                    statefulset.kubernetes.io/pod-name=csi-attacher-doplugin-0
Annotations:        <none>
Status:             Pending
IP:
Controlled By:      StatefulSet/csi-attacher-doplugin
Containers:
  csi-attacher:
    Image:      quay.io/k8scsi/csi-attacher:v0.2.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from csi-attacher-token-ljwps (ro)
  digitalocean-csi-plugin:
    Image:      digitalocean/do-csi-plugin:v0.1.4
    Port:       <none>
    Host Port:  <none>
    Args:
      --endpoint=$(CSI_ENDPOINT)
      --token=$(DIGITALOCEAN_ACCESS_TOKEN)
      --url=$(DIGITALOCEAN_API_URL)
    Environment:
      CSI_ENDPOINT:               unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      DIGITALOCEAN_API_URL:       https://api.digitalocean.com/
      DIGITALOCEAN_ACCESS_TOKEN:  <set to the key 'access-token' in secret 'digitalocean'>  Optional: false
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from csi-attacher-token-ljwps (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  socket-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  csi-attacher-token-ljwps:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  csi-attacher-token-ljwps
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  3m (x16906 over 1d)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
$ kubectl describe pods csi-provisioner-doplugin-0  -n kube-system
Name:               csi-provisioner-doplugin-0
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=csi-provisioner-doplugin
                    controller-revision-hash=csi-provisioner-doplugin-6d6b94d986
                    statefulset.kubernetes.io/pod-name=csi-provisioner-doplugin-0
Annotations:        <none>
Status:             Pending
IP:
Controlled By:      StatefulSet/csi-provisioner-doplugin
Containers:
  csi-provisioner:
    Image:      quay.io/k8scsi/csi-provisioner:v0.2.0
    Port:       <none>
    Host Port:  <none>
    Args:
      --provisioner=com.digitalocean.csi.dobs
      --csi-address=$(ADDRESS)
      --v=5
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from csi-provisioner-token-p7gcp (ro)
  digitalocean-csi-plugin:
    Image:      digitalocean/do-csi-plugin:v0.1.4
    Port:       <none>
    Host Port:  <none>
    Args:
      --endpoint=$(CSI_ENDPOINT)
      --token=$(DIGITALOCEAN_ACCESS_TOKEN)
      --url=$(DIGITALOCEAN_API_URL)
    Environment:
      CSI_ENDPOINT:               unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      DIGITALOCEAN_API_URL:       https://api.digitalocean.com/
      DIGITALOCEAN_ACCESS_TOKEN:  <set to the key 'access-token' in secret 'digitalocean'>  Optional: false
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from csi-provisioner-token-p7gcp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  socket-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  csi-provisioner-token-p7gcp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  csi-provisioner-token-p7gcp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  3m (x16906 over 1d)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.

from csi-digitalocean.

fatih avatar fatih commented on June 16, 2024

@lucasecdb seems like Kubernetes is not able to schedule any pods to your nodes. Did you somehow taint them (make them unschedulable) ? Because that's what happening here. Can you run:

kubectl get nodes -o wide

You should see the node maybe is not ready. Do a describe to the nodes to see what is happening to them. The issue here is that you couldn't deploy the plugin correctly.

from csi-digitalocean.

lucasecdb avatar lucasecdb commented on June 16, 2024

I don't think I did that, I'm still learning kubernetes and this is actually my first try at deploying an app.

NAME      STATUS    ROLES     AGE       VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
cramkle   Ready     master    3d        v1.11.2   142.93.205.191   <none>        Ubuntu 16.04.5 LTS   4.4.0-134-generic   docker://18.6.1

The first lines in the output of the describe:

Name:               cramkle
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=cramkle
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket=/var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl=0
                    projectcalico.org/IPv4Address=142.93.205.191/20
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Sun, 26 Aug 2018 21:01:00 +0000
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Thu, 30 Aug 2018 10:45:46 +0000   Sun, 26 Aug 2018 21:01:00 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Thu, 30 Aug 2018 10:45:46 +0000   Sun, 26 Aug 2018 21:01:00 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 30 Aug 2018 10:45:46 +0000   Sun, 26 Aug 2018 21:01:00 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 30 Aug 2018 10:45:46 +0000   Sun, 26 Aug 2018 21:01:00 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 30 Aug 2018 10:45:46 +0000   Sun, 26 Aug 2018 21:05:00 +0000   KubeletReady                 kubelet is posting ready status. AppArmor enabled

from csi-digitalocean.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.