Git Product home page Git Product logo

ibm-vpc-block-csi-driver's Introduction

ibm-vpc-block-csi-driver

Build Status Coverage Status

IBM VPC Block Container Storage Interface (CSI) Driver provides a CSI interface used by Container Orchestrators to manage the lifecycle of IBM VPC Block Data volumes.

Supported orchestration platforms

The following are the supported orchestration platforms suitable for deployment for IBM VPC Block CSI Driver.

Orchestration platform Version Architecture
Red Hat® OpenShift® 4.7 x86
Red Hat® OpenShift® 4.8 x86
Red Hat® OpenShift® 4.9 x86
Kubernetes 1.19 x86
Kubernetes 1.20 x86
Kubernetes 1.21 x86

Prerequisites

Following are the prerequisites to use the IBM VPC Block CSI Driver:

  1. User should have either Red Hat® OpenShift® or kubernetes cluster on IBM VPC Gen 2 infrastructure.
  2. Should have compatible orchestration platform.
  3. Install and configure ibmcloud is CLI or get the required worker/node details by using IBM Cloud Console
  4. Cluster's worker node should have following labels, if not please apply labels before deploying IBM VPC Block CSI Driver.
"ibm-cloud.kubernetes.io/worker-id"
"failure-domain.beta.kubernetes.io/region"
"failure-domain.beta.kubernetes.io/zone"
"topology.kubernetes.io/region"
"topology.kubernetes.io/zone"

Apply worker labels

Please use apply-required-setup.sh script for all the nodes in the cluster which will need couple of inputs like 

instanceID:  That you can get from ibmcloud is ins 

node-name: this is as per node name in the kubernetes node check by using kubectl get nodes

region-of-instanceID:  region of the instanceID, this you can get the by using ibmcloud is in <instanceID>

zone-of-instanceID: Zone of the instanceID, this you can get the by using ibmcloud is in <instanceID>

Example :- ./apply-required-setup.sh

Build the driver

For building the driver docker and GO should be installed on the system

  1. On your local machine, install docker and Go.

  2. GO version should be >=1.16

  3. Set the GOPATH environment variable.

  4. Build the driver image

    Clone the repo or your forked repo

    $ mkdir -p $GOPATH/src/github.com/kubernetes-sigs
    $ cd $GOPATH/src/github.com/kubernetes-sigs/
    $ git clone https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver.git
    $ cd ibm-vpc-block-csi-driver
    

    Build project and runs testcases

    $ make
    

    Build container image for the driver

    $ make buildimage
    

    Push image to registry

    Image should be pushed to any registry from which cluster worker nodes have access to pull

    You can push the driver image to docker.io registry or IBM public registry under your namespace.

    For pushing to IBM registry:

    Create an image pull secret in your cluster

    1. ibmcloud login to the target region

    2. Run - ibmcloud cr region-set global

    3. Run - ibmcloud cr login

    4. Make sure kubectl is configured to use the cluster

    5. Review and retrieve the following values for your image pull secret.

      <docker-username> - Enter the string: iamapikey.

      <docker-password> - Enter your IAM API key. For more information about IAM API keys, see Understanding API keys .

      <docker-email> - Enter the string: iamapikey.

    6. Run the following command to create the image pull secret in your cluster. Note that your secret must be named icr-io-secret

      
       kubectl create secret docker-registry icr-io-secret --docker-server=icr.io --docker-username=iamapikey --docker-password=-<iam-api-key> --docker-email=iamapikey -n kube-system
      
      

Deploy CSI driver on your cluster

IBM VPC endpoints which supports Gen2 is documented here

  • Install kustomize tool. The instructions are available here
  • Export cluster config i.e configuring kubectl command
  • Deploy IBM VPC Block CSI Driver on your cluster
    • You can use any overlays available under deploy/kubernetes/driver/kubernetes/overlays/ and edit the image tag if you want to use your own build image from this source code, although default overlays are already using released IBM VPC Block CSI Driver image

    • gcr.io/k8s-staging-cloud-provider-ibm/ibm-vpc-block-csi-driver:master image is always the latest image build using master branch code.

    • Example using stage overlay to update the image tag

      • Change iks-vpc-block-driver image name in deploy/kubernetes/driver/kubernetes/overlays/stage/controller-server-images.yaml
      • Change iks-vpc-block-driver image name in deploy/kubernetes/driver/kubernetes/overlays/stage/node-server-images.yaml
    • Deploy plugin

      • bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage

Testing

  • Create storage classes
    • ls deploy/kubernetes/storageclass/ | xargs -I classfile kubectl apply -f deploy/kubernetes/storageclass/classfile
  • Create PVC
    • kubectl create -f examples/kubernetes/validPVC.yaml
  • Create POD with volume
    • kubectl create -f examples/kubernetes/validPOD.yaml

Delete CSI driver from your cluster

  • Delete plugin
    • bash deploy/kubernetes/driver/kubernetes/delete-vpc-csi-driver.sh stage

E2E Tests

Please refer this repository for e2e tests.

How to contribute

If you have any questions or issues you can create a new issue here .

Pull requests are very welcome! Make sure your patches are well tested. Ideally create a topic branch for every separate change you make. For example:

  1. Fork the repo

  2. Create your feature branch (git checkout -b my-new-feature)

  3. Commit your changes (git commit -am 'Added some feature')

  4. Push to the branch (git push origin my-new-feature)

  5. Create new Pull Request

  6. Add the test results in the PR

Vendor changes

For any changes to go.mod or go.sum, be sure to run go mod vendor to update dependencies in the vendor/ directory. You can verify that the vendor directory is up-to-date before filing a PR by running hack/verify-vendor.sh.

ibm-vpc-block-csi-driver's People

Contributors

ambiknai avatar arahamad avatar arahamad-zz avatar chillandimprove avatar dobsonj avatar gunakkibm avatar jsafrane avatar k8s-ci-robot avatar meghnasiingh avatar prankulmahajan avatar romanbednar avatar sameshai avatar shipra101 avatar stevemar avatar xmudrii avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

ibm-vpc-block-csi-driver's Issues

node staging fails for block volumes

Volume resizing fails for block volumes:

Jan 23 16:38:19.572: INFO: At 2023-01-23 16:28:34 +0000 UTC - event for inline-volume-tester-tw978: {kubelet ci-op-p95ft5gl-3565d-mbqkm-worker-2-xhdlf} VolumeResizeFailed: NodeExpandVolume.NodeExpandVolume failed for volume "pvc-43193e9e-e642-48cd-9088-7ee4c57eb081" : Expander.NodeExpand failed to expand the volume : rpc error: code = Internal desc = {RequestID: 8ca47df4-4c62-4bdc-a58f-5b127e628a8b , Code: FileSystemResizeFailed, Description: Failed to resize the file system, BackendError: failed to read size of device devtmpfs: exit status 1: blockdev: cannot open devtmpfs: No such file or directory, Action: Please check if there is any error in PVC describe related with volume resize}

The root cause is that NodeStageVolume gRPC does not check if a volume is block device. So if the volume is a block device it evaluates volumePath incorrectly. In the event shown above it's set to devtmpfs which is not a real path to device. When this value is passed to resizer (NeedResize function specifically) which tries to find the device so it can compare block and fs sizes - it fails with No such file or directory.

This is currently causing our Openshift CI to fail when rebasing this driver downstream: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_ibm-vpc-block-csi-driver/30/pull-ci-openshift-ibm-vpc-block-csi-driver-master-e2e-ibmcloud-csi/1617543897712431104

Failed to create IBM Cloud VPC Block CSI Driver on Single Node OpenShift in IBM Cloud VPC

Environment details:
Single Node OpenShift cluster running on IBM Cloud VPC VSI.
OpenShift: 4.12

Problem Description:
I tried to follow the instructions here, and built the image from the master branch and pushed it to IBM Cloud Container Registry. However this command failed

bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage

with this issue

This will install 'stage' version of vpc csi driver!
darwin22
Error: invalid Kustomization: json: cannot unmarshal string into Go struct field Kustomization.patches of type types.Patch
error: no objects passed to apply

I contacted @ambiknai who was very helpful and very generous with her time and troubleshooted the issue with me, and gave me manual yaml files which progressed the CSI driver creation, but still it's failing with the below issue.

Attached are the logs.
csi-provisioner.log
ibm-vpc-block-csi-controller.log

Format and mount fail issue

We are seeing an intermittent issue where nodeStage request is failing with "exit 32, already staged" error.

Re-enable Sanity Test

Sanity test was disabled as part of this PR: #100
We have to enable it again and check if everything runs fine.

Resize filesystem of a larger volume created from a snapshot/volume

Upstream issue introduces improvements to volume resizing. The change added new resize code and moved to kubernetes/mount-utils.

This change should be reflected in all CSI drivers. New module should be imported and it's resize logic added.

Dependency should be updated to use the current k8s/mount-utils module and CSI driver should use the resize code it provides in NodeStageVolume to perform the resize if needed. Example of similar change can be found here

Issue originated in BZ1877261

How to reproduce:

  1. create original pod and pvc
  2. create snapshot from original pvc
  3. create pvc from the snapshot specifying larger size than the original
  4. create pod to consume the new pvc
  5. observe fs size of the new volume inside the pod does not match the requested size

IBM VPC block csi driver crashes when a PVC is resized from 100 GB to 5000 GB

Steps To Reproduce:

  1. Create a PVC using below spec.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: ibmc-vpc-block-5iops-tier
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  1. Edit PVC to increase its size to 5000GB.

Environment:
VPC Gen2 RKOS 4.10.9_1515 cluster in air-gapped environment.

ibmcloud ks cluster addon ls --cluster vinayak-3
OK
Name                   Version   Health State   Health Status
vpc-block-csi-driver   4.3       normal         Addon Ready. For more info: http://ibm.biz/addon-state (H1500)

Expected Behaviour:

  1. CSI driver should not crash/panic.
  2. If CSI driver crashes, it should recover from crash after restart. Should not get stuck in crashLoopBackOff.
  3. If volume resize from 100 GB to 5000 GB is not supported then, in kubectl describe pvc <pvc-name> appropriate reason should be updated.

Actual Behaviour:

  1. CSI driver crashes with panic: runtime error: invalid memory address or nil pointer dereference error. (See attached iks-vpc-block-driver.log)
  2. CSI driver continusly crashes even after restart. It gets stuck in crashLoopBackOff. In below output, it has restarted 12 times in last 26 mins.
kubectl get pods -nkube-system | grep ibm-vpc-block-csi-controller-0
ibm-vpc-block-csi-controller-0              6/6     Running            12 (62s ago)   26m
  1. PVC resize fails not with error that 100 GB to 5000 GB is not supported but with resize volume "pvc-f690801f-8095-4f0a-a679-6adfedd1d2a2" by resizer "vpc.block.csi.ibm.io" failed: rpc error: code = Unavailable desc = transport is closing error.
kd pvc my-pvc
Name:          my-pvc
Namespace:     default
StorageClass:  ibmc-vpc-block-5iops-tier
Status:        Bound
Volume:        pvc-f690801f-8095-4f0a-a679-6adfedd1d2a2
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
               volume.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      100Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       <none>
Conditions:
  Type       Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----       ------  -----------------                 ------------------                ------  -------
  Resizing   True    Mon, 01 Jan 0001 00:00:00 +0000   Tue, 31 May 2022 08:04:36 +0000
Events:
  Type     Reason                 Age                From                                                                                      Message
  ----     ------                 ----               ----                                                                                      -------
  Normal   Provisioning           13m                vpc.block.csi.ibm.io_ibm-vpc-block-csi-controller-0_b50d217e-c8a5-4e7b-a5af-f000bd650c19  External provisioner is provisioning volume for claim "default/my-pvc"
  Normal   ExternalProvisioning   12m (x6 over 13m)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "vpc.block.csi.ibm.io" or manually created by system administrator
  Normal   ProvisioningSucceeded  12m                vpc.block.csi.ibm.io_ibm-vpc-block-csi-controller-0_b50d217e-c8a5-4e7b-a5af-f000bd650c19  Successfully provisioned volume pvc-f690801f-8095-4f0a-a679-6adfedd1d2a2
  Warning  ExternalExpanding      12m                volume_expand                                                                             Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   Resizing               12m                external-resizer vpc.block.csi.ibm.io                                                     External resizer is resizing volume pvc-f690801f-8095-4f0a-a679-6adfedd1d2a2
  Warning  VolumeResizeFailed     4m52s              external-resizer vpc.block.csi.ibm.io                                                     resize volume "pvc-f690801f-8095-4f0a-a679-6adfedd1d2a2" by resizer "vpc.block.csi.ibm.io" failed: rpc error: code = Unavailable desc = transport is closing
  Normal   Resizing               4m47s              external-resizer vpc.block.csi.ibm.io                                                     External resizer is resizing volume pvc-f690801f-8095-4f0a-a679-6adfedd1d2a2

csi-resizer.log
iks-vpc-block-driver.log

cc: @arahamad

The deploy-csi-driver-on-your-cluster section in Readme.md need be updated.

When I follow https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/blob/master/README.md#deploy-csi-driver-on-your-cluster to install ibm-vpc-block-csi-driver to my IKS cluster.

I meet follow error:

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
Error: invalid Kustomization: json: cannot unmarshal string into Go struct field Kustomization.patches of type types.Patch
error: no objects passed to apply
root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver#

I tried to fix the issue by replace the patches: to patchesStrategicMerge: in deploy/kubernetes/driver/kubernetes/overlays/stage/kustomization.yaml

The error message is changed to:

bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Error: no matches for Id StatefulSet.v1.apps/ibm-vpc-block-csi-controller.[noNs]; failed to find unique target for patch StatefulSet.v1.apps/ibm-vpc-block-csi-controller.[noNs]
error: no objects passed to apply

I tried to fix the new issue by add namespace namespace: kube-system to controller-server-images.yaml and node-server-images.yaml under deploy/kubernetes/driver/kubernetes/overlays/stage the new error looks like:

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
Error: no matches for Id StatefulSet.v1.apps/ibm-vpc-block-csi-controller.kube-system; failed to find unique target for patch StatefulSet.v1.apps/ibm-vpc-block-csi-controller.kube-system
error: no objects passed to apply

I tried to fix the new issue by update the StatefulSet to Deployment of deploy/kubernetes/driver/kubernetes/overlays/stage/controller-server-images.yaml
The new error looks like:

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# bash deploy/kubernetes/driver/kubernetes/deploy-vpc-block-driver.sh stage
This will install 'stage' version of vpc csi driver!
linux-gnu
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
serviceaccount/ibm-vpc-block-controller-sa created
serviceaccount/ibm-vpc-block-node-sa created
clusterrole.rbac.authorization.k8s.io/vpc-block-driver-registrar-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-driver-snapshotter-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-external-attacher-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-external-resizer-role created
clusterrole.rbac.authorization.k8s.io/vpc-block-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-driver-registrar-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-driver-snapshotter-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-external-attacher-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-external-resizer-binding created
clusterrolebinding.rbac.authorization.k8s.io/vpc-block-provisioner-binding created
configmap/addon-vpc-block-csi-driver-configmap created
configmap/ibm-vpc-block-csi-configmap created
secret/storage-secret-store created
csidriver.storage.k8s.io/vpc.block.csi.ibm.io created
Error from server (BadRequest): error when creating "STDIN": Deployment in version "v1" cannot be handled as a Deployment: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'
Error from server (BadRequest): error when creating "STDIN": DaemonSet in version "v1" cannot be handled as a DaemonSet: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'

Try to find the built out yaml file, find that the resources section broken the deploy, there are many {{kube-system.addon-vpc-block-csi-driver-configmap.xxx}} need be replaced?

root@pp-liudali-amd64-node-0:~/go/src/github.com/kubernetes-sigs/ibm-vpc-block-csi-driver# kustomize build deploy/kubernetes/driver/kubernetes/overlays/stage/
...
        resources:
          limits:
            cpu: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPULimit}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPULimit}}120m{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPULimit}}'
            memory: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryLimit}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryLimit}}300Mi{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryLimit}}'
          requests:
            cpu: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryRequest}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryRequest}}30m{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverMemoryRequest}}'
            memory: '{{kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPURequest}}{{^kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPURequest}}75Mi{{/kube-system.addon-vpc-block-csi-driver-configmap.NodeDriverCPURequest}}'
...

I install kustomize by command

curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

the version is 'v5.0.3'

The go path

GOPATH=/root/go

proxy environment variables are ignored by ibm-vpc-block-csi-driver

When setting the HTTPS_PROXY, HTTP_PROXY, or NO_PROXY environment variables on the ibm-vpc-block-csi-driver, the driver ignores them and does not use the requested proxy.
This causes a failure during volume provisioning on OpenShift if the cluster-wide proxy is configured.
The difference, compared to AWS or Azure, is that the IBM driver doesn't use http.ProxyFromEnvironment when setting up the HTTP transport.

go.mod is missing replace for k8s.io/endpointslice

What happened:

Since this repo depends on k8s.io/kubernetes it is required to replace all modules in staging, even though they're not a real dependency of this repo. Discussion here: kubernetes/kubernetes#79384 (comment)

After rebasing to a new release v5.2.8 there are missing dependencies in go.mod and go.sum

and it is now causing go list to fail:

go list -mod=readonly -m all

go: k8s.io/[email protected]: invalid version: unknown revision v0.0.0

What you expected to happen:

go list should pass

How to reproduce it:

Run go list -mod=readonly -m all in master.

Anything else we need to know?:

This was found by OpenShift CI during rebase where we run dependency verification, I believe adding this line to go.mod should fix the issue:

k8s.io/endpointslice => k8s.io/endpointslice v0.28.6

Environment:

CSI Driver version: v5.2.8
Kubernetes version (use kubectl version): n/a
OS (e.g. from /etc/os-release): n/a
Kernel (e.g. uname -a): n/a
Install tools: n/a
Others: n/a

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.