openebs / mayastor-docs Goto Github PK

https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor#control-plane

Currently following command is to be used:

helm install mayastor mayastor/mayastor -n mayastor --create-namespace --set values.csi.node.kubeletDir="/var/snap/microk8s/common/var/lib/kubelet
But values.csi.node.kubeletDir is wrong, it should be csi.node.kubeletDir
i.e. the correct command is :

helm install mayastor mayastor/mayastor -n mayastor --create-namespace --set csi.node.kubeletDir="/var/snap/microk8s/common/var/lib/kubelet"

In the Mayastor docs on https://mayastor.gitbook.io/introduction/quickstart/configure-mayastor#what-is-a-mayastor-pool-msp we list the nvme addressing scheme.

The example given is: nvme://nqn.2014-08.com.vendor:nvme:nvm-subsystem-sn-d7843

The nvme tool reports in this syntax.

$ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     S462NF0MA01130H      Samsung SSD 970 PRO 1TB                  1         843.50  GB /   1.02  TB    512   B +  0 B   1B2QEXP7
/dev/nvme1n1     PHKS750500HT375AGN   INTEL SSDPED1K375GA                      1         375.08  GB / 375.08  GB    512   B +  0 B   E2010324

$ sudo nvme list-subsys
nvme-subsys0 - NQN=nqn.2014.08.org.nvmexpress:144d144dS462NF0MA01130H     Samsung SSD 970 PRO 1TB                 
\
 +- nvme0 pcie 0000:01:00.0 live 
nvme-subsys1 - NQN=nqn.2014.08.org.nvmexpress:80868086PHKS750500HT375AGN  INTEL SSDPED1K375GA                     
\
 +- nvme1 pcie 0000:04:00.0 live 

We should consider providing an example detailing how to build this syntax up from commonly available tools.

On "Deploy Mayastor" the very last command:

kubectl mayastor get nodes

should read

kubectl -n mayastor get nodes

It's very difficult for users to notice the single bdev limitation from https://mayastor.gitbook.io/introduction/quickstart/configure-mayastor#create-mayastor-pool-s.

Users have shown to be quite confused by this the error produced by using multiple disks, so we should place the notice with some level of prominence.

I tried to create mayastore storageclass using next example:

https://mayastor.gitbook.io/introduction/quickstart/configure-mayastor#create-mayastor-storageclass-s

but I´ve got:
Error from server (BadRequest): error when creating "STDIN": StorageClass in version "v1" cannot be handled as a StorageClass: v1.StorageClass.Parameters: ReadString: expects " or n, but found 6, error found in #10 byte of ...|Timeout":60,"protoco|..., bigger context ...|name":"mayastor-nvmf"},"parameters":{"ioTimeout":60,"protocol":"nvmf","repl":"1"},"provisioner":"io.|...

If I remove line 9, I can create it:

ioTimeout: 60

If users miss the bottom step of https://mayastor.gitbook.io/introduction/quickstart/preparing-the-cluster#label-mayastor-node-candidates before they start running https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor#create-mayastor-application-resources, they may see their kubectl -n mayastor get msp showing a blank under "state".

We can help mitigate this user frustration by giving them a breadcrumb reminding them about the labels before.

In https://mayastor.gitbook.io/introduction/quickstart/configure-mayastor#what-is-a-mayastor-pool-msp we list io_uring as an option, it is not. uring is!

Please update the resources available on you official webpage as the resources that are being set for the resources is hugely to low, not only for rest-deployment but also for other components. When working with such resources they like to fail on large workloads and this for sure gives people the bad feeling that Mayastor is incomplete.

Referring to: https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor
In site header: REST
Link: https://raw.githubusercontent.com/openebs/mayastor-control-plane/v1.0.4/deploy/rest-deployment.yaml

There you've got:

      containers:
        - name: rest
          resources:
            limits:
              cpu: 100m
              memory: 64Mi
            requests:
              cpu: 50m
              memory: 32Mi

In our case the minimum required setup for the REST Deployment (Pod) to get out of OOMKill loop was:

      containers:
        - name: rest
          resources:
            limits:
              cpu: 1500m
              memory: 2048Mi
            requests:
              cpu: 500m
              memory: 256Mi

And the above was only to get out of the Mayastor deployment stage.

Lab-Environment, Kubernetes is a bare metal kubespray deployment:

kubernetes: "v1.25.3"
nodes:
- name: node_group_1
  count: 4
  linux_distro: "Ubuntu 22.04"
  hardware:
    cpu_type: "Intel Xeon ICX"
    cpu_count: 1
    ram: "512GB"
    drives:
    - device: "nvme0"
      size: "349.32GB"
    - device "sda"
      size: "256GB"
- name: node_group_2
  count: 4
  linux_distro: "Ubuntu 22.04"
  hardware:
    cpu_type: "Intel Xeon ICX"
    cpu_count: 2
    ram: "1024GB"
    drives:
    - device: "nvme0"
      size: "1.46TB"
    - device: "nvme1"
      size: "1.46TB"
    - device: "nvme2"
      size: "1.46TB"
    - device: "nvme3"
      size: "1.46TB"
    - device: "sda"
      size: "447GB"

https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor and further pages

In several location it is said we should be waiting for an action to complete. It would be good to have a baseline time specified, e.g.

1. apply mayastor-daemonset.yaml
2. this should take a few seconds/minutes/up to half an hour etc.

When the expected time is exceeded, I would like to try and troubleshoot the issue, there is no explanation on how to do that - which logs and where to check, what to look for in those logs etc.

Greetings,

It seems for me personally this part of the docs put me in a very difficult situation to debug: https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor#etcd has no examples on a mayatstor deployment when etcd its already part of the k8s cluster, like in case of Talos.dev .

Would be ideal when you mention that mayastor needs etcd, and that users are responsible for installing it, what about where its installed already how do we link that to mayastor, without the need to have 2 etcd deployments, since if we apply the examples given by the docs, we will end up at least on Talos OS deployments with 2 different etcd deployments.

Best/ideal would be when mayastor can detect if etcd is present, or a switch to allow the deployment of mayastor in a ready etcd os or in an os where etcd lacks.

After deploying all that was to deploy, alto our cluster has etcd out of the box we get:

[2021-12-17T23:43:34.090193527+00:00 ERROR mayastor::persistent_store:persistent_store.rs:89] Failed to connect to etcd on endpoint mayastor-etcd:2379. Retrying...

Following commands will indeed deploy etcd yet like mentioned above, we already run etcd.

Would like to know how or what we can use when we have etcd in place. Seems by default this etcd is not known to mayastor, and neither is this much documented, or maybe I missed it out.

kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/etcd/statefulset.yaml 
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/etcd/svc.yaml
kubectl apply -f https://raw.githubusercontent.com/openebs/Mayastor/master/deploy/etcd/svc-headless.yaml

https://mayastor.gitbook.io/introduction/quickstart/deploy-mayastor#data-plane

https://raw.githubusercontent.com/openebs/mayastor/master/v1.0.2/mayastor-daemonset.yaml