Git Product home page Git Product logo

Comments (8)

psavva avatar psavva commented on September 27, 2024

Please find the Manifests as they are deployed here
manifests

from rook.

Madhu-1 avatar Madhu-1 commented on September 27, 2024

can you try ceph fs subvolume ls myfs csi? please provide pod describe output where you are seeing the problem

from rook.

psavva avatar psavva commented on September 27, 2024

can you try ceph fs subvolume ls myfs csi? please provide pod describe output where you are seeing the problem

[
    {
        "name": "csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0"
    }
]

from rook.

Madhu-1 avatar Madhu-1 commented on September 27, 2024

I see subvolume is created and PVC is also in bound state. please check pod describe output for the application pod

from rook.

psavva avatar psavva commented on September 27, 2024

I see subvolume is created and PVC is also in bound state. please check pod describe output for the application pod

Name:             csicephfs-demo-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             autoscaled-nbg1-cx41-5bdfc7261e366007/10.0.0.11
Start Time:       Tue, 12 Mar 2024 12:47:41 +0000
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Containers:
  web-server:
    Container ID:
    Image:          nginx
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/www/html from mypvc (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d4ngn (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  mypvc:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  cephfs-pvc
    ReadOnly:   false
  kube-api-access-d4ngn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age   From     Message
  ----     ------       ----  ----     -------
  Warning  FailedMount  60m   kubelet  MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.216.60:3300,10.43.232.125:3300,10.43.222.172:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-3223765604,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T12:50:23.052+0000 7f7599343140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy
  Warning  FailedMount  59m  kubelet  MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.216.60:3300,10.43.232.125:3300,10.43.222.172:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-710911240,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T12:51:27.170+0000 7f9cbb0f2140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy
  Warning  FailedMount  58m  kubelet  MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.216.60:3300,10.43.232.125:3300,10.43.222.172:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-2465381711,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T12:52:32.636+0000 7f421a09c140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy
  Warning  FailedMount  57m  kubelet  MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.216.60:3300,10.43.232.125:3300,10.43.222.172:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-2219036612,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T12:53:42.178+0000 7f9cf09fa140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy
  Warning  FailedMount  55m  kubelet  MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.232.125:3300,10.43.222.172:3300,10.43.216.60:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-1903664150,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T12:54:59.928+0000 7f3350a21140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy
  Warning  FailedMount  54m  kubelet  MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.232.125:3300,10.43.222.172:3300,10.43.216.60:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-2169577009,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T12:56:33.747+0000 7f0209197140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy
  Warning  FailedMount  52m  kubelet  MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.232.125:3300,10.43.222.172:3300,10.43.216.60:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-1752183312,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T12:58:39.859+0000 7fb4662ee140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy
  Warning  FailedMount  27s (x17 over 49m)  kubelet  (combined from similar events): MountVolume.MountDevice failed for volume "pvc-e41e26db-0558-4bcf-be26-55938af40f8f" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 10.43.232.125:3300,10.43.222.172:3300,10.43.216.60:3300:/volumes/csi/csi-vol-0046642c-420b-49ee-b6e3-138eca552bf0/e2d3cf80-0c1c-457b-bebb-7c995672a5ea /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/d4745aec34b74b951e98fe822bac1f46f6c78f06df7a7f9d5b567e942b037898/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-855622619,mds_namespace=myfs,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2024-03-12T13:50:25.361+0000 7f18273a8140 -1 failed for service _ceph-mon._tcp
mount error: no mds server is up or the cluster is laggy

from rook.

Madhu-1 avatar Madhu-1 commented on September 27, 2024

rook/deploy/examples/cluster.yaml

Lines 284 to 295 in b6b5628

csi:
readAffinity:
# Enable read affinity to enable clients to optimize reads from an OSD in the same topology.
# Enabling the read affinity may cause the OSDs to consume some extra memory.
# For more details see this doc:
# https://rook.io/docs/rook/latest/Storage-Configuration/Ceph-CSI/ceph-csi-drivers/#enable-read-affinity-for-rbd-volumes
enabled: false
# cephfs driver specific settings.
cephfs:
# Set CephFS Kernel mount options to use https://docs.ceph.com/en/latest/man/8/mount.ceph/#options.
# kernelMountOptions: "" 

csi:
  cephfs:
    kernelMountOptions: "ms_mode=prefer-crc"

add above entry to the ceph cluster CR and delete and recreate the application pod and see if it works?

from rook.

psavva avatar psavva commented on September 27, 2024

That solved the problem.

Thank you very much

from rook.

lucsoft avatar lucsoft commented on September 27, 2024

Hey thanks this also solved my problem? May i ask why this happens and maybe we should make this a default?

from rook.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.