Comments (15)
guits
commented on May 27, 2024
1
@sp98 when deploying OSDs with lvm batch method, if you pass a db device, ceph-volume will by default basically use all the available space on it and create as many LVs as you pass block devices. If later you want to add a new OSD, then the existing db device won't be available as there's no more free space on it.
from rook.
sp98
commented on May 27, 2024
1
slots seem like a good workaround! Thanks guits! @sp98 can we configure that from the cluster crd?
Reopened. I'll take a look if we can configure this from the Ceph CRD.
from rook.
barrettMCW
commented on May 27, 2024
1
you'll want a new issue for this @eyitayoafolabi
my issue was attempting to add drives to a metadata device that is already in use. there is no resolution for this issue barring recreating all the osds used by the metadata device and instantiating them all at the same time, or the only other option is using no metadata device and storing metadata on the new device itself.
from rook.
sp98
commented on May 27, 2024
@barrettMCW can you share the lsblk output of one of the nodes where nvme3n1 is available?
from rook.
barrettMCW
commented on May 27, 2024
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0
7:0 0 63.5M 1 loop /snap/core20/2015
...
loop4
7:4 0 40.4M 1 loop /snap/snapd/20671
sda 8:0 0 16.4T 0 disk
└─ceph--0018fad4--ec7c--472d--acee--5c6ad2305b48-osd--block--3b356982--71cd--4450--a117--8426d2174c4c
253:8 0 16.4T 0 lvm
sdb 8:16 0 18.2T 0 disk
└─ceph--6d11166f--ef0c--4d2c--a06e--94727c59a934-osd--block--3e706c20--898e--4976--8fd2--d904c8a6f753
253:0 0 18.2T 0 lvm
sdc 8:32 0 16.4T 0 disk
└─ceph--4d251ead--7e52--4301--b770--2ad6192fdb70-osd--block--fd0501c3--e32b--4ec2--b94e--7e6d354bfb2a
253:2 0 16.4T 0 lvm
sdd 8:48 0 18.2T 0 disk
└─ceph--292b55f8--0f23--4d7f--a88c--af915894a3a0-osd--block--1d3af60c--d238--4eea--9ec7--ad326ec8d696
253:4 0 18.2T 0 lvm
sde 8:64 0 16.4T 0 disk
└─ceph--3c0a7a64--fe23--4401--8b7e--72c13fbce90d-osd--block--e192c1ef--f391--431e--bb80--b4b2ae75168d
253:6 0 16.4T 0 lvm
sdf 8:80 0 18.2T 0 disk
sdg 8:96 0 18.2T 0 disk
nbd0 43:0 0 0B 0 disk
...
nbd7 43:224 0 0B 0 disk
rbd0 252:0 0 100G 0 disk /var/lib/kubelet/pods/344cb762-cec7-4b12-aa93-130575acadc0/volumes/kubernetes.io~csi/pvc-d8bbd221-9a47-4ac0-ab39-012eec5b1ecc/mount
/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi
.ceph.com/af5c0db78c316dd044c5e48533cd2549952b1acfb949c3c163
18574ef0d3a7e8/globalmount/0001-0009-rook-ceph-0000000000000
004-add81f26-0f00-44e5-9b41-5f814da49505
...
rbd3 252:48 0 100G 0 disk /var/lib/kubelet/pods/d65172ad-13b5-4e89-b3a3-198a2699c5db/volumes/kubernetes.io~csi/pvc-5bca7ff9-1aec-4cea-897c-1d2c93e9ebc4/mount
/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi
.ceph.com/96adff1e36231f8464fba9e78556711615a390cfea91936f6e
403f83312c640b/globalmount/0001-0009-rook-ceph-0000000000000
004-b2bd479e-98dc-4b43-8738-ccfc506564d6
nvme0n1
259:0 0 3.5T 0 disk
nvme1n1
259:1 0 3.5T 0 disk
nvme2n1
259:2 0 3.5T 0 disk
nvme3n1
│ 259:3 0 3.5T 0 disk
├─ceph--41226caa--c499--4330--b0d5--f3eaf04a5518-osd--db--9826873c--daec--40bd--97a6--79bc2a7bf184
│ 253:1 0 715.4G 0 lvm
├─ceph--41226caa--c499--4330--b0d5--f3eaf04a5518-osd--db--6dce529c--3be7--4195--8b12--ceea3ab42e7c
│ 253:3 0 715.4G 0 lvm
├─ceph--41226caa--c499--4330--b0d5--f3eaf04a5518-osd--db--22eec1a6--bd11--4673--a519--b33509702d40
│ 253:5 0 715.4G 0 lvm
├─ceph--41226caa--c499--4330--b0d5--f3eaf04a5518-osd--db--d799133e--7393--4a32--a146--2822a51ab7ca
│ 253:7 0 715.4G 0 lvm
└─ceph--41226caa--c499--4330--b0d5--f3eaf04a5518-osd--db--1c26b5eb--2560--4347--a56c--1d8ae6a250b1
253:9 0 715.4G 0 lvm
nvme4n1
│ 259:4 0 894.3G 0 disk
├─nvme4n1p1
│ 259:5 0 1G 0 part /boot/efi
├─nvme4n1p2
│ 259:6 0 64G 0 part [SWAP]
└─nvme4n1p3
259:7 0 829.2G 0 part /var/lib/kubelet/pods/90d3adb7-88a8-4c23-9fb1-a3beeb416966/volume-subpaths/omero-web-config/omero-web/0
/
nbd8 43:256 0 0B 0 disk
...
nbd15
43:480 0 0B 0 disk from rook.
sp98
commented on May 27, 2024
@barrettMCW Looks like its not supported based on this comment - #13221 (comment)
Need to confirm the reason on why its not supported.
from rook.
barrettMCW
commented on May 27, 2024
dang! that discussion slipped past my web search! thanks! if this is something you guys are looking to add i'm happy to help. otherwise I'd imagine that this is something i could do manually with dmsetup or lv then add those as the metadata device?
from rook.
sp98
commented on May 27, 2024
dang! that discussion slipped past my web search! thanks! if this is something you guys are looking to add i'm happy to help. otherwise I'd imagine that this is something i could do manually with dmsetup or lv then add those as the metadata device?
logical volumes should work.
from rook.
barrettMCW
commented on May 27, 2024
gotcha, i'll do work on that. any word on reasons for not supporting? beyond that we should be good to close.
Thanks Santosh!
from rook.
sp98
commented on May 27, 2024
@guits Hi. Do you know if there is any ceph-volume limitation for adding new OSDs to use the existing metadata device?
from rook.
sp98
commented on May 27, 2024
gotcha, i'll do work on that. any word on reasons for not supporting? beyond that we should be good to close. Thanks Santosh!
reason for not supporting it is mentioned in the comment above. Closing it.
from rook.
guits
commented on May 27, 2024
@sp98 you might want to take a look at https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/#sizing
from rook.
barrettMCW
commented on May 27, 2024
slots seem like a good workaround! Thanks guits!
@sp98 can we configure that from the cluster crd?
from rook.
sp98
commented on May 27, 2024
slots seem like a good workaround! Thanks guits! @sp98 can we configure that from the cluster crd?
Reopened. I'll take a look if we can configure this from the Ceph CRD.
A general consensus in the team is for the user to preprovision their metadataDisk with partitions/lvs rather than using Rook to do it via the CephCluster CR.
from rook.
eyitayoafolabi
commented on May 27, 2024
I just encountered this today, in my case I was adding about 24 disk devices using same metadata device and all configured same time. I had 6 nodes which such specification only one node was successful (i.e all 24 OSDs were created on same metadata device) others failed, only creating about 3- 4 osds before hitting to the error failed to configure devices: failed to initialize osd: metadata device /dev/nvme0n1 is not found
from rook.
Related Issues (20)
- mgr: adding filters for prometheus endpoint configuration HOT 2
- CSI config `centry.Namespace` is left empty HOT 6
- There is always an osd process that takes up high cpu and memory. HOT 3
- rgw: default crush_location {{host=rook-ceph-rgw-my-store-a-599b797b97-bnn9x,root=default}} HOT 6
- Creating COSI user causes the object store reconcile to fail several times before finally succeeding HOT 2
- Add a filter for docs check to detect 3rd person("you, they") HOT 4
- mgr prometheus not running. HOT 1
- Update the joblib version to fix CVE-2022-21797 CRITICAL vulnerability HOT 5
- RDMA network support (RoCEv2) HOT 2
- Serviceaccount 'rook-ceph-default' is not being created during deployment through helm (causes mon startup failure) HOT 2
- feat: Add the ability to specify image pull secrets for rook operator HOT 2
- CephFS Pods stuck on ContainerCreating HOT 7
- During object store deletion, delete the RADOS namespaces that were created for the shared object store pools
- Add CI for disabled holder pods
- OSD PDB is created when using an external cluster HOT 4
- disable rbd plugin on one node HOT 5
- doc bug: new multus docs have poorly formatted bullets HOT 1
- Include Grafana dashboards as option to enable in Helm charts
- Remove PdbV1Beta1 support as k8s 1.25 doesnt support it HOT 3
- Rook deletes ceph csi deployments and daemonsets even if it isn't the owner HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rook.