After upgrading to Rook 1.12.10 and Ceph Reef <code c

exec "ceph osd tree" to see the type of osd ; is have ssd ? </blockqu

crush rules being recreated, deviceClass has changed (but it hasn't?) about rook HOT 6 OPEN

briend commented on June 7, 2024

crush rules being recreated, deviceClass has changed (but it hasn't?)

from rook.

Comments (6)

briend commented on June 7, 2024

I suppose this is because "default~ssd" != "ssd" (but that seems too obvious)

from rook.

wanghui-devops commented on June 7, 2024

exec "ceph osd tree" to see the type of osd ; is have ssd ?

from rook.

briend commented on June 7, 2024

exec "ceph osd tree" to see the type of osd ; is have ssd ?

Yes, they are all ssd in this test cluster

ceph osd tree | grep osd
ID   CLASS  WEIGHT   TYPE NAME                STATUS  REWEIGHT  PRI-AFF
  2    ssd  0.93149                  osd.2        up   1.00000  1.00000
  1    ssd  0.93149                  osd.1        up   1.00000  1.00000
  0    ssd  0.93149                  osd.0        up   1.00000  1.00000
  3    ssd  0.93149                  osd.3        up   1.00000  1.00000

from rook.

sp98 commented on June 7, 2024

I suppose this is because "default~ssd" != "ssd" (but that seems too obvious)

This looks like the root cause.

With no deviceClass set in the Pool, I have following crush rule:

{
        "rule_id": 1,
        "rule_name": "replicapool",
        "type": 1,
        "steps": [
            {
                "op": "take",
                "item": -1,
                "item_name": "default"
            },
            {
                "op": "choose_firstn",
                "num": 0,
                "type": "osd"
            },
            {
                "op": "emit"
            }
        ]
    },

The rook operator is comparing "default" != "" over here for the deviceClass.

Thoughts @subhamkrai ?

from rook.

briend commented on June 7, 2024

We do have deviceClass set on all of our pools, so maybe that is unusual. We had to do that to avoid overlapping roots with multiple device classes.

This probably ought to extract the deviceClass after the "~" character in ItemName ?

rook/pkg/daemon/ceph/client/pool.go

Line 544 in 4a12234

deviceClass = step.ItemName

Here are the decompiled crush rules in case that is helpful:

rule ceph-ops1-block-pool-1 {
        id 11
        type replicated
        step take default class ssd
        step chooseleaf firstn 0 type host
        step emit
}
rule ceph-ops1-block-pool-1_host_ssd {
        id 12
        type replicated
        step take default class ssd
        step chooseleaf firstn 0 type host
        step emit
}

from rook.

subhamkrai commented on June 7, 2024

Thanks for all the clues.

I have not tested this but I guess below sample code will fix

func extractPoolDetails(rule ruleSpec) (string, string) {
	---
	for i, step := range rule.Steps {
		---
		if step.ItemName != "default" {
			deviceClass = step.ItemName
		}
	---
	}

	if strings.Contains(deviceClass, "default~") && len(strings.Split(deviceClass, "~")) == 2 {
		deviceClass = strings.Split(deviceClass, "~")[1]
	}
	return failureDomain, deviceClass
}