Can we consider to add "fsid" as a filter variable alongside with "cluster"? about ceph_exporter HOT 4 CLOSED

@neurodrone
Thanks! I hadn't heard re-label before. This is really a useful feature. I came out a working configuration.

scrape_configs:
  - job_name: 'ceph_rgw_exporter'
    relabel_configs:
    - source_labels: ["cluster"]
      replacement: "rgw"
      action: replace
      target_label: "clusters"
    static_configs:
    - targets: ['localhost:9128']
      labels:
        alias: ceph_rgw_exporter

  - job_name: 'cephfs_exporter'
    relabel_configs:
    - source_labels: ["cluster"]
      replacement: "cephfs"
      action: replace
      target_label: "clusters"
    static_configs:
    - targets: ['10.19.250.136:9128']
      labels:
        alias: cephfs_exporter

  - job_name: 'ceph_rbd_exporter'
    relabel_configs:
    - source_labels: ["cluster"]
      replacement: "rbd"
      action: replace
      target_label: "clusters"
    static_configs:
    - targets: ['10.19.250.132:9128']
      labels:
        alias: ceph_rbd_exporter

from ceph_exporter.

To note in our production clusters we are not using single ceph-exporter anymore. Each of our clusters have their own ceph-exporters. We have found this model much more simpler to manage and should resolve the issue you are facing as well.

from ceph_exporter.

@neurodrone
Thanks for your information, but I am still little confused. Do you mean each cluster has its own Prometheus?

By far, I also use separate ceph-exporter for different clusters too, and these three ceph-exporter deliver data to the same Prometheus. Although they are separated into jobs, in the Grafana dashboard, these data still cannot be differentiated.

I doodled a picture to show what the architecture of my environment. So I suppose you guys have different Prometheus server for different ceph cluster, right?

from ceph_exporter.

Thanks for the elaborate diagram, @blackpiglet! We have the exact setup - so multiple ceph-exporters, each per cluster and they are all scraped by a single prometheus (typically a unified global set of prometheus servers for fault-tolerance).

Since the instances that run ceph-exporter are distinct we can relabel the targets within (as seen here) the prometheus server config to persist the time-series separately for each cluster by adding a unique label for every ceph-exporter. That should give you the per-cluster breakdown you need to query. Let me know if this approach makes sense otherwise I can further elaborate.

from ceph_exporter.