[Bug] Metrics are not aggregated properly as OTEL collector targeting per node about skywalking HOT 15 OPEN

ES checked, the metrics data from one cluster-level API.

from skywalking.

To build service level metrics with instance/node level metrics through AggregateLabels Operation, I think AggregateLabels should support aggregation through specified labels. Like the scenario below:

instance/node HTTP status metrics

# The number means the occurred times
http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2

The MQE express to aggregate service level metrics should be

aggregate_labels(http_status,'status',sum) # aggregate by status label

The result will be

http_status{status=401} 8

from skywalking.

@weixiang1862 As you asked about, we posted #11992 for v10.

from skywalking.

@wu-sheng I'll try to check pulsar first.

from skywalking.

Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.

The question is how to aggregate the metrics of the cluster?

The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.

OK. Then it is fine for now. But you could start a new issue about polishing the Pulsar monitoring as we discussed.

from skywalking.

@CodePrometheus If you could help this, please let me know.

from skywalking.

For all the owners, this list doesn't mean all monitoring at service level is inaccurate due to the way od scraping from OTEL. Please recheck first and only fix if it fits the bug descriptions.

from skywalking.

Kafka monitoring service's MAL aggregates the metrics with cluster and broker, so it's good.

from skywalking.

I don't think we support multiple labels after you have done the persistent.

http_status{host=192.168.0.1, status=401} 5
http_status{host=192.168.0.3, status=401} 1
http_status{host=192.168.0.2, status=401} 2

This is not a proper example, because OAP doesn't persistent data like this.

from skywalking.

In your case, the data in the DB should be http_status_401{192.168.0.1=5, 192.168.0.3=1, 192.168.0.2=2}. Please verify.

from skywalking.

@wu-sheng I would be willing, of course! I'll do recheck on this weekend.

from skywalking.

@wu-sheng I would be willing, of course! I'll do recheck on this weekend.

Which one will you check? Let's avoid duplicate works with others.

from skywalking.

Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.

from skywalking.

Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.

The question is how to aggregate the metrics of the cluster?

from skywalking.

Pulsar monitoring is to aggregate the cluster and node(broker) metrics together now, it should be fine.

The question is how to aggregate the metrics of the cluster?

The prometheus-exporter collects from a single node, the collection of the cluster requires collecting and summarizing each node.

from skywalking.