Adding “volume group” metrics to pure-fa-openmetrics-exporter and pure-fb-openmetrics exporters about pure-fa-openmetrics-exporter HOT 15 CLOSED

Hi @s-kuchi

The metrics for all volumes in a volume group is already collected using the /volumes endpoint.

The volume name is prefixed with the volume group name

volume-group-name/volume-name

The volume group is also included as a label volume_group.

# HELP purefa_volume_performance_throughput_iops FlashArray volume throughput
# TYPE purefa_volume_performance_throughput_iops gauge
purefa_volume_space_bytes{naa_id="naa.624a9370200000000000000000000000",name="test-vol-group/test-vol",pod="",space="total_physical",volume_group="test-vol-group"} 0

I would aggregate these at the observability platform level, for example in Prometheus we could use the following queries in PromQL.

Prometheus Example

PromQL to sum all volumes by volume group and return only volumes with name 'vol-group'

sum(purefa_volume_space_bytes{instance="array01", space="total_provisioned", name=~".+vol-group.+"}) by (volume_group) /1024/1024/1024

PromQL sum provisioned capacity by volume group not equal to "".

sum(purefa_volume_space_bytes{instance="array01", space="total_provisioned", name!=""}) by (volume_group) /1024/1024/1024

If this solution doesn't work for you, can you provide us with a bit more context of your use case?

from pure-fa-openmetrics-exporter.

Hi @james-laing : Thank you for that info. I am trying to grab the performance data (read,write, latency, iops and bandwidth) for "Volume groups" which is available on Pure GUI/web console. Currently, I am running the following query to collect read latency for all the volumes in the volume group, sum them and average them, but i am 100% sure if that is how Pure GUI is displaying it and i want to match what Pure GUI is showing me.

avg(sum(purefa_volume_performance_latency_usec{instance=~"vapdsms",name=~"PP3SMS/.+",dimension="usec_per_read_op"}) by (name))

from pure-fa-openmetrics-exporter.

@s-kuchi, you're almost there.

Instead of averaging all datapoints, take a sum of the avg_over_time of the defined interval.

If you are using the Pure Storage FlashArray Overview Grafana dashboard you can take a look at how we developed the queries.

Try this:

sum(avg_over_time(purefa_volume_performance_latency_usec{instance="vapdsms",dimension="usec_per_read_op",name=~"PP3SMS/.+"}[2m])) by (name)

If you are using Grafana consider using the $__interval variable to set a dynamic interval to the timeframe.

Don't try to match the graphs exactly like-for-like between monitoring platforms as they all aggregate differently and therefore will never match.

from pure-fa-openmetrics-exporter.

@james-laing : I did try the query you mentioned, but it gives me one line for each volume in the volume group, which is not what Pure GUI volume group metric shows. The GUI shows one aggregated datapoint for all the volumes in the volume group which I am assuming is the average of the sum. I agree that I shouldn't try to match but I was trying to get to as close to the GUI's metric as possible and sum(avg_over_time...) is not it.

from pure-fa-openmetrics-exporter.

Just remove 'by (name)' from the end of the query and it will sum all the averages for the entire volume group.

from pure-fa-openmetrics-exporter.

Tried that but the value of each datapoint is not accurate. As you see compare Grafana shows 7.5 ish msec vs Pure GUI One shows < 0.6 ish msec

from pure-fa-openmetrics-exporter.

The difference is the units of measurement.

FlashArray GUI is in milliseconds (msec).
The metrics from the exporter are in microseconds (usec).
Grafana looks to be configured to apply milliseconds (msec) to the data points.

In Grafana GUI, right panel are all the options to customise the graph. Scroll down to 'Standard Options' and change the units to microseconds (μs).

Also consider using the $__interval variable in Grafana to set a dynamic interval to the timeframe.
https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#__interval

from pure-fa-openmetrics-exporter.

@james-laing : It was/is set to microseconds.

from pure-fa-openmetrics-exporter.

Excellent - thanks for confirming. Let me recreate this and get back to you.

from pure-fa-openmetrics-exporter.

@s-kuchi please could you provide me with all the versions of software you are running in this environment?

• Pure Storage FA OpenMetrics Exporter version
• Prometheus version
• Grafana version
• Purity version

from pure-fa-openmetrics-exporter.

Openmetrics version is the latest
[root@pdgllvsanmgt01 ~]# docker image inspect quay.io/purestorage/pure-fa-om-exporter:latest
"RepoTags": [
"quay.io/purestorage/pure-fa-om-exporter:latest"
"DockerVersion": "20.10.12",

Prometheus version :
prometheus, version 2.44.0-rc.0 (branch: HEAD, revision: d70688038e5d444ab8beac53722df15d244038e4)
build user: root@c3e52c6dd464
build date: 20230426-09:46:54
go version: go1.20.3
platform: linux/amd64

Grafana version: 9.5.1

Purity version : 6.2.13

from pure-fa-openmetrics-exporter.

Apologies for the radio silence, it's holiday season. I have been trying to recreate this issue in the lab but so far unsuccessful. I have built an environment similar to your own using the same versions of each platform.

Perhaps you could reach out to your Pure Storage Account Manager and ask them to setup a meeting with you and I and we could take a look at this problem together?

from pure-fa-openmetrics-exporter.

The only version I could not test was Purity 6.2.13 as 6.2.x was a standard release with a short life cycle which has gone end of life.

I managed to test on Purity releases:

• 6.1.23
• 6.3.8
• 6.4.5

The best way to test this accurately is to run PromQL query in Prometheus to see the raw output against the //FA GUI.

Reach out to your Pure Storage Account Manager or System Engineer if you wish to setup a call with me.

from pure-fa-openmetrics-exporter.

@james-laing : Can you provide me your query which generated the above graphs in Grafana?

from pure-fa-openmetrics-exporter.

Sure - these are just slightly modified versions of ones in the dashboard.
https://grafana.com/grafana/dashboards/18515-pure-storage-flasharray-overview/

Latency:

sum(avg_over_time(purefa_volume_performance_latency_usec{instance=~"^$instance",env=~"^$env",dimension=~"usec_per_read_op|usec_per_write_op|san_usec_per_read_op|san_usec_per_write_op",name=~"JL-VolGroup-01.+"}[$__interval])) by (dimension)

IOps

sum(avg_over_time(purefa_volume_performance_throughput_iops{instance=~"^$instance",env=~"^$env",dimension=~"reads_per_sec|writes_per_sec",name=~"JL-VolGroup-01.+"}[$__interval])) by (dimension)

Bandwidth Utilisation

sum(avg_over_time(purefa_volume_performance_bandwidth_bytes{instance=~"^$instance",env=~"^$env",dimension=~"read_bytes_per_sec|write_bytes_per_sec",name=~"JL-VolGroup-01.+"}[$__interval])) by (dimension)

from pure-fa-openmetrics-exporter.