When I access the /metrics endpoint on my gluster peers.

I currently only get the endpoints for gluster_* are

gluster_cpu_percentage
gluster_elapsed_time_seconds
gluster_memory_percentage
gluster_resident_memory
gluster_virtual_memory

Linux g01 4.14.5-92 #1 SMP PREEMPT Mon Dec 11 15:48:15 UTC 2017 armv7l armv7l armv7l GNU/Linux

glusterfs 4.1.4

In order to include the gluster_exporter in the official Fedora containers (and later CentOS), gluster-prometheus needs to be included in the Fedora repository. Please see if one of the main contributors can add and maintain the Fedora package. The steps in the Package Review Process explain what is needed to get the package included.

Feel free to contact me if any assistance is needed.

Gluster Prometheus exporter should be compatible with glusterd1 and
glusterd2 whenever it is possible.

Below flags to exporter will help to identify the management daemon
used.

--gluster-mgmt=<name>  (Choices: glusterd1, glusterd2. Default is "glusterd1")
--glusterd-dir=<path> (Default is `/var/lib/glusterd2` in case of
    glusterd2 else `/var/lib/glusterd`)

Plan

Metrics collectors should not run gluster cli or REST api calls. They
all consume input json files and export the metrics or collects the
additional details and exports it.

Example - Number of Volumes: A separate thread which runs gluster volume info --xml if configured to use glusterd1 else calls REST API
GET http://localhost:24007/v1/volumes. Both will generate the output
json in the same format which will be understood by metrics collector
thread. Metrics collector thread need not check the data coming from
glusterd1 or glusterd2.

Example - Brick utilization: Volume info json collected as above,
metrics collector will look into the volinfo.json and collects the
utilization details and exports the details.

Example - Volume Profile: Compatibility thread will run CLI and
consume xml in case of glusterd1, runs REST API in case of
glusterd2.

Support of enable/disable of Metrics

All exported metrics should expose configuration via CLI or config to
enable/disable. For example, --utilization-metrics=enabled. This
will give more control for choosing the metrics which are compatible
for the target deployment.

Auto disable Metrics collection if not compatible

If a metric is not supported with glusterd1, that should be
automatically disabled based on the flag --gluster-mgmt=glusterd2

Glusterfs 4.0 brings tons of useful metrics, which can be read from /var/run/gluster/metrics after sending USR2 to gluster processes (https://docs.gluster.org/en/latest/release-notes/4.1.0/#monitoring).
Is it planning to add those metrics into exporter?
Would be very helpful for us.

This metric is useful for having alerts if the allowed limits for a peer for these counts exceed the limits.

RPM package name and archive tar name is gluster-prometheus-exporter but the binary name is gluster-exporter and service file name is gluster-exporter.service. Change this name to gluster-prometheus-exporter and gluster-prometheus-exporter.service

We should write the rpm spec so that rpm can be built out of this project

Hi, please provide a roadmap for this gluster prometheus exporter

Currently JSON format is used for configuration files(global and collectors). We can merge both the configs into one and use toml format for better usability.

Example config file

[globals]
gluster-mgmt = "glusterd"
glusterd-dir = "/var/lib/glusterd"
gluster-binary-path = "gluster"
gd2-rest-endpoint = "http://192.168.10.11:24007"
port = 8080
metrics-path = "/metrics"

[collectors.gluster_ps]
name = "gluster_ps"
sync-interval = 5
disabled = false

[collectors.gluster_brick]
name = "gluster_brick"
sync-interval = 5
disabled = false

Volumes that are deleted from the cluster still reporting metrics in all the counters. To reproduce just create and delete a volume. restarting the gluster-exporter.service does not clean and lsof does not list a file that should be cleaned to reset the metrics. Do i miss something on how to operate the exporter?

Monitoring Mixins are the way to add grafana dashboard templates and alerting rules to the K8s/OpenShift.

https://github.com/kubernetes-monitoring/kubernetes-mixin

Inside Geo Replication, we will need:

1. gluster_geo_rep_session_total
2. gluster_geo_rep_session_up
3. gluster_geo_rep_session_stopped
4. gluster_geo_rep_session_created
5. gluster_geo_rep_session_partial
6. gluster_geo_rep_session_paused
7. gluster_geo_rep_session_down

is this project replacing https://github.com/ofesseler/gluster_exporter or any relationship?
This looks pretty much not maintained however in Prometheus third party integration is the one mentioned:
https://prometheus.io/docs/instrumenting/exporters/

I have tested that exporter it seems collecting some metrics that here are not available?

thanks

Problem:

There are a number of isolated discussions regarding how this code will be used in kube:

• #3 - general how to deploy this repo
• #17 - how to deploy
• #22 - What to do about cluster-wide metrics vs node-level metrics

Aside from a request for general how-to documentation, there is a need to figure out how to provide both node-level and cluster-level metrics. Providing cluster-level metrics from all nodes (pods) will lead to duplication.

Discussion

The configuration seems to be going the way of being able to enable/disable various metrics per instance (#24). This seems like a reasonable way to fix the above issue. It would entail running a per-node collector for node-level metrics and a single Deployment per cluster for cluster-level metrics.

An alternative would be to have the node-level collectors participate in leader-election, with the result being a single instance that exports cluster-level metrics in addition to its node-level metrics. This strikes me as considerably more complicated than the static approach, above.

Request

Choose one of the above (or another) approaches and document it as the plan for eventual deployment. It need not be implemented immediately, but the choice here affects other projects, namely https://github.com/gluster/anthill.

Can pull peerID automatically either by

$ gluster pool list | awk '/localhost/ {print $1}'
or
$ cat /var/lib/glusterd/glusterd.info which stores the Local Peers UUID

The second seems a better option because a go script can read the file.

parts := strings.Split(gdInfo, "=")
parts[1]

Should return the UUID

Add support for collecting and exporting Volume profile metrics compatible to Glusterd1 and Glusterd2

While trying to perform an install of the latest software, I get an error:

Odroid HC2 - Armbian - Ubuntu 18.04.1 LTS - 4.14.69-odroidxu4
go version go1.10.1 linux/arm

Go1.9 or later is available on the system.

dep package is missing on the system
gometalinter package is missing on the system
Makefile:61: recipe for target 'check-reqs' failed
make: *** [check-reqs] Error 1

gometalinter isn't released for arm32 or arm64 so has to be installed from source at build time.

This is required to make sure if profiling related metrics needs to be collected, profiling should be enabled for the volumes.

This would be useful metrics figuring out if the maximum allowed no of bricks on a gluster node is reached.
We can add a metrics namely gluster_volume_snapshot_brick_count by looking at no of snapshots for a volume and deciding the no of bricks for snapshots.

we are dependent on git for generating versions. if we have src tarball copied to some other location, say its not git-repo, make gluster-exporter fails, in Centos I get
/usr/lib/golang/pkg/tool/linux_amd64/link: -B argument must have even number of digits: 0xundefined

1. gluster_rebalance_files
2. gluster_rebalance_bytes
3. gluster_rebalance_skipped
4. gluster_rebalance_failures

Some volumes may be used by many clients. Performance may be impacted if many clients use the same volume. A count of the clients connected to a volume (or even per brick?) will be useful.

With the v0.1 of GCS, we now have working gd2 + CSI driver combo, which can be deployed on k8s cluster, and one can start using PVC to get PV for their pods...

It would be great to get this project integrated with the project to show the real value to our users!

We need to document how the gluster-prometheus works and how it fits and gets consumed in overall gluster eco-system or GCS (Gluster Container Storage) eco system.

Not sure if it relates to the expose disk stats issue but would be useful to have the device mapper like (dm-X) name as label for any brick statistic since it would make it easier to identify the brick in telegraf diskio.

Current registering function has a signature as follows,
func () error
During the implementation side, we tend to use global variables (which are initialized in the main function).
This issue is created to standardize the registering function in such a way that we avoid any use of global variables and instead we pass the needed variables as argument into the function.

For example, it can be changed to accept a GInterface object (as most of the time we use a GInterface object to get LocalPeerID etc)
func (glusterutils.GInterface) error

or if you require / need config details also in the registering function, the signature can be,
func (glusterutils.GInterface, *conf.Config) error

We may have to asses the existing code and see what all global variables are being used and generalize the function signature.

1. gluster_volume_status
2. gluster_brick_status

Can you provide an templated dashboard at grafana.net for the exporter? It's better to have them there, rather than having to hand-roll yourself each time.

I've created a Dockerfile, the project builds and runs, but it doesn't serve any Gluster info (no crash though).

I didn't give it access to any resources (like Gluster directories, files, devices and sockets), because I don't know which ones are needed.

Could someone explain (or document) these requirements?

Sub volume utilization is exported from each nodes, Rules needs to be added to get Volume utilization.

It would be useful to have the number of volumes provided as well. With this users can see if there are any kubernetes/openshift jobs requesting many volumes at a certain time.

Sorry for creating an issue but I was not sure where ti place a question.

I am using glusterfs storage inside opensshift, it runs as pods and provides storage to to other pods/apps in the openshift cluster.

I assume that gluster-prometheus is designed to be running on the same host where gluster is running. But how can I use it to scrape data from the pods in the openshift, if that is at all possible?

Are there any metrics available to monitor gluster heal status and peer status?

Thanks!

Based on brick path export the metrics about the underlying LVM. Below metrics would be required

• LVM data size
• LVM used percentage
• LVM metadata size
• LVM metadata usage percentage

@aravindavk @JohnStrunk @atinmu @amarts comments?
Or is there something else which we need to capture as well as part of LVM usage metrics?

Currently GP provides gluster_volume_profile_total_writes, gluster_volume_profile_total_reads, gluster_volume_profile_duration_secs which are cumulative stats. These cannot be used to calculate the IOPS for a brick or a volume. GP should provide calculated brick level iops based on the refresh interval of GP.

This is required to figure out total capacity of volume to calculate used percentage for alerting purpose

RHGSWA - supports multi-cluster scenario so this will need the unique identifier.

Underlying disks stats for bricks would be meaningful to figure out how the actual disks are performing.

1. gluster_prometheus_up
2. gluster_server_up
3. gluster_gd2_up
4. gluster_csi_up
5. gluster_block_csi_up
6. gluster_block_up
7. gluster_operator_up

Add a metric for currently install glusterfs and glusterd versions.
Easily see which nodes in the cluster aren't running at the latest/current stable version.

Provide brick metrics like inode utilization, thin pool utilization, free/total space in brick level.
provide fops count by enabling diagnostics of fops.

We need to define the hierarchical structure of metrics which would be maintained with prometheus. A sample structure could be something like

clusters/{cluster-id}/{cluster-specific-metrics like cluster health, no of connections etc}
clusters/{cluster-id}/volumes/{volume-id}/{volume-specific-metrics like no of bricks, status etc}
clusters/{cluster-id}/volumes/{volume-id}/bricks/{brick-id/brick-name}/{brick-specific-metrics like utilization etc}

Hi,

i am not sure how to provide debug data but i while running some tests with smallfile utility i noticed that the metrics exposed by gluster-prometheus-exporter gets invalidated at some point and they take really long time to recover or decrease. Example:

• version tag 0.3
• cpu metric hits > 100%
• cpu metrics remains high
• after aprox 12 hrs cpu metric slowly goes down but does not reflect reality
• i do not run any aggregation on the metric
  see graphs for example.
  
  As you can see the cPU hit the 200% then even if i stopped the volume the metric remains high. overnight with the volume stopped the metric decreases slowly but without reaching the actual value.
  Not sure if this could affect other counters but does not look like this. Also if this % metric is going above 100 should it have a ceil() prometheus function?

provide metrics on volume level and cluster level metrics.
Number of volumes, state of volumes, number of nodes with state, volume utilization

Add support for exporter to export Self-Heal related metrics compatible to both Glusterd1 and Glusterd2.

how to use this exporter in kubernetes

Following metrics are required on volume profiling info:
File Operations For Locks Trends
File Operations for Inode Operations
File Operations for Entry Operations
File Operations for Read/Write

Created this list as one place where all the required metrics from WA perspective are captured. These might be overlapped with other issues already listed

• Cluster Level
  -- Overall Status (healthy/unhealthy)
  -- No of snapshots
  -- No if peers
  -- No of volumes
  -- No of bricks
  -- geo-rep session (state wise breakup)
  -- no of client connections
  -- Overall IOPS
  -- Overall capacity utilization
  -- Capacity Available
  -- Weekly growth rate
  -- weeks pending
  -- Throughput
  -- Volume status wise breakup
  -- Host (peer) status wise breakup
  -- Brick status wise breakup

• Volume Level
  -- State
  -- Bricks (status wise breakup)
  -- No of bricks
  -- geo-rep sessions (state wise breakup)
  -- rebalance counters
  -- rebalance status
  -- no of snapshots
  -- capacity utilization
  -- capacity available
  -- weekly growth rate
  -- weeks pending
  -- IOPS
  -- LVM utilization (metadata and data)
  -- profiling info (e.g. fop for locks, read-write, inode operations, entry operations)

• Brick Level
  -- status
  -- capacity utilization
  -- capacity available
  -- weekly growth rate
  -- weeks pending
  -- heal information
  -- IOPS
  -- LVM usage (metadata and data)
  -- throughput
  -- latency
  -- mount point utilization

• Host/Peer Level
  -- Status (connected/disconnected)
  -- no of bricks
  -- bricks status wise breakup
  -- memory available
  -- memory utilization
  -- swap free
  -- swap utilization
  -- CPU utilization
  -- network throughput
  -- dropped packets per sec
  -- errors per sec