chaostoolkit-incubator / chaostoolkit-prometheus Goto Github PK

Hi guys,
At the moment it seems that we are pretty limited when query prometheus on steady-state.
we are using jsonpath tolerance type in order to verify our promQL result.
for example:

{
    "type": "probe",
    "name": "prometheus steady state  consumer rate query - 1 min ago",
    "tolerance": {
        "expect": "1",
        "type": "jsonpath",
        "path": "$.data.result[0].value[1]"
    },
    "provider": {
        "type": "python",
        "module": "chaosprometheus.probes",
        "func": "query",
        "arguments": {
            "query": "${prometheus_ts_consumer_rate_query}",
            "when": "1 minutes ago"
        }
    }
}

this type of query can only be boolean and we need a way to do more in depth verification on the results. for example:

• checks if the Prometheus results are below a given threshold
• checks if Prometheus results are higher than a given threshold

this features will able prometheus probes to be much more flexible :)

Prometheus returns scalar values as strings in order to support NaN, Inf, and -Inf. Unfortunately, this makes it quite difficult to use the Toolkit's jsonpath support to filter based on these scalar values, as jsonpath2 does not perform comparisons between numbers and numbers formatted as strings.

I would like to use the following tolerance to ensure a metric is within a certain range:

{
    "type": "jsonpath",
    "path": "$.data.result[*][?(@.value[1] > 90)]",
    "count": 1,
    "target": "stdout"
}

In order to do so, I am currently relying on a custom probe script which will fetch the Prometheus metrics and reformat the scalars represented as strings into floating point values. I think this would be useful functionality to have in the Prometheus extension, and it is easier to implement the change here than in jsonpath2.

The build badge in the README points to the wrong Travis CI repository. Should point to https://travis-ci.org/chaostoolkit-incubator/chaostoolkit-prometheus

The discover function does not seem to be working for this extension.
Running chaos discover chaostoolkit-prometheus give the following output:

$ chaos discover chaostoolkit-prometheus
[2019-07-21 12:20:35 INFO] Attempting to download and install package 'chaostoolkit-prometheus'
[2019-07-21 12:20:36 INFO] Package downloaded and installed in current environment
[2019-07-21 12:20:36 CRITICAL] package 'chaosprometheus' does not export a `discover` function

If I understand things correctly (and comparing with other extensions) the prometheus package does not expose the required discovery functions in file chaosprometheus/__init__.py.

Would adding the required code in chaosprometheus/__init__.py resolve this issue?