swift-server / swift-prometheus Goto Github PK

Compiling with [SwiftSetting] = [.enableExperimentalFeature("StrictConcurrency=complete")] produces couple now all too familiar warnings related to concurrency.

Looking at the source code of PrometheusClient as well as other Metrics it seems that most code (if not all) is guarded with a Lock so all public classes could essentially be @unchecked Sendable.

I have in my code opted for this...

import Prometheus

extension PrometheusClient: @unchecked Sendable {}
extension PromGauge: @unchecked Sendable {}
extension PromHistogram: @unchecked Sendable {}
extension PromSummary: @unchecked Sendable {}
let prometheus = PrometheusClient()

let gauge = prometheus.createGauge(forType: Int.self, named: "peers_total")

...to silence compiler and am now testing to see if anything breaks.

Looks like official Sendable conformance can be added to the library, correct?

I developed a API (based on vapor 4) that throws errors with additional information

enum APIError: Error {
    ...
    case invalidParameter(String),
    ...
}

I want to monitor the reasons for a failed response with prometheus so I added the following line:

prometheus.makeCounter(label: "response_error", dimensions: [("error", "\(error)")]).increment(by: 1)

But this results in the following line in the output:

response_error{error="invalidParameter("Invalid value for parameter <code>")"} 28

According to the documentation the "Label values may contain any Unicode characters." but Prometheus can't read this.

Steps to reproduce

import Prometheus

let prometheus = PrometheusClient()
prometheus.makeCounter(label: "response_error", dimensions: [("error", "a\"b\"c")]).increment(by: 1)
let f : (String) -> () = { print($0) }
prometheus.collect(f)

Actual behavior

the result is response_error{error="a"b"c"} 1

Expected behavior

the result should be escaped to response_error{error="a'b'c"} 1. In my first tests response_error{error="a\"b\"c"} 1 also worked but not sure.

Environment

• OS version: MacOS Big Sur Version 11.0.1
• Swift version: Apple Swift version 5.3.1 (swiftlang-1200.0.41 clang-1200.0.32.8)
• SwiftPrometheus 1.0.0-alpha

The threading / locking seems to be wrong in collects.

We should revisit how locking is done, but also perhaps look into a larger redesign; the way we're handling locking is a bit all over the place.

eceived signal 4. Backtrace:
0x55e60b3f9392, Backtrace.(printBacktrace in _B82A8C0ED7C904841114FDF244F9E58E)(signal: Swift.Int32) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:66
0x55e60b3f9627, closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:80
0x55e60b3f9627, @objc closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/<compiler-generated>:79
0x7fbb6950351f
0x55e60b9ccb64, Swift runtime failure: Unexpectedly found nil while unwrapping an Optional value at /build/.build/checkouts/swift-nio/Sources/NIOCore/CircularBuffer.swift:0
0x55e60b9ccb64, NIOCore.CircularBuffer.subscript.getter : (NIOCore.CircularBuffer<A>.Index) -> A at /build/.build/checkouts/swift-nio/Sources/NIOCore/CircularBuffer.swift:158
0x55e60b9cd487, NIOCore.CircularBuffer.subscript.read : (NIOCore.CircularBuffer<A>.Index) -> A at /build/<compiler-generated>:0
0x55e60b9cd3fc, protocol witness for Swift.Collection.subscript.read : (A.Index) -> A.Element in conformance NIOCore.CircularBuffer<A> : Swift.Collection in NIOCore at /build/<compiler-generated>:0
0x7fbb69f7e351
0x55e60bbf3d31, closure #5 (Prometheus.DimensionLabels, Prometheus.PromSummary<A>) -> () in Prometheus.PromSummary.collect() -> Swift.String at /build/.build/checkouts/SwiftPrometheus/Sources/Prometheus/MetricTypes/Summary.swift:96
0x55e60bbf5295, reabstraction thunk helper <A where A: Prometheus.DoubleRepresentable> from @callee_guaranteed (@guaranteed Prometheus.DimensionLabels, @guaranteed Prometheus.PromSummary<A>) -> (@error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed (key: Prometheus.DimensionLabels, value: Prometheus.PromSummary<A>)) -> (@error @owned Swift.Error) at /build/<compiler-generated>:0
0x55e60bbf5295, partial apply forwarder for reabstraction thunk helper <A where A: Prometheus.DoubleRepresentable> from @callee_guaranteed (@guaranteed Prometheus.DimensionLabels, @guaranteed Prometheus.PromSummary<A>) -> (@error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed (key: Prometheus.DimensionLabels, value: Prometheus.PromSummary<A>)) -> (@error @owned Swift.Error) at /build/<compiler-generated>:0
0x7fbb6a007533
0x55e60bbf365d, Prometheus.PromSummary.collect() -> Swift.String at /build/.build/checkouts/SwiftPrometheus/Sources/Prometheus/MetricTypes/Summary.swift:94
0x55e60bbf4e7a, protocol witness for Prometheus.PromMetric.collect() -> Swift.String in conformance Prometheus.PromSummary<A> : Prometheus.PromMetric in Prometheus at /build/<compiler-generated>:0
0x55e60bbf8ed2, closure #1 (Prometheus.PromMetric) -> Swift.String in closure #2 @Sendable () async -> Swift.String in Prometheus.PrometheusClient.collect() async -> Swift.String at /build/.build/checkouts/SwiftPrometheus/Sources/Prometheus/Prometheus.swift:31
0x55e60bbf8ed2, generic specialization <Swift.Dictionary<Swift.String, Prometheus.PromMetric>.Values, Swift.String> of (extension in Swift):Swift.Collection.map<A>((A.Element) throws -> A1) throws -> Swift.Array<A1> at /build/<compiler-generated>:0
0x55e60bbf8ed2, (1) suspend resume partial function for closure #2 @Sendable () async -> Swift.String in Prometheus.PrometheusClient.collect() async -> Swift.String at /build/.build/checkouts/SwiftPrometheus/Sources/Prometheus/Prometheus.swift:31
0x7fbb6a4d849d
0x7fbb6a4d8c4b
0x7fbb6add33f4
0x7fbb6add31a2
0x7fbb6addea91
0x7fbb69555b42
0x7fbb695e79ff
0xffffffffffffffff
Illegal instruction (core dumped)

Feature Request

When reporting Prometheus metrics, it's sometimes useful to always append a certain set of dimensions to every published metric (e.g. nodeId, appName, etc.).

Motivation Behind Feature

We don't control metrics creation in dependencies and can't set dimensions to their metrics. This would mean we won't be able to distinguish such metrics published from different apps or nodes.

Feature Description

PrometheusClient may accept a set of common dimensions it'll append to every published metric.

class PrometheusClient {
  let dimensions: [(String, String)]

  <...>
}

I can see it might clash with current Labels signature and may require a "merge" method on the MetricLabels protocol:

public protocol MetricLabels: Encodable, Hashable {
    /// Create empty labels
    init()

    func append(dimensions: [(String, String)])
}

Alternatives or Workarounds

Currently I'm trying to use a custom conformance to MetricsFactory

class DimensionalPrometheusMetrics: MetricsFactory {
    let prometheus: PrometheusClient
    let commonDimensions: [(String, String)]

    <...>
}

Which is not ideal because it will only work when I publish metrics via swift-metrics api. If need custom buckets/quantiles I'll have to set dimensions manually.

I am trying to use this in my Vapor 4 app. I have followed the Readme and I'm bootstrapping PrometheusClient in the main configure(_ app: Application) method, like this:

let myPrometheus = PrometheusClient()
MetricsSystem.bootstrap(PrometheusMetricsFactory(client: myPrometheus))

And for the /metrics route, I am using the basic implementation too:

app.get("metrics") { request -> EventLoopFuture<String> in
        let promise = request.eventLoop.makePromise(of: String.self)
        DispatchQueue.global().async {
            do {
                try MetricsSystem.prometheus().collect(into: promise)
            } catch {
                promise.fail(error)
            }
        }
        return promise.futureResult
    }

But when I try accessing /metrics in the web browser, I get the following error:

[ WARNING ] prometheusFactoryNotBootstrapped(bootstrappedWith: "PrometheusMetricsFactory(client: Prometheus.PrometheusClient, configuration: Prometheus.PrometheusMetricsFactory.Configuration(labelSanitizer: Prometheus.PrometheusLabelSanitizer(allowedCharacters: "abcdefghijklmnopqrstuvwxyz0123456789_:"), timerImplementation: Prometheus.PrometheusMetricsFactory.TimerImplementation(_wrapped: Prometheus.PrometheusMetricsFactory.TimerImplementation._Wrapped.summary(defaultQuantiles: [0.01, 0.05, 0.5, 0.9, 0.95, 0.99, 0.999])), defaultRecorderBuckets: Prometheus.Buckets(buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 1.7976931348623157e+308])))") [request-id: 1704CFC1-E848-41E3-992A-7DB5CA7B1C0F]

Any help with this would be much appreciated 🙏. And thank you for creating this package!

Hi, @MrLotU ! I read the your discussion. But I still did't understand how to set up SwiftPrometneus to work with tests. In my case it is required to check that metrics are returned along the route.
I initialize the application:
override func setUpWithError() throws { app = Application(.testing) try configure(app) }
Next, I call the route and at least wait for an answer - Ok. But the server error comes.
func testAppMetrics() throws { try app.test(.GET, "v1/metrics", afterResponse: { (res) in XCTAssertEqual(res.status, .ok) let contentType = try XCTUnwrap(res.headers.contentType) XCTAssertEqual(contentType, .plainText) XCTAssertNotNil(res.content) XCTAssertNotNil(res.body) }) }
I use
@testable import App @testable import Metrics import XCTVapor
And in main.swift - MetricsSystem.bootstrap(PrometheusClient())
I would be grateful if you could explain why .internalServerError returns? But when I call the same route through the API client everything works.

I tried to use the library from an iOS app, and everything is compatible except that the compiler gets a bit confused of where to useCGFloat.

While trying to compile we get this error
Use of undeclared type 'CGFloat'

This gets fixed by adding a compiler flag to know if we need to import CoreGraphics on iOS.

Steps to reproduce

1. Import Prometheus library to an iOS App
2. Try to compile the app
3. Error will prevent compiling

Expected behavior

Just be able to compile on iOS

• OS version: iOS 12
• Swift version: 5.1
• Serverside Swift Framework: -
• Framework version: 1.0.0-alpha

Hey! I understand that PrometheusClient () should be initialized only once, but during the execution of unit tests, the tests are run asynchronously and a new Application instance is created in each test, after which I get an error. How can I solve this?
Thanks!!!

Example test
`
func testForceDeleteRelatedEntity() throws {
let app = try self.createTestApp()
let port = app.http.server.configuration.port
let adminToken = try self.createAdminJWT(app)
let entityID = self.entityID
let adminHeaders = HTTPHeaders([("Authorization", "Bearer (adminToken)")])
defer { app.shutdown() }

    try app.testable(method: .running(port: port)).test(.DELETE, "v1/related-entities/\(entityID)/force-delete", headers: adminHeaders, afterResponse: { (res) in
        XCTAssertEqual(res.status, .noContent)
    })
    try app.autoRevert().wait()
}

`

Steps to reproduce

Setup Prometheus and Swift Metrics:

        let client = PrometheusClient()
        
        // Setup Factory
        let factory = PrometheusMetricsFactory(client: client, configuration: .init(defaultRecorderBuckets: [
            0.05, 0.1, 0.5, 1, 5, 10, 25, 50, 75, 100
        ]))
        
        MetricsSystem.bootstrap(factory)
        
        // Collect Metrics
        app.get("metrics") { _ async throws -> String in
            return await client.collect()
        }

Create a Recorder and record a value:

Recorder(label: "x").record(100)

View output and see that bucket sizes are the default bucket sizes rather than the ones I've provided in the setup.

Expected behavior

Bucket sizes should be those which I provided to the factory

Actual behavior

Bucket sizes are instead the defaulted bucket sizes hard coded into the library

Environment

• OS version: 12.4
• Swift version: 5.6
• Serverside Swift Framework: Vapor
• Framework version: 4

Solution

The root cause appears to be that the makeHistogram function calls createHistogram which has a variable called buckets: Buckets = .defaultBuckets. This defaults to defaultBuckets which is incorrect, it should instead be set to configuration.defaultRecorderBuckets (which itself defaults to defaultBuckets but can be overridden by the user, me, as above).

The piece of code implementing the metrics Gauge.record() by making them an add operation is likely not good: https://github.com/MrLotU/SwiftPrometheus/blob/master/Sources/PrometheusMetrics/PrometheusMetrics.swift#L24-L44

I say likely, since the API does not really well define it and we need to fix it over there as well: apple/swift-metrics#34 Gauge.record() is under-defined

Let's see what outcome there is and then make sure we implement the right style here -- i.e. it may have to become a "set".

The parts that are missing:

• Custom bucketing functions (linear and exponential)
• time() -> Some way to time code for users in seconds

See: https://prometheus.io/docs/instrumenting/writing_clientlibs/#histogram

See https://prometheus.io/docs/instrumenting/writing_clientlibs/#overall-structure

This can probably be close to a rename of PromMetric

Feature Request

Motivation Behind Feature

Hi, @MrLotU.
When adapting prometheus data types to swift-metrics api, there's multiple ways to back API types with prometheus types. In particular, both Summary and Histogram can serve as a backend for Timer. In their docs, prometheus compares Summary and Histogram implementations: https://prometheus.io/docs/practices/histograms/#quantiles.
One major difference there is that Summary backed values are not aggregatable. For server side application it's common to have multiple replicas of your backend. This makes Summary based timer a show stopper.

Feature Description

For the reference (it contradicts my point, but I still have to mention it
One solution I see is to make MetricsFactory conformance a wrapper and not an extension to PrometheusClient. This will make the code much more flexible, allow users to pick/write their own MetricsFactory conformance and potentially help us with more elaborate Prometheus -> swift-metrics adapters (see my other issue #36; being able to store baseLabels separately from PrometheusClient will simplify the solution).

Alternatives or Workarounds

It contradicts my point, but I have to mention that java dropwizard adapter uses Summary as a backed for Timer. However, it still forces an anti pattern

avg(http_request_duration_seconds{quantile="0.95"}) // BAD!

Prometheus mentions on their docs.
Python prometheus-flask-exporter uses Histogram as backing type for timer.

In my local fork I wrap PrometheusClient with a custom metrics factory that does this:

    private func makeHistogramBackedTimer(label: String, dimensions: [(String, String)]) -> TimerHandler {
        let createHandler = { (histogram: PromHistogram) -> TimerHandler in
            HistogramBackedTimer(histogram: histogram, dimensions: dimensions)
        }
        if let histogram: PromHistogram<Int64, DimensionHistogramLabels> = prometheus.getMetricInstance(with: label, andType: .histogram) {
            return createHandler(histogram)
        }
        return createHandler(prometheus.createHistogram(forType: Int64.self, named: label, buckets: defaultExponentialBuckets, labels: DimensionHistogramLabels.self))
    }

/// This is a `swift-metrics` timer backed by a Prometheus' `PromHistogram` implementation.
/// This is superior to `Summary` backed timer as `Summary` emits a set of quantiles, which is impossible to correctly aggregate when one wants to render a percentile for a set of instances.
/// `Histogram` aggregation is possible with server-side math magic.
class HistogramBackedTimer: TimerHandler {
    let histogram: PromHistogram<Int64, DimensionHistogramLabels>
    let labels: DimensionHistogramLabels?
    // this class is a lightweight wrapper around heavy prometheus metric type. This class is not cached and each time
    // created anew. This allows us to use variable timeUnit without locking.
    var timeUnit: TimeUnit?

    init(histogram: PromHistogram<Int64, DimensionHistogramLabels>, dimensions: [(String, String)]) {
        self.histogram = histogram
        if !dimensions.isEmpty {
            self.labels = DimensionHistogramLabels(dimensions)
        } else {
            self.labels = nil
        }
    }

    func preferDisplayUnit(_ unit: TimeUnit) {
        self.timeUnit = unit
    }

    func recordNanoseconds(_ duration: Int64) {
        // histogram can't be configured with timeUnits, so we have to modify incoming data
        histogram.observe(duration / Int64(timeUnit?.scaleFromNanoseconds ?? 1), labels)
    }
}

It works ok, but it's clumsy and forces every user with similar use case to wrap.

Feature Request

PushGateway, for the IoT.

Motivation Behind Feature

Because not everything is DMZ webservers.

Feature Description

See https://github.com/prometheus/pushgateway

Steps to reproduce

Specify this in your Package.swift

.package(url: "https://github.com/MrLotU/SwiftPrometheus.git", from: "0.0.0-alpha.1")

and you'll pull in version 0.3.0.

I know mucking with published git tags isn't ideal, but maybe we should delete the old 0.3.0 releases and rename them to something that won't collide with the NIO1-supported tags? 🤔

Expected behavior

We get "mainline" 0.0.0 versions.

Hey, consider me an early adaptor as I am running a fork of VaporMonitoring 1.0 currently in production to monitor a microservice. One change I had to do in my fork was to isolate the shared state of collected metrics behind a lock (serial queue). Otherwise you might run into subtle undefined behavior. Xcode will warn you about this, if you enable the thread sanitizer runtime diagnostic.

In this project the same problem exists. The shared prometheus instance will be accessed from different EventLoops (threads) at the same time reading and mutating the metrics storage. This needs to be synchronized by some form of locking.

You can see this by running your example and:

wrk -t 1 -c 100 -d 5m "http://localhost:8080/hello"

and then accessing the prometheus endpoint. Xcode (with thread sanitizer enabled) will show you a race-condition warning.

So with the @_exports gone, the following is not true anymore right?

import PrometheusMetrics // Auto imports Prometheus too, but adds the swift-metrics compatibility

?

Prometheus has a new experimental feature called Native Histograms.

Native Histograms currently require export via deprecated protobufs, which is a bit weird.

More info:

• PromCon EU 2022: Native Histograms in Prometheus
• Youtube: Prometheus Native Histograms in Production - Björn Rabenstein, Grafana Labs

Feature Request

Add dimension name sanitizer along already existing LabelSanitizer.

Motivation Behind Feature

Dimension names are limited by Prometheus:

may contain ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*

names beginning with __ are reserved for internal use.

Note: Dimension values may contain any Unicode characters.
Note: The metric name (sanitized by LabelSanitizer) follows different rules ([a-zA-Z_:][a-zA-Z0-9_:]*).

The reasoning behind this feature is same as in the #29.

Spec can be found here: https://prometheus.io/docs/instrumenting/writing_clientlibs

Currently SwiftPrometheus has most of these features, but misses a few. We should implement all of these before a 1.0.0 release to also make SwiftPrometheus listable on the prometheus website.

If one creates a Gauge or a Counter with dimensions, an exported value actually contains 2 metrics, this when visualized in Grafana will also result in 2 graphs, one which displays the initial 0 value and one with the actual metric:

let g = Gauge(label: "test_gauge", dimensions: [("abc", "123")])
g.record(10)

This will produce the output of:

# TYPE test counter
test_gauge 0
test_gauge{abc="123"} 10

The issue is slightly touched on in #77

Potential fix/workaround might be master...ordo-one:SwiftPrometheus:gauge_label

Feature Request

We use Prom a lot in Vapor projects. In the contoller we have to handle the future:

    func index(_ req: Request) throws -> EventLoopFuture<String> {
        let promise = req.eventLoop.makePromise(of: String.self)
        try MetricsSystem.prometheus().collect(into: promise)
        return promise.futureResult
    }

This feature request change the usage to

    func index(_ req: Request) async throws -> String {
        return try await MetricsSystem.prometheus().collect()
    }

Motivation Behind Feature

Vapor itself will turn to fully async/await. It would be nice to habe SwiftPrometheus async/await, too.

Feature Description

Support Async/Await

Alternatives or Workarounds

We could write an extension to wrap it into a/a.

It would be good to release the 1.0 since the project was promoted to Incubating tho conditionally on the 1.0 release.

https://forums.swift.org/t/sswg-package-maturity-review-2021/51308

SwiftPrometheus

• adopted by various real projects and very responsive maintainer
• @ktoso fills role of secondary maintainer, in addition to the primary author @MrLotU
• Recommendation:
  • conditional on stable release (pending) and 2nd maintainer (done), promote to Incubating

Do you need help with making this happen @MrLotU ?
Last time we chatted we just wanted to double check some APIs afair?

We can always cut more releases; it's not like staying in these "alpha" really changes all that much -- we're trying to not break anyone and the project is being used in production right now anyway :-)

I do not want to use the default endpoint from Vapor metrics for pulling.
How to tell Prometheus to use a custom one?
for instance localhost/api/data/metric

I'm seeing errors in in collecting metrics using prom.collect(String)

Steps to reproduce

Can't share exact code but will work on reproducer, so far tried to reproduce in tests here but failed, something seems weird.

I got a counter, with some labels and a gauge; Exactly like in the snippet below.

Expected behavior

Scraping should not break.

Actual behavior

# TYPE thingy counter
thingy 0
thingy{a="aaa", b="x"} 9
thingy{a="bbb", b="x"} 6# TYPE my_gauge gauge
my_gauge 0.0

So the thingy{a="bbb", b="x"} 6# TYPE my_gauge gauge is missing a newline and this breaks scraping:

level=warn ts=2019-09-27T08:39:48.356Z caller=scrape.go:930 component="scrape manager" scrape_pool=local target=http://127.0.0.1:8888/metrics msg="append failed" err="strconv.ParseFloat: parsing \"6#\": invalid syntax"

Environment

• Framework version: master branch right now; 05eed57

I'm serving this with NIO directly:

        prom.collect { (buf: ByteBuffer) in
            //...
            context.write(self.wrapOutboundOut(.body(.byteBuffer(buf))))

So pretty sure I've had no chance to break this somehow manually.

Hi,

I was wondering if there are plans for a numbered release soon? It's been over a year since the last one and there seem to have been quite a lot of changes since then some of which would be useful to me.

Thanks

-Peter

The number of PromSummary.values is not limited in any way. They are sorted on everycollect() call and the calculation time grows potentially limitless.

See https://prometheus.io/docs/instrumenting/writing_clientlibs/#overall-structure

When I was using the Summary and Histogram type I noticed, that if I record a value for a metric with name and dimensions, the value is also recorded for the dimension less version of the metric. This is in contrast to the Counter, where an increment on a Counter that was created with a dimension, does not increment the dimension less counter.

We should have a consistent behavior here. @ktoso do you know by any chance what is correct?

Steps to reproduce

    func testHistogramWithLabels() {
        let prom = PrometheusClient()
        var config = PrometheusMetricsFactory.Configuration()
        config.timerImplementation = .histogram()

        let metricsFactory = PrometheusMetricsFactory(client: prom)
        let timer = metricsFactory.makeTimer(label: "duration_nanos", dimensions: [("foo", "bar")])
        timer.recordNanoseconds(1)

        let counter = metricsFactory.makeCounter(label: "simple_counter", dimensions: [("foo", "bar")])
        counter.increment(by: 1)

        prom.collect() { (output: String) in
            print(output)

            XCTAssert(output.utf8.hasSubSequence(#"duration_nanos_count 1"#.utf8)) // <--- 🚨 inconsistent part 1
            XCTAssert(output.utf8.hasSubSequence(#"duration_nanos_count{foo="bar"} 1"#.utf8))

            XCTAssert(output.utf8.hasSubSequence(#"simple_counter 0"#.utf8)) // <--- 🚨 inconsistent part 1
            XCTAssert(output.utf8.hasSubSequence(#"simple_counter{foo="bar"} 1"#.utf8))
        }
    }

    func testSummaryWithLabels() {
        let prom = PrometheusClient()
        var config = PrometheusMetricsFactory.Configuration()
        config.timerImplementation = .summary()

        let metricsFactory = PrometheusMetricsFactory(client: prom)
        let timer = metricsFactory.makeTimer(label: "duration_nanos", dimensions: [("foo", "bar")])
        timer.recordNanoseconds(1)

        let counter = metricsFactory.makeCounter(label: "simple_counter", dimensions: [("foo", "bar")])
        counter.increment(by: 1)

        prom.collect() { (output: String) in
            print(output)

            XCTAssert(output.utf8.hasSubSequence(#"duration_nanos_count 1"#.utf8)) // <--- 🚨 inconsistent part 2
            XCTAssert(output.utf8.hasSubSequence(#"duration_nanos_count{foo="bar"} 1"#.utf8))

            XCTAssert(output.utf8.hasSubSequence(#"simple_counter 0"#.utf8)) // <--- 🚨 inconsistent part 2
            XCTAssert(output.utf8.hasSubSequence(#"simple_counter{foo="bar"} 1"#.utf8))
        }
    }
}

extension Sequence {
    func hasSubSequence<Other: Collection>(_ other: Other) -> Bool where Other.Element == Self.Element, Other.Element: Equatable {
        var otherIterator = other.makeIterator()

        for element in self {
            let next = otherIterator.next()
            if element == next {
                continue
            } else if next == nil {
                return true
            } else {
                // unequal. reset iterator
                otherIterator = other.makeIterator()
            }
        }

        // check if other is also done
        if otherIterator.next() == nil {
            return true
        }

        return false
    }
}

Environment

SwiftPrometheus: 1.0.0

It'd be helpful to people coming by this project to provide a little bit of context about what Prometheus is, that one might want to have a client for it, Swift or otherwise. The fix for this could be a simple one-liner in the README, along the lines of:

A Swift client for the [Prometheus](https://github.com/prometheus/prometheus) monitoring system,
supporting counters, gauges and histograms.

Swift Metrics 2.0 just got released. It is almost completely compatible with Swift Metrics 1.x unless you exhaustively switch on the TimeUnit enum. I would highly recommend depending on swift-metrics like the following:

// swift-metrics 1.x and 2.x are almost API compatible, so most clients should use
.package(url: "https://github.com/apple/swift-metrics.git", "1.0.0" ..< "3.0.0"),

alternatively, from: "2.0.0" will also work but may in the interim cause compatibility issues with packages that specify from: "1.0.0".

If at all possible, this should be done before tagging the next release here. If using "1.0.0" ..< "3.0.0" this isn't even a SemVer major change because 1.x versions are still accepted.

It's an experimental feature to corelate traces to metrics.

OpenMetrics introduces the ability for scrape targets to add exemplars to certain metrics. Exemplars are references to data outside of the MetricSet. A common use case are IDs of program traces.

Exemplar storage is implemented as a fixed size circular buffer that stores exemplars in memory for all series. Enabling this feature will enable the storage of exemplars scraped by Prometheus. The config file block storage/exemplars can be used to control the size of circular buffer by # of exemplars. An exemplar with just a traceID= uses roughly 100 bytes of memory via the in-memory exemplar storage. If the exemplar storage is enabled, we will also append the exemplars to WAL for local persistence (for WAL duration).

https://prometheus.io/docs/prometheus/latest/feature_flags/#exemplars-storage

Not sure entirely where these are attached when emitting metrics, probably need some extra metadata param or something

Ran into an interesting issue when attempting to track request duration via:

let start = Date()
let oneSecond = -start.timeIntervalSince(start.addingTimeInterval(1.0))

It resulted in much larger values than I expected, to show some sample test output comparing the unexpected values:

error: -[SwiftPrometheusTests.PrometheusMetricsTests testSummaryDouble] : XCTAssertEqual failed: ("# TYPE summary_seconds summary
summary_seconds{quantile="0.01"} 1000000000.0
summary_seconds{quantile="0.05"} 1000000000.0
summary_seconds{quantile="0.5"} 4000000000.0
summary_seconds{quantile="0.9"} 10000000000000.0
summary_seconds{quantile="0.95"} 10000000000000.0
summary_seconds{quantile="0.99"} 10000000000000.0
summary_seconds{quantile="0.999"} 10000000000000.0
summary_seconds_count 5
summary_seconds_sum 10130000000000
summary_seconds{quantile="0.01", myValue="labels"} 123000000000.0
summary_seconds{quantile="0.05", myValue="labels"} 123000000000.0
summary_seconds{quantile="0.5", myValue="labels"} 123000000000.0
summary_seconds{quantile="0.9", myValue="labels"} 123000000000.0
summary_seconds{quantile="0.95", myValue="labels"} 123000000000.0
summary_seconds{quantile="0.99", myValue="labels"} 123000000000.0
summary_seconds{quantile="0.999", myValue="labels"} 123000000000.0
summary_seconds_count{myValue="labels"} 1
summary_seconds_sum{myValue="labels"} 123000000000") 

is not equal to

("# TYPE summary_seconds summary
summary_seconds{quantile="0.01"} 1.0
summary_seconds{quantile="0.05"} 1.0
summary_seconds{quantile="0.5"} 4.0
summary_seconds{quantile="0.9"} 10000.0
summary_seconds{quantile="0.95"} 10000.0
summary_seconds{quantile="0.99"} 10000.0
summary_seconds{quantile="0.999"} 10000.0
summary_seconds_count 5
summary_seconds_sum 10130
summary_seconds{quantile="0.01", myValue="labels"} 123.0
summary_seconds{quantile="0.05", myValue="labels"} 123.0
summary_seconds{quantile="0.5", myValue="labels"} 123.0
summary_seconds{quantile="0.9", myValue="labels"} 123.0
summary_seconds{quantile="0.95", myValue="labels"} 123.0
summary_seconds{quantile="0.99", myValue="labels"} 123.0
summary_seconds{quantile="0.999", myValue="labels"} 123.0
summary_seconds_count{myValue="labels"} 1
summary_seconds_sum{myValue="labels"} 123")

The underlying swift-metrics saves everything as Int64 and assumes nanoseconds, as all other methods convert then call to that method. From the Prometheus docs on naming:

Metrics must use base units (e.g. seconds, bytes) and leave converting them to something more readable to graphing tools. No matter what units you end up using, the units in the metric name must match the units in use.

This follows an assumption that Prometheus client libraries use seconds as well.

I believe we should strive to create an idiomatic client, which I think means exporting the metrics in the desired unit. Given the assumption that units are encoded in the metric name, I think that's potentially a good path to investigate further.

Feature Request

We should implement the workflow laid out by swift-metrics which will allow this to be plugged in as one of the backends services can choose to utilize.

Motivation Behind Feature

Support the SSWG standardization.

Feature Description

This will require a bump in major version since it will drastically change the API.

Steps to reproduce

Prometheus makes a few too many assumptions about my metrics system factory. I use the multiplex factory that comes from Apple and bootstrap prometheus with it, but then this code tries to cast the multiplexed factory and fails:

guard let prom = self.factory as? PrometheusWrappedMetricsFactory else {
    throw PrometheusError.prometheusFactoryNotBootstrapped(bootstrappedWith: "\(self.factory)")
}
return prom.client

Expected behavior

No error is thrown...cause it was in fact bootstrapped

Actual behavior

It throws the error

Environment

• OS version:
• Swift version:
• Serverside Swift Framework:
• Framework version:

The parts that are missing:

• time() -> Some way to time code for users in seconds.

See https://prometheus.io/docs/instrumenting/writing_clientlibs/#summary

I am a bit confused about how to use the Histogram style of metrics successfully with swift-metrics.

When interacting with the Timer interface from swift-metrics you can specify the unit of your time measurements. But when configuring PrometheusMetricsFactory I cannot specify the unit of my histogram buckets. If I use the default values ([0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]), most measurements will end up in the +Inf bucket, because internally the MetricsHistogramTimer always records nanoseconds. I think the only way to make it work right know would be to change the buckets to be in ns, [0.005e9, 0.01e9, 0.025e9, 0.05e9, 0.1e9, 0.25e9, 0.5e9, 1e9, 2.5e9, 5e9, 10e9] which seems quite unintuitive.

Probably it would make sense to default to seconds for the unit and change the MetricsHistogramTimer implementation to convert the nanoseconds to seconds. Since this is a global setting it will apply to ALL "timer" metrics emitted by any library in the process. So maybe this should be part of the TimerImplementation configuration.

Example

let prometheusClient = PrometheusClient()
MetricsSystem.bootstrap(PrometheusMetricsFactory(client: prometheusClient,
                                                 configuration: .init( timerImplementation: .histogram())))

let timer = Timer(label: "my_timer")

timer.record(.seconds(5))

prometheusClient.collect { (metrics : String) -> () in
    print(metrics)
}

# TYPE startup_time histogram
startup_time_bucket{le="0.005"} 0
startup_time_bucket{le="0.01"} 0
startup_time_bucket{le="0.025"} 0
startup_time_bucket{le="0.05"} 0
startup_time_bucket{le="0.1"} 0
startup_time_bucket{le="0.25"} 0
startup_time_bucket{le="0.5"} 0
startup_time_bucket{le="1.0"} 0
startup_time_bucket{le="2.5"} 0
startup_time_bucket{le="5.0"} 0
startup_time_bucket{le="10.0"} 0
startup_time_bucket{le="+Inf"} 1
startup_time_count 1
startup_time_sum 5000000000

• Swift version: 5.6.1
• Framework version: 1.0.0
• swift-metric: 2.3.1

The parts that are missing:

• setToCurrentTime() -> Set gauge to current unixtime in seconds
• trackInprogress() -> A way to track in-progress requests in some piece of code/function
• time() -> A way to time a piece of code and set the gauge to its duration in seconds

See: https://prometheus.io/docs/instrumenting/writing_clientlibs/#gauge

The GitHub repo link still links to the old https://mrlotu.github.io/SwiftPrometheus, which is 404 now. I guess it should be a link to the Swift Package Index instead.

Feature Request

Offer some configuration to replace characters in labels, i.e. when people in their libraries have Counter("things.specificThing.thingsToCound") that's an invalid label for prometheus. So we should allow some labelSubstitutions = [".": "_"] or perhaps some other way?

Motivation Behind Feature

Libraries which offer metrics via Swift Metrics generally "should not know" details about formats of labels that any specific backends require. This is mostly about the use of . (prom crashes on those, other backends like those) as separator vs using _ (prom likes those, other backends dislike those).

Alternatives or Workarounds

• No go alternative: force all possible libraries to avoid . 😉