SPIKE: Secretless benchmark metrics are defined about secretless-broker HOT 2 CLOSED

In the feature spec we are focused on TCP - can we focus on TCP here as well? Can we also ensure scope is clear, eg right now we only need to measure post-connection streaming latency and throughput. In particular, from the doc we have:

Secretless latency under ‘light’ load and maximum throughput (Mbps) per single Secretless container has been measured. ‘Light’ load or operating profile is defined as above, where Secretless is communicating with a database cluster with variable data size per request.

I think in this card we are asking: how can we measure post-authentication handshake latency / throughput?

One other relevant requirement:

The telemetry data output by Secretless must be easy to query across an arbitrary period of time with at least a week of history, and related events should be tagged in a way that makes it easy to aggregate them.

So at a minimum, the data that's output has to be timestamped and tagged to indicate which specific connection the datapoint is relevant to.

from secretless-broker.

The outcome from this spike is captured in the telemetry branch, where there is a reference implementation of metric measurement, collection, export and analysis.

For making measurements we start by noting that network I/O in Go is blocking. That means reads will block a goroutine until the buffer has something. We use io.Copy to implement unidirectional streaming, it takes as input a destination io.Writer and a source io.Reader. Bidirectional TCP streaming in Secretless (see duplexStream) is the result of 2 Go routines each carrying out streaming (via io.Copy) in a particular direction, taking as input the client and target TCP connections. io.Copy handles all the reading, writing and buffering. The io.Copy for each direction blocks in its goroutine for the lifetime of the streaming.

In order to take measurements, we must instrument each TCP connection instance by wrapping it to intercept reads and writes.

Metrics definitions

The metrics are defined below. NOTE that each metric is e

TCP streaming latency

The time between when a source connection's Read unblocks from the arrival of some packet to when the destination connection's Write for that packet returns.

NOTE: some small experiments indicate that the order of events in Secretless is that there is always a source connection Read followed by a destination connection's Write.

TCP streaming throughput

The ongoing sum of the number of bytes written in any direction.

POC implementation

The POC implementation follows the ideas above.

The metrics are defined and labelled with connector specific information like the Secretless service name and connector type.

The connections are instrumented to allow measurement of latency and throughput, and measurements are taken via OpenTelemetry.

from secretless-broker.