• run Go checks and tests (#7)
• build custom collector images (#14)
• build hubble-otel image (#14)

It appears that #19 (2288e84) broke the encoding of nested objects. This needs to be fixed and there should be a proper test also.

This should work one based on http.* attributes.

It might be rather viable to implement a receiver interface and include it in the custom collector build and use OTel operator for deployment.

Trace ID are currently stored locally in an ephemeral store using Badger. It would be easy to enable persistence across restarts, but in oder to run hubble-otel on each node, there will need to be an alternative database option. It's also possible Badger might work on top of a volume, which maybe worse investigating. It'd be quite viable to use a hosted K/V store instead.

Spec

It would be nice to have an example of how to run this as just a deployment and not have to deal the operator

As HTTP responses actually come from Envoy, which runs in host netns, there are no pod labels of the original source endpoint.
This results in fallback service name being used.

• capture sample DNS & HTTP flows (#19)
• implement trace ID generation logic
• add sample to the test suite (#19)
• use trace IDs from HTTP header whenever present (#20)

• add prometheus metrics for buffer size and utilisation, overall throughput, error counts and badger ops
• add tracing to key functions

When I build this docker image and try to run it with hubble and jaeger I get this error message after a minute or so: "observed Hubble lost events". It works for like a minute and then crashes.

OpenTelemetry has some guidelines to how spans should be named:

• HTTP-specific
• general

At present Hubble CLI printer is used, which is quite verbose.

It should be possible to determine a link TCP spans by caching last span ID for each trace ID. One span can be linked to another between different traces.

Example: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/e08d7e3aec54a84ffca2978f0493ba4aca180132/receiver/jaegerreceiver/config.go#L40

The GH Action fails if run on a PR from an external fork, getting a 403 when trying to push the built container image. Would be better if it didn't attempt the push for an external fork

How do i set hubble.tls.enable to false?

Here is the code:

func main() {
flagsHubble := flags{
address: flag.String("hubble.address", "localhost:4245", "connect to Hubble on this address"),
tls: &flagsTLS{
enable: flag.Bool("hubble.tls.enable", false, "connect to Hubble using TLS"),
insecureSkipVerify: flag.Bool("hubble.tls.insecureSkipVerify", false, "disable TLS verification for Hubble"),
clientCertificate: flag.String("hubble.tls.clientCertificate", "", ""),
clientKey: flag.String("hubble.tls.clientKey", "", ""),
certificateAuthority: flag.String("hubble.tls.certificateAuthority", "", ""),
},
}

But i can't seem to set hubble.tls.enable. It complains that there's no flag.

Whenever an HTTP request includes trace headers, these should be used to set trace/span IDs.

• make trace database persisted to disk by default, with use-settable path
• let user pass common badger options, e.g. in-memory
• let user set trace TTL

I'm using hubble-otel to get hubble events and send to Jaeger. It works for a few minutes and then fails with the message, 'observed Hubble lost events'.

There are a of spans that are not very meaningful, in particular Cilium policy verdict: redirected stand out (as seen above). It probably makes sense to only export L7 spans by default, and leave low-level items as an opt-in for advanced users.

deleting

Currently L7 response latency is not useful, as it make a trace appear to be twice as long as it should be.

This depends on #1 & #2.

The TTL on trace ID should be updated when new flows belonging to the same connection are detected, so that window is extended for long lived connection that see ongoing activity.

For wider and easier adoption of Cilium/Hubble observability, this should be changed to be a receiver included in the OTel contrib collector package. I have opened a new component issue there and it has gotten several likes and views.
open-telemetry/opentelemetry-collector-contrib#12781

• daemonset option with unix socket
  • this is for GKE and cases where collector runs as a daemonset also
• deployment option with TLS

This depends on #1.