Simplify / fix group_id usage about qlog HOT 4 CLOSED

Thinking about this some more and having implemented splitting traces on group_id in quicvis, I feel allowing group_ids across traces is still the best way to go.

If you're logging on multiple network intermediaries (so multiple vantage points) at once and then merge those logs, you will always have the same group_id split over multiple traces. However, each of those traces SHOULD then represent a different vantage_point. So the real restriction should be: cannot spread events from the same vantage point across different traces within the same qlog file. Put differently: each trace should contain only events from a single vantage point.

As such, we might rename group_id to flow_id instead, since that makes the semantics a bit clearer. That would say "this event belongs to flow with flow_id x, as observed from entity y".

I am trying to think of a reason why you would want to combine events from different vantage points into the same trace, but can't seem to find a use case. Either way, that would require changing up how we define vantage_point now, since it's per-trace and not part of common_fields or event_fields.

Any thoughts @nibanks?

from qlog.

Are there really people signed up to trace multiple different vantage points and put them all in the same file? I don't know about other companies, but getting logs from more than one machine all into the same file is practically impossible for the Windows scenario. Would it be so bad that the tools need to parse a file per vantage point?

I want qlog to succeed, but the more complicated it it, the less the chance I see that of becoming a reality. IMO, this is a place where simplicity should win.

from qlog.

1. About the use case of having multiple traces in 1 file, I'm not directly thinking of the Windows scenario or using this in production, but more about use cases such as research, education and case studies. There, it's handy to be able to group everything needed for a single "context" in a single file to be shared and interpreted easily. For production, automated gathering or aggregation in separate datastores is certainly also possible. The current setup does not prevent you loading separate files, 1 per vantage point, either.

2. I'm not sure how only allowing a single trace from a single vantage point per file would help the original problem that it's difficult for tool developers to deal with group_ids occurring across traces... client and server will still have events with the same group_id (e.g., ODCID)

As such, I'm not 100% sure what you're proposing? Do you want to do away with the "traces" array and replace it with a single "trace" per qlog file? Or do you want to do away with group_id at the "event_fields" level, requiring each individual trace to only contain events from a single connection?

In the latter case, I think that would actually be an obstacle to adoption, since currently several implementers are simply logging all events on the server in a single trace, tagged with ODCID for later splitting. This is generally much simpler than generating a single file (or trace) per connection on the server.

If you want to do away with group_id completely, that would leave out a whole bunch of other use cases, e.g., in-network observers like spindump (https://github.com/EricssonResearch/spindump, CC @jariarkko, @ihlar). This might be good enough for the QUIC use case (though barely), but not if qlog would grow to a more flexible format.

For draft-01, I've decided to keep the setup as-is, since there are users employing group_ids already (e.g., quant). I did specify the intended uses a bit more and am certainly open to more discussion on this design. Will you be in Singapore, @nibanks?

from qlog.

the group_id field was not seeing much use in practice and as discussed above had some issues.

It has been considerably simplified for draft02, from f5db7cd onward.

from qlog.