Comments (4)
Thinking about this some more and having implemented splitting traces on group_id in quicvis, I feel allowing group_ids across traces is still the best way to go.
If you're logging on multiple network intermediaries (so multiple vantage points) at once and then merge those logs, you will always have the same group_id split over multiple traces. However, each of those traces SHOULD then represent a different vantage_point. So the real restriction should be: cannot spread events from the same vantage point across different traces within the same qlog file. Put differently: each trace should contain only events from a single vantage point.
As such, we might rename group_id to flow_id instead, since that makes the semantics a bit clearer. That would say "this event belongs to flow with flow_id x, as observed from entity y".
I am trying to think of a reason why you would want to combine events from different vantage points into the same trace, but can't seem to find a use case. Either way, that would require changing up how we define vantage_point now, since it's per-trace and not part of common_fields or event_fields.
Any thoughts @nibanks?
from qlog.
Are there really people signed up to trace multiple different vantage points and put them all in the same file? I don't know about other companies, but getting logs from more than one machine all into the same file is practically impossible for the Windows scenario. Would it be so bad that the tools need to parse a file per vantage point?
I want qlog to succeed, but the more complicated it it, the less the chance I see that of becoming a reality. IMO, this is a place where simplicity should win.
from qlog.
-
About the use case of having multiple traces in 1 file, I'm not directly thinking of the Windows scenario or using this in production, but more about use cases such as research, education and case studies. There, it's handy to be able to group everything needed for a single "context" in a single file to be shared and interpreted easily. For production, automated gathering or aggregation in separate datastores is certainly also possible. The current setup does not prevent you loading separate files, 1 per vantage point, either.
-
I'm not sure how only allowing a single trace from a single vantage point per file would help the original problem that it's difficult for tool developers to deal with group_ids occurring across traces... client and server will still have events with the same group_id (e.g., ODCID)
As such, I'm not 100% sure what you're proposing? Do you want to do away with the "traces" array and replace it with a single "trace" per qlog file? Or do you want to do away with group_id at the "event_fields" level, requiring each individual trace to only contain events from a single connection?
In the latter case, I think that would actually be an obstacle to adoption, since currently several implementers are simply logging all events on the server in a single trace, tagged with ODCID for later splitting. This is generally much simpler than generating a single file (or trace) per connection on the server.
If you want to do away with group_id completely, that would leave out a whole bunch of other use cases, e.g., in-network observers like spindump (https://github.com/EricssonResearch/spindump, CC @jariarkko, @ihlar). This might be good enough for the QUIC use case (though barely), but not if qlog would grow to a more flexible format.
For draft-01, I've decided to keep the setup as-is, since there are users employing group_ids already (e.g., quant). I did specify the intended uses a bit more and am certainly open to more discussion on this design. Will you be in Singapore, @nibanks?
from qlog.
the group_id
field was not seeing much use in practice and as discussed above had some issues.
It has been considerably simplified for draft02, from f5db7cd onward.
from qlog.
Related Issues (20)
- H3Settings is a bit underspecified HOT 4
- packet_sent doesn't allow logging of IP address (for packets sent outside of a connection) HOT 4
- `ConnectionClose` storing a uint64 error_code and a uint64 error_code_value seems redundant HOT 4
- Split out ECN "capable" state HOT 1
- Difference between negotiated vs actually accepted/enabled TPs HOT 2
- Logging spin bit policy HOT 2
- Give concrete guidance on how to extend qlog
- Connectivity::Connection_closed application trigger type is confusing HOT 1
- Logging non-UTF-8 reasons HOT 2
- Logging of HTTP field names or values that contain illegal bytes
- Discuss extensibility of extension points through type or group sockets
- Make logging raw byte values consistent
- Move Trigger section underneath Data section
- Register application/qlog+json and application/qlog+json-seq media types
- protocol_type or protocol_types
- How is "trigger" a common subfield of "data"? HOT 1
- is parameters_restored missing fields defined in parameters_set?
- Extensions to events can not be indicated by schema
- JSON-SEQ is painful HOT 14
- parsing the `serialization_format` field requires prior knowledge of the serialization format HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from qlog.