Invariants for IPLD Codecs about specs HOT 3 OPEN

I think there's some good idea here, but also we should be cautious of overgeneralizing this.

Not all IPLD codecs have an "underlying" codec. For those that do: our specs will still be clearest if we specify what the IPLD codec does first; and specify what relationship this has to any other codecs in the wild second. And in all forms of interop: the practical details matter; and what various tools and libraries do in the wild can be just as interesting as what a spec or a reference implementation says, especially if those things diverge.

I think our codecs specs and documentation should be individually clear, for each codec, about any other widely-known systems they expect to be cross-compatible with, and how, and any conditions and limits there may be on that.

(And we need these detailed statements anyway: they're the other half of fully specifying any increasing strictness an IPLD codec might have versus a general understanding of that codec.)

Having the holistic goal of interop in mind for those codecs which do aim to have interoperability with existing systems is... good? But also tautological. If there's some phrasing of this that will help us write detailed interoperability reports per codec, I'm all for it; I'm just not sure what kind of statement that would be and what kind of explicitness it can really have that will be useful.

from specs.

I don't think I made my point clear enough. I don't want to oppose additional constraints on codecs, I'd like to document on the current state.

I think what I describe is how people currently understand, implement and use IPLD Codecs. I want to make sure our specs are precise and remove the chance for misinterpretation.

@warpfork would it help if more people comment/emoji here whether what I describe here matches their expectations about IPLD Codecs or not? I ask, because I observe a disconnect between the IPLD Team and the outside world. And I obviously also only have a limited view of the outside world. So it might help to get people from various backgrounds to chime in here.

from specs.

Here's a quick update after talking to various folks (thanks everyone!).

The byte-identical copy of the original data is not possible to enforce. Spec compliant implementations won't be strict enough. Examples:

• @aschmahmann mentioed that Protocol Buffers might not always get serialized to the same bytes even if the input data is the same
• @ribasushi mentioned float encoding in CBOR. The CBOR spec has some recommendation that you could use the smallest representation possible. I.e. if you have a 64-bit IEE-754 float that can be represented lossless in an 32-bit IEEE-754 float, then use a 32-bit float (example of such a float). Though implementations can just not do it (we plan for DAG-CBOR to always require 64-bit floats, which is what the Go implementation is already doing). So there could be two implementations encoding it differently, while still both being spec compliant.

Hence I propose a less strict version:

In case there is an underlying codec, the data produced by an IPLD Codec MUST be decodable by any spec compliant/reference implementation of the underlying codec.

"underlying codec" means a existing codec we apply additional constraints on. Examples are DAG-CBOR and DAG-JSON, where the underlying codecs would be CBOR and JSON.

We will likely get into SHOULD vs. MUST discussions. The reasons I'm in favour of" MUST":

So far all cases I've seen where the condition above wasn't met were bugs. Of course there are bugs that lead to data that doesn't comply to a spec (hence violate the MUST), that's the nature of bugs. I don't think those bugs should weaken the spec itself. The reasons those bugs exists is due to the spec not being precise enough. It's now the chance to add this precision. I don't think it should be a "SHOULD", just because there could be bugs.

This doesn't mean that I want to break all the data that was produced due to the bugs. I think it's totally fair to create libraries that can deal with such data. They are free to do so and even should if they care about backwards compatibility with pre-existing data. Though the MUST not produce such invalid data moving forward.

"producing data" is a bit fuzzy and I'd like that we operate with common sense here and not putting too much efforts on tighten the spec language. So if your library deals with invalid data, it might as well write it again. What I'm after is that we try to make those bugs less likely in the future.

This also (to me) does not mean that you always need to ensure that the data you write is valid. You just need to acknowledge that your data you produce might be invalid and might not work with other implementations. This would be similar to what Google is doing with Protocol Buffers. In Go you can produce invalid data (strings with arbitrary bytes), but you won't be able to read that with the Python implementation. Or if you store arbitrary bytes in strings in the Protocol Buffers JavaScript implementation (without doing any special additional work), they will be serialized differently from the ones that the Go implementation would serialize (although the original input bytes were the same).

from specs.