[cbor] Support writing Map (Object) with finite length about jackson-dataformats-binary HOT 17 OPEN

Thanks @cowtowncoder , actually, I've found out by reading the source code that the method to write a Map with a fixed size exists: public final void writeStartObject(int elementsToWrite)
So, I only have to cast jgen into a CBORGenerator and it works like a charm!
((CBORGenerator) jgen).writeStartObject(1);

from jackson-dataformats-binary.

(modified title as #1 already covers case of known-length arrays)

from jackson-dataformats-binary.

I'm facing the same issue, which lead to have some unnecessary additional bytes to the serialized data.
Example and data in SOF:
https://stackoverflow.com/questions/48991682/serialize-fixed-size-map-to-cbor

from jackson-dataformats-binary.

As per example, usage would be via Streaming API (generator), and supporting that should be a bit easier than general-purpose databind. At databind, Maps could perhaps be supported as well.

But one big problem is that there is no applicable method use: writeStartObject() would require explicit length indicator: otherwise buffering would need to be used to calculate entry count, then encode (since indicator is variable-length).
And this can not be done in a patch (API change).
So most likely this is 3.0 only at this point.

from jackson-dataformats-binary.

Thanks @cowtowncoder. Do you see any possible workaround at this point?

from jackson-dataformats-binary.

Not really. You can by-pass encoder with writeRaw / writeRawValue but that pretty much defeats the purpose.

from jackson-dataformats-binary.

Ah. Well spotted. The reason I missed that was that I had thought of adding method in JsonGenerator, but hadn't because of being unsure if and how to combine with possible argument of forObject. Will probably add combined in 3.0.
But it is good this is available for CBOR already, and that it actually works as expected -- keeping track of elements actually written is implemented to safeguard against invalid usage.

from jackson-dataformats-binary.

I'm having a problem with this feature.
The case is that I have a complicated/nested String. First, I can convert it to JsonNode, and later, use an ObjectMapper to convert JsonNode to CBOR byte[].
In the results byte[], the array size is finite, but the map size is not.
I haven't found any workaround by now. But for general purpose, I think you should put a flag in SerializationFeature like WRITE_FIXED_SIZE_OBJECT, so that it is posible to write a complicated json string to CBOR with sized map to remove extra '-1' bytes

from jackson-dataformats-binary.

@quannh0308 I don't think this can work as general SerializationFeature as it is CBOR-specific, unfortunately: tracking of sizes for all uses complicates code and adds overhead, primarily because then potentially full output hierarchy would need to be buffered.

But I think it will be possible to add necessary method in JsonGenerator for 2.10:

https://github.com/FasterXML/jackson-core/issues/517

and then eventually make ObjectNode and Map serializers use it: in these case we do know actual count of entries without buffering, and can encode things efficiently.

What will not be easy or perhaps even possible, unfortunately, will be doing the same for POJOs: problem is that things like @JsonAnyGetter and various filtering methods will affect number of properties actually written.

But I think that ability to at least force this via Tree Model (it is, after all, always possible to convert from POJO to Tree Model, then serialize that) would make this possible in general case and by users who really need to force length prefix.

I do think however that requiring use of length-prefix is a bad idea and implementations should not require it.

from jackson-dataformats-binary.

Hi cowtowncoder, I wanted to use finite length Object serialization through ObjectMapper. I m doing

        final CBORFactory cborFactory = new CBORFactory();
        mapper = new ObjectMapper(cborFactory);

What I understood from your conversations is, the support is added since 3.0. But it looks like it hasn't been made in Jackson-databind package which means I cannot use ObjectMapper to get the finite length support. Can you comment on it.

And second question is, do you think there should be a feature or flag to enable disable this functionality instead of making own assumption? If flag needs to be added, where will be added. Would like to know next steps on supporting finite length maps. Thanks for your hardwork on supporting the feature in 3.0.

from jackson-dataformats-binary.

@rangaataws 3.0 has not been released, and will not yet for quite some time. Current releases are 2.x, latest being 2.10.0.
But since underlying write method has been added, it should actually be possible to add support before 3.x, perhaps in 2.11, as 2.10 added this method in JsonGenerator API:

public void writeStartObject(Object forValue, int size) throws IOException;

so that databind could start calling it in 2.11.

Now... as to feature to add... perhaps. I think the way it would work is simply that databind would call method that passes length, when known -- it is not always practical to try to calculate it, for example when entries are filtered -- and then dataformat would decide to use it (or not).
So that would be CBORGenerator.Feature.ADD_MAP_LENGTH_PREFIX or something like that.

As to next steps: to be completely honest, I don't think I will have time to work on this in near future.
If anyone else wants to work on it that would be great. Changes should go in 2.11 branch.

from jackson-dataformats-binary.

Thanks for the quick response. If it's a lot of work, then I may use CborGenerator directly instead of using it via databind (although I like databind because it exposes clean interface by hiding internal implementation).

I was looking at the 2.10 version of JsonGenerator where the method is implemented like below. I don't see size is used for writing the fixed length of the object. Maybe I m missing something.

public void writeStartObject(Object forValue, int size) throws IOException
    {
        writeStartObject();
        setCurrentValue(forValue);
    }

Unrelated to this, do you think fixed length serialization adds value? I think the providing size may be helpful for languages like C where we have to allocate the memory.

from jackson-dataformats-binary.

Right, that part is missing. Just needs to be wired. That would then go in 2.11, along with unit tests to ensure working operation. This issue is still open and not implemented.

But aside from encoding length-prefix, generator will also need to keep track that exactly correct number of values will be written, to avoid producing corrupt content (claiming to write N entries, writing some other number). I think existing Array prefix handling code could show guidance.

from jackson-dataformats-binary.

Thanks. In short,the latest released (2.10) version doesn't have support for fixed length which means I have to wait for 2.11 :) - Any estimated release date you know of ?

reg. length prefix, you are right. Calculating the number of keys in the object is extra logic that could go wrong due to access specifiers and suppressable annotations what not. On the other hand, if customer needs to write the logic, they would have to do the same so I feel the size calculation logic can go into Generator to simplify customer serialization logic. Thanks.

from jackson-dataformats-binary.

On calculating: what I mean is both databind side (which like you said can be complex / impractical), but also format (generator) side which must keep track of real value writes, not just what was alleged on writeStartObject().

As to 2.11: plan to get first release candidates by Dec 2019, release Jan 2020.
Still, someone needs to work on this format module. I don't think work would be superbly difficult, but it is not trivial enough to "just go do it".

from jackson-dataformats-binary.

Any updates on this? For now I simply manually write through generator to ease the pain, but I see no reason why serializing a simple Map (of Maps) should not be able to add lengths.

from jackson-dataformats-binary.

Contributions welcome!

As to why it is not trivial to implement: Jackson generators are by default fully streaming, incremental, which means that length-prefix use requires separate and additional, possibly nested buffering; adding both overhead and complexity.
So while it can simply decoding/parsing part, it tends to add complexity on encoding/generation side.
Nothing insurmountable of course (as this has been implemented for arrays, and for prefix-mandating format backends like protobuf), just not trivial to "just do it".

from jackson-dataformats-binary.