Is there an existing issue for this?

• I have searched the existing issues

Description

I am trying to do something similar to what is explained in this Python CBOR library:
https://cbor2.readthedocs.io/en/latest/customizing.html#using-the-cbor-tags-for-custom-types

I want to be able to encode and decode a custom struct using a custom tag.

I understand that this is possible, by looking at Value, however, it would be very useful to have a simple example explaining how to do it.

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

Is there an existing issue for this?

• I have searched the existing issues

Description

Right now, all the functions support Read and Write traits. When using async Rust, it would probably be better to support AsyncRead and AsyncWrite. I wonder if that's something you'll be willing to accept.

We can obviously feature flag this, and make it experimental at first. But I wanted to know what you think.

Acceptance Criteria

• Supporting AsyncRead implementations in ciborium
• Supporting AsyncRead in ciborium-ll, or even Stream?

Suggestions for a technical implementation

Having a future feature flag to not add burden of current apps that don't need async support.

Value itself is canonical for serialization, but can't be used in a map anymore after the changes in 0f00eae.

One way to improve the situation would be to have a new type, let's call it CanonicalValue that can be used as a key in maps, and sorts according to canonical sorting rules. This way we don't prevent Value from being "pure", but also allow people to canonicalize CBOR using a simple (pseudo-rust) into_writer(BTreeMap<CanonicalValue, Value>::deserialize(bytes))) line.

Is there an existing issue for this?

• I have searched the existing issues

Description

Both serde_json and serde_cbor have very nice top-level documentation for their crate, showing all the common usage modes. In comparison, ciborium says at the top level:

You’re probably looking for from_reader() and into_writer(), which are the main functions. Note that byte slices are also readers and writers and can be passed to these functions just as streams can.

For dynamic CBOR value creation/inspection, see Value.

And then peeking into from_reader shows just:

Deserializes as CBOR from a type with impl ciborium_io::Read

There are no examples on what all that trait is implemented on, or in general common ways to use the library.

I believe a similar top-level documentation to serde_json and serde_cbor should also be written for ciborium and I believe it could boost the usage of the crate.

Acceptance Criteria

No response

Suggestions for a technical implementation

I may be able to provide a pull for this, if I get encouragement to do so.

Is there an existing issue for this?

• I have searched the existing issues

Description

When decoding CBOR data that contains unassigned simple values, as in those that aren't of false, true, null or undefined, an error is thrown:

A minimal repro was posted by @Anrock at anweiss/cddl#90 (comment).

It would be great to be able to capture these values so that CDDL tools can validate these values.

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

Is there an existing issue for this?

• I have searched the existing issues

Description

The readme offers this rationale:

This decision was made because this type preserves the order of the pairs on the wire. Further, for those that need the properties of BTreeMap or HashMap, you can simply collect() the values into the respective type. This provides maximum flexibility.

I believe using IndexMap could be an even better compromise:

• Order of pairs on the wire is preserved
• IndexMap performance is quite good, on par with HashMap for many uses
• Looking up a key from an object does not require collecting to Map first, or linear traversal through all entries

Currently, using Value for any traversal is really tedious, and really expensive if there are many map lookups, reducing it's usability significantly.

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

Serde.rs still points to the serde_cbor. Based on the maintenance ticket in the other repo this crate seems to be a successor. If it is so and this crate is stable please request an update on the serde page.
At the moment due to the lack of "official" link and relatively low commit frequency i'm unsure of the stability/long term support of this crate and hence afraid of switching (from bincode)

First of all: Thanks for making this crate! I know there is a lot of demand for a good CBOR crate in the Rust community, I hope this crate will work for many use-cases.

Deserialization and serialization are prone to logic errors resulting in crashes, from serde_cbor I know that it can be useful to fuzz the crate to detect such errors.

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

Deserialization appears to use unusually large amounts of memory.

Expected Behaviour

Deserialization shouldn't use more than 2 times the amount of serialization (especially in case both use a Vec buffer, and no streaming).

Environment Information

The following commit reduces the memory usage of an example run of example-bootstrap + yzix-server, it replaces ciborium with bincode. YZITE/yzix@e7b995d
(the server is the interesting part, the deserialization of an somewhat larger than 118MiB CBOR block causes allocations up to ~3GiB, which is unexpected; occurs at the point which serializes a linux kernel tarball inside a Dump::Regular object)

Steps To Reproduce

Should be possible by extracting the Dump enum definition, and filling an instance of it with multiple megabytes of data, then serailizing it (probably already efficient enough) and then deserializing it again (which causes unexpected high memory usage).
Not done yet, and might be exacerbated by a suboptimal usage in yzix_proto::codec.

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

The following program crashes:

use ciborium::cbor;

fn main() {
    let data = cbor!({
        "type" => "thing2",
        "name" => "somename",
        "data" => ciborium::tag::Required::<Vec<u8>, 64>(vec![1, 2, 3, 4]),
    }).unwrap();
    let mut bytes = Vec::new();
    ciborium::ser::into_writer(&data, &mut bytes).unwrap();

    let _thing: DistinctThings = ciborium::de::from_reader(bytes.as_slice()).unwrap();
}

#[derive(serde::Deserialize)]
#[serde(tag = "type")]
enum DistinctThings {
    #[serde(rename = "thing1")]
    Thing1 {
        #[serde(rename = "name")]  
        some_name: String,
    },
    #[serde(rename = "thing2")]
    Thing2 {
        #[serde(rename = "name")]  
        some_name: String,
    }
}

With the error:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Semantic(None, "untagged and internally tagged enums do not support enum input")', src/main.rs:12:78
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Expected Behaviour

The DistinctThings enum should be succesfully deserialized.

Environment Information

Linux manjaro-desktop 5.15.85-1-MANJARO #1 SMP PREEMPT Wed Dec 21 21:15:06 UTC 2022 x86_64 GNU/Linux

Steps To Reproduce

No response

When using negative numbers in Value::from() I get off by one (e.g. Value::from(-3) serializes to the bytes 22 which is negative 2). I think that's because this line is wrong; https://github.com/enarx/ciborium/blob/main/ciborium/src/value/integer.rs#L48. We don't want the bit-swap, we want the complement by 2 of the number.

It also seems the same logic is applied to BIGNEG numbers.

If you want to keep representing negative zeros as part of this API, you should make it a special constructor on Integer (e.g. Integer::negative_zero(). But serializing Value::from(-111) should give me the expected byte value (negative 111).

I saw that you convert CBOR bigints to 128-bit integers in Rust. In case the number exceeds 2^128-1, you raise an error. I don't think this is the right approach: For most common tasks 64-bit numbers are more than big enough, but for tasks that use big integers the numbers can easily exceed 128 bits, for example in cryptography.

Is there an existing issue for this?

• I have searched the existing issues

Description

In -ll, when creating an Encoder::from a &mut [u8], it seems to be impossible to recover the cursor position. Items are pushed into the slice, and the slice is advanced internally, but there is no way from the outside to determine how much was actually written. This severely limits the usefulness of having &mut [u8] encoders to situations where the length written is known exactly.

Acceptance Criteria

No response

Suggestions for a technical implementation

There should not be a need to track the number of bytes written itself. It would suffice if the encoder could be dismantled (eg. .into()'d to its writer); then, the difference in .len() of the slices can be used to determine what was written.

Consider these two structs:
#[derive(Serialize, Deserialize, Clone, Debug)] pub enum NumberEnum { num(u128), stringy(String), nuffing, }
#[derive(Serialize, Deserialize, Clone, Debug)] pub struct SimpleStructBackend3 { pub number: u16, pub string: String, pub numbenum: NumberEnum, pub uuid: Uuid, }

Initialising a SimpleStructBackend3 with numbenum = NumberEnum::num(42) or numberenum = NumberEnum::stringy(String::from("forty-two") is fine. Initialising it with numbenum = NumberEnum::nuffing causes uuid to fail to deserialise. No problems found if numbenum is the final field in SimpleStructBackend3. For some reason, choosing a untyped Enum value causes the next field in the struct (uuid, in this case) to have the mismatch Title = Other, Major = Text, which causes a panic.

Open the crate's home page on docs.rs and click on the first code link (de::from_reader()).

Opening this page fails as follows (Safari, macOS Monterey 12.1):

Safari can't open the specified address.

Safari can't open "crate::de::from_reader" because macOS doesn't recognize Internet addresses starting with "crate:".

(Note: the raw Markdown contains a zero-width-space so as to prevent the rendered markdown from having a German flag.)

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

This code :

#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct S {
    pubKey: [u8; 32] // or Vec<u8>, it produce the same error
}


fn main() {
    let data = vec![161, 102, 112, 117, 98, 75, 101, 121, 88, 32, 189, 182, 166, 93, 174, 37, 6, 247, 39, 203, 228, 101, 121, 197, 203, 42, 98, 138, 145, 12, 12, 76, 145, 168, 132, 185, 90, 18, 54, 28, 248, 170];
    let r = ciborium::de::from_reader::<S, _>(&data[..]);
    println!("{:?}", r);
}

Produce this error :

Err(Semantic(None, "invalid type: bytes, expected array"))

Expected Behaviour

Decoding bytes into a Vec<u8> or [u8; N] would be nice

Environment Information

Cargo.toml:

[package]
name = "z"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1", features = ["derive"] }
ciborium = "0.2.0"

Steps To Reproduce

No response

Is there an existing issue for this?

• I have searched the existing issues

Description

Please publish a new release of ciborium so we can avail of the changes merged in #34

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

Hi!

Good to see some action in rust cbor space! Thank you for your efforts.

I have a question regarding the attempt to use the smallest possible datatype to represent the floating point value. For integer values, I'm 100% convinced this is a good idea. For floating point values, I have my doubts, let me list my thoughts:

• floating point is always tricky, there is nan, inf, +inf. There exists signalling NaN and quiet NaN. Is this all handled correctly when shrinking to a smaller floating point type?
• Is the automatic shrinking compatible with other languages? Does the shrinking of the floating point value, based upon it's value, play nice with other languages which use cbor? How do other implementations of cbor deserialization react when a field is sometimes encoded as f64 and sometimes as f32?

I tried to reproduce the issue I faced, I'll give it another shot. It's still located here: pyfisch/cbor#169

Serialising and deserialising Structs works fine, but when trying to do the same with a Vec of them, we get the following error with the test case below:

[mike@nail test-and-try]$ cargo run --bin ciborium
   Compiling test-and-try v0.1.0 (/home/mike/programming/test-and-try)
    Finished dev [unoptimized + debuginfo] target(s) in 0.51s
     Running `target/debug/ciborium`
incoming string = onetwothree
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Semantic(Some(0), "expected array")', src/ciborium.rs:36:66
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

ciborium-test.rs.txt

Is there an existing issue for this?

• I have searched the existing issues

Description

I often want to encode a bunch of identically-shaped objects like

{"name": "raylu", "id": 1}
{"name": "bob", "id": 2}

packed encoding (#93) would help a lot, but
https://github.com/kriszyp/cbor-x#record--object-structures supports "records", which seems even better

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

Is there an existing issue for this?

• I have searched the existing issues

Description

When working in no_std environments, a heapless::Vec is a popular target to write into: it provides a pre-allocated buffer, but unlike a standard Vec (even with capacity), it is bound in size and thus suitable for stack allocation.

Currently, users need to implement this through a newtype due to the Orphan rule, but it'd be great if one of the two crates that can implement it (ciborium-io and heapless) did, and I think heapless is the more basic crate (so ciborium-io should do it).

Acceptance Criteria

No response

Suggestions for a technical implementation

As far as versions, compatibility and features are concerned, this would in its simplest form target the latest heapless version (there has been a stable 0.7 since const generics landed). This would be an optional, non-default dependency.

I think it'd be prudent to re-export the heapless that's being used, so that users can be sure to use the version of heapless that is actually impl'd.

If heapless does release an incompatible version, it should be possible to impl all of them by depending on different versions of the heapless crate; I think we can cross that bridge when we get there. (Also, who knows, maybe there's a breaking version of ciborium pending at that time anyway, and we can just lockstep).

As an alternative to whack-a-mole'ing heapless (and any other similar crates where a similar request might be done, both in this and other CBOR implementation crates), it might be an option to just go through embedded-io.

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

I am getting the error Semantic(None, "invalid type: bytes, expected bytes")

Expected Behaviour

deserialization works

Environment Information

Linux 6.5.4-76060504-generic #202309191142169599894322.04~070916d SMP PREEMPT_DYNAMIC Fri S x86_64 x86_64 x86_64 GNU/Linux

Steps To Reproduce

The error is launched from here

I can reproduce the error in a downstream crate with a type implementing custom serde Serialize/Deserialize: ElementsProject/rust-elements@45bd2bf
This type is longer than 4k bytes

When I try to reproduce without the downstream crate using something like Wrap(vec) where vec is bigger than 4k bytes, things work because in this line:

the scratch space has grown to accommodate the len of the bytes, (it is 8k even though it is initialized to 4k) while in the case of the error it's probably not growing for some reason I could not grasp yet.

Is there an existing issue for this?

• I have searched the existing issues

Description

I just noticed, that your MapAccess implementor here does not implement size_hint, but it seems that should be pretty easy to implement:

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

The following test fails with

'called `Result::unwrap()` on an `Err` value: Custom("invalid type: integer `42`, expected unit")'

#[test]
fn serde_other_value() {
    #[derive(Deserialize)]
    enum E {
        A(u32),
        #[serde(other)]
        Unknown,
    }

    #[derive(Serialize)]
    enum F {
        A(u32),
        B(u64),
        #[serde(other)]
        Unknown,
    }

    let mut buf = vec![];
    let b = F::B(42);
    ciborium::ser::into_writer(&b, Cursor::new(&mut buf)).unwrap();
    let unk_from_reader: E = ciborium::de::from_reader(Cursor::new(&buf)).unwrap();
    // It works when deserializing directly from the reader
    assert!(matches!(unk_from_reader, E::Unknown));

    let v: ciborium::Value = ciborium::de::from_reader(Cursor::new(&buf)).unwrap();
    let unk_from_value: E = v.deserialized().unwrap();
    // This is the failing assertion
    assert!(matches!(unk_from_value, E::Unknown));
}

Expected Behaviour

I expect the Unknown variant to be deserialized regardless of whether there's an intermediate Value step.

Environment Information

N/A

Steps To Reproduce

No response

It would be great if we could have some benchmarks as well as overall optimization for speed.

When I tried deserializing a structure with borrowed strings, an error was returned, stating that an owned string was not expected. It would be great to support zero-copy deserializing.

Is there an existing issue for this?

• I have searched the existing issues

Description

Pretty self-explanatory: the provided functions are helpful, but I'm writing a library that extends various Serializers and Deserializers, and so I'm requesting that those types be made public 🙏

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

serde_cbor and serde_json provide StreamDeserializer, which is quite useful for some journal file.

I have a custom implementation of Deserialize that works with serde_cbor, serde_json, and others.

In my visit_seq function at the beginning I get:

[src/primary.rs:91] seq.size_hint() = Some(
    8,
)

I then parse a few fields with .next_element() and later on check again:

[src/primary.rs:118] seq.size_hint() = Some(
    8,
)

The number stays the same no matter how often I get a next element.

Usually, the number should decrease, or am I missing something?

I just learned about this project after I asked the same question on serde_cbor. RFC 8949 section 3.4.4 specifies BigFloat. What do you think of implementing this using rust-decimal? If not, since this crate does support BigNum via i128, can I use serde(with = "") with this crate to scale/unstable them myself?

ciborium fails to serialise a Struct containing a uuid::Uuid. The attached Rust file fails with the following output:

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/ciborium`
incoming string = onetwothree
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Semantic(Some(0), "expected bytes")', src/ciborium.rs:41:65
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

serde_cbor handles this fine.
uuid::Uuid does seem to implement both the Serialize and Deserialize traits.

Cargo.toml.txt
ciborium.rs.txt

Is there an existing issue for this?

• I have searched the existing issues

Description

Looking to get packed encoding support added. I use CBOR to save cached OCR data

the file is around 8.6mb which is around 50% smaller than the JSON equivalent. Reading the cbor file in a hex editor shows a lot of stuff are repeated like field names/certain fields that repeat the same value over and over.

Packed encoding would be wonderful 😄

Acceptance Criteria

• Works
• Smaller file size

Suggestions for a technical implementation

No response

Is there an existing issue for this?

• I have searched the existing issues

Description

CDDL is a concise data definition language.

Acceptance Criteria

It should be possible to generate rust objects parsed from cbor defined by a cddl schema.

Suggestions for a technical implementation

zcbor project implements cddl generation for C code.

Is there an existing issue for this?

• I have searched the existing issues

Description

The CBOR format would be useful in persistently storing data described in JSON form. So it would be handy to be able to generally bidirectional convert between JSON structures and CBOR structures.

Acceptance Criteria

No response

Suggestions for a technical implementation

The code below is a simple and straightforward implementation in my own project, without consideration in specification-compliant of number processing and even error handling, let alone an Error enum:

use serde_json::{Value as JsonValue, value::Number as JsonNumber};
use ciborium::value::{Value as CborValue, Integer as CborInt};

fn cbor_into_string(cbor: CborValue) -> Option<String> {
    match cbor {
        CborValue::Text(string) => Some(string),
        _ => None,
    }
}

fn cbor_to_json(cbor: CborValue) -> JsonValue {
    match cbor {
        CborValue::Null => JsonValue::Null,
        CborValue::Bool(boolean) => JsonValue::Bool(boolean),
        CborValue::Text(string) => JsonValue::String(string),
        CborValue::Integer(int) => JsonValue::Number({
            let int: i128 = int.into();
            if let Ok(int) = u64::try_from(int) {
                JsonNumber::from(int)
            } else if let Ok(int) = i64::try_from(int) {
                JsonNumber::from(int)
            } else {
                JsonNumber::from_f64(int as f64).unwrap()
            }
        }),
        CborValue::Float(float) => JsonValue::Number(JsonNumber::from_f64(float).unwrap()),
        CborValue::Array(vec) => JsonValue::Array(vec.into_iter().map(cbor_to_json).collect()),
        CborValue::Map(map) => JsonValue::Object(map.into_iter().map(|(k, v)| (cbor_into_string(k).unwrap(), cbor_to_json(v))).collect()),
        CborValue::Bytes(_) | CborValue::Tag(_, _) => unimplemented!(),
        _ => unimplemented!(),
    }
}

fn json_to_cbor(json: JsonValue) -> CborValue {
    match json {
        JsonValue::Null => CborValue::Null,
        JsonValue::Bool(boolean) => CborValue::Bool(boolean),
        JsonValue::String(string) => CborValue::Text(string),
        JsonValue::Number(number) => {
            if let Some(number) = number.as_u64() {
                CborValue::Integer(CborInt::from(number))
            } else if let Some(number) = number.as_i64() {
                CborValue::Integer(CborInt::from(number))
            } else if let Some(number) = number.as_f64() {
                CborValue::Float(number)
            } else {
                unreachable!()
            }
        },
        JsonValue::Array(vec) => CborValue::Array(vec.into_iter().map(json_to_cbor).collect()),
        JsonValue::Object(map) => CborValue::Map(map.into_iter().map(|(k, v)| (CborValue::Text(k), json_to_cbor(v))).collect()),
    }
}

I hope that there will be a specification-compliant and robust bidirectional conversion implemented in this crate. Considering that it introduces a dependency on serde_json crate, it could be provided as an optional feature, with a separate Error enum.

Is there an existing issue for this?

• I have searched the existing issues

Description

Hey everyone!

I have a use-case where I am deserializing a recursive data structure. The data I'm deserializing can become quite deep, to the point where I'm running into the RecursionLimitExceeded errors when deserializing.

I'm aware there's a risk in increasing the recursion limit of 256. In my particular case, increasing that limit would solve my issue.

Acceptance Criteria

No response

Suggestions for a technical implementation

I can see a few solutions to address this:

1. Add a new method with the following signature to deserialize with a specific recursion limit:

pub fn from_reader_with_recursion_limit<T: de::DeserializeOwned, R: Read>(reader: R, recurse: usize) -> Result<T, Error<R::Error>

2. Add a feature, say no_recursion_limit, and if this feature is enabled, the recurse value is set to usize::MAX, effectively disabling the recursion limit check.

3. Increase the hard-coded value of recurse from 256 to, say, 512.

Thoughts? I'm happy to create a PR once we agree on an approach.

Is there an existing issue for this?

• I have searched the existing issues

Description

I end up using ciborium via the cddl crate, for which I am maintaining a Python wrapper: https://gitlab.com/tahoe-lafs/pycddl.

The CDDL validation code ends up being used to validate potentially large (megabytes, hundreds of megabytes) CBOR documents. Parsing the CBOR is a reasonable chunk of CPU time for CDDL validation, and CDDL validation is noticeable chunk of CPU time for handling requests in end-user application. A larger buffer size shrinks this CPU overhead quite a bit.

Two examples, parsing a CBOR-encoded bytestring:

Acceptance Criteria

I imagine current scratch is small for use in restricted memory contexts, and uses no allocation to support executions context without an allocator. So that can't go away.

On the other hand, anything using Python, not to mention plenty of Rust applications, would be perfectly fine with a 64KB buffer in return for faster parsing of documents (though that shouldn't be on the stack probably).

Suggestions for a technical implementation

Some options:

1. Make Deserializer public. This is a low-level option, and increases API surface.
2. Accept an optional scratch buffer argument to from_reader. This would be backwards incompatible.
3. New public API, from_reader_with_buffer(). This would be backwards compatible.

[[package]]
name = "ciborium"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b0c137568cc60b904a7724001b35ce2630fd00d5d84805fbb608ab89509d788f"
dependencies = [
 "ciborium-io",
 "ciborium-ll",
 "serde",
]

[[package]]
name = "ciborium-io"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "346de753af073cc87b52b2083a506b38ac176a44cfb05497b622e27be899b369"

[[package]]
name = "ciborium-ll"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "213030a2b5a4e0c0892b6652260cf6ccac84827b83a85a534e178e3906c4cf1b"
dependencies = [
 "ciborium-io",
 "half",
]

[[package]]
name = "serde"
version = "1.0.130"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f12d06de37cf59146fbdecab66aa99f9fe4f78722e3607577a5375d66bd0c913"
dependencies = [
 "serde_derive",
]

I have the following code in a project (https://git.ytrizja.de/zseri/yzix/src/commit/28e4990ada02b09141cdf4080d89818f55bd4630/crates/yzix-core/src/proto.rs):

#[derive(Clone, Debug, Deserialize, Serialize)]
#[serde(rename_all = "lowercase", tag = "type")]
pub enum ControlCommand {
    /// schedule a bunch of commands
    Schedule {
        graph: Graph<()>,

        /// if set to false, no logs will be sent or kept
        /// about your submitted build graph
        attach_to_logs: bool,
    },
}

when I omit the serde rename+tag line, serialization+deserialization works fine, but if it is present, it fails with Semantic(None, "invalid type: Option value, expected unit").
Option value stems from https://docs.rs/serde/1.0.130/src/serde/de/mod.rs.html#406
unit stems from https://docs.rs/ciborium/latest/src/ciborium/de/mod.rs.html#497

this is seemingly impossible, because in https://docs.rs/ciborium/0.2.0/src/ciborium/de/mod.rs.html#21-45, no Unexpected::Option is ever created.

cbor as byte vector:

[163, 100, 116, 121, 112, 101, 104, 115, 99, 104, 101, 100, 117, 108, 101, 101, 103, 114, 97, 112, 104, 164, 101, 110, 111, 100, 101, 115, 130, 164, 100, 110, 97, 109, 101, 98, 104, 105, 100, 107, 105, 110, 100, 162, 100, 116, 121, 112, 101, 102, 117, 110, 100, 117, 109, 112, 99, 100, 97, 116, 163, 100, 116, 121, 112, 101, 103, 114, 101, 103, 117, 108, 97, 114, 106, 101, 120, 101, 99, 117, 116, 97, 98, 108, 101, 245, 104, 99, 111, 110, 116, 101, 110, 116, 115, 135, 24, 101, 24, 99, 24, 104, 24, 111, 24, 32, 24, 72, 24, 105, 102, 108, 111, 103, 116, 97, 103, 1, 100, 114, 101, 115, 116, 246, 164, 100, 110, 97, 109, 101, 102, 117, 115, 101, 32, 104, 105, 100, 107, 105, 110, 100, 163, 100, 116, 121, 112, 101, 99, 114, 117, 110, 103, 99, 111, 109, 109, 97, 110, 100, 130, 129, 161, 102, 83, 116, 114, 105, 110, 103, 100, 98, 97, 115, 104, 129, 161, 107, 80, 108, 97, 99, 101, 104, 111, 108, 100, 101, 114, 101, 104, 105, 105, 110, 112, 100, 101, 110, 118, 115, 160, 102, 108, 111, 103, 116, 97, 103, 2, 100, 114, 101, 115, 116, 246, 106, 110, 111, 100, 101, 95, 104, 111, 108, 101, 115, 128, 109, 101, 100, 103, 101, 95, 112, 114, 111, 112, 101, 114, 116, 121, 104, 100, 105, 114, 101, 99, 116, 101, 100, 101, 101, 100, 103, 101, 115, 129, 131, 1, 0, 161, 107, 80, 108, 97, 99, 101, 104, 111, 108, 100, 101, 114, 101, 104, 105, 105, 110, 112, 110, 97, 116, 116, 97, 99, 104, 95, 116, 111, 95, 108, 111, 103, 115, 245]

Map(
    [
        (
            Text(
                "type",
            ),
            Text(
                "schedule",
            ),
        ),
        (
            Text(
                "graph",
            ),
            Map(
                [
                    (
                        Text(
                            "nodes",
                        ),
                        Array(
                            [
                                Map(
                                    [
                                        (
                                            Text(
                                                "name",
                                            ),
                                            Text(
                                                "hi",
                                            ),
                                        ),
                                        (
                                            Text(
                                                "kind",
                                            ),
                                            Map(
                                                [
                                                    (
                                                        Text(
                                                            "type",
                                                        ),
                                                        Text(
                                                            "undump",
                                                        ),
                                                    ),
                                                    (
                                                        Text(
                                                            "dat",
                                                        ),
                                                        Map(
                                                            [
                                                                (
                                                                    Text(
                                                                        "type",
                                                                    ),
                                                                    Text(
                                                                        "regular",
                                                                    ),
                                                                ),
                                                                (
                                                                    Text(
                                                                        "executable",
                                                                    ),
                                                                    Bool(
                                                                        true,
                                                                    ),
                                                                ),
                                                                (
                                                                    Text(
                                                                        "contents",
                                                                    ),
                                                                    Array(
                                                                        [
                                                                            Integer(
                                                                                Integer(
                                                                                    101,
                                                                                ),
                                                                            ),
                                                                            Integer(
                                                                                Integer(
                                                                                    99,
                                                                                ),
                                                                            ),
                                                                            Integer(
                                                                                Integer(
                                                                                    104,
                                                                                ),
                                                                            ),
                                                                            Integer(
                                                                                Integer(
                                                                                    111,
                                                                                ),
                                                                            ),
                                                                            Integer(
                                                                                Integer(
                                                                                    32,
                                                                                ),
                                                                            ),
                                                                            Integer(
                                                                                Integer(
                                                                                    72,
                                                                                ),
                                                                            ),
                                                                            Integer(
                                                                                Integer(
                                                                                    105,
                                                                                ),
                                                                            ),
                                                                        ],
                                                                    ),
                                                                ),
                                                            ],
                                                        ),
                                                    ),
                                                ],
                                            ),
                                        ),
                                        (
                                            Text(
                                                "logtag",
                                            ),
                                            Integer(
                                                Integer(
                                                    1,
                                                ),
                                            ),
                                        ),
                                        (
                                            Text(
                                                "rest",
                                            ),
                                            Null,
                                        ),
                                    ],
                                ),
                                Map(
                                    [
                                        (
                                            Text(
                                                "name",
                                            ),
                                            Text(
                                                "use hi",
                                            ),
                                        ),
                                        (
                                            Text(
                                                "kind",
                                            ),
                                            Map(
                                                [
                                                    (
                                                        Text(
                                                            "type",
                                                        ),
                                                        Text(
                                                            "run",
                                                        ),
                                                    ),
                                                    (
                                                        Text(
                                                            "command",
                                                        ),
                                                        Array(
                                                            [
                                                                Array(
                                                                    [
                                                                        Map(
                                                                            [
                                                                                (
                                                                                    Text(
                                                                                        "String",
                                                                                    ),
                                                                                    Text(
                                                                                        "bash",
                                                                                    ),
                                                                                ),
                                                                            ],
                                                                        ),
                                                                    ],
                                                                ),
                                                                Array(
                                                                    [
                                                                        Map(
                                                                            [
                                                                                (
                                                                                    Text(
                                                                                        "Placeholder",
                                                                                    ),
                                                                                    Text(
                                                                                        "hiinp",
                                                                                    ),
                                                                                ),
                                                                            ],
                                                                        ),
                                                                    ],
                                                                ),
                                                            ],
                                                        ),
                                                    ),
                                                    (
                                                        Text(
                                                            "envs",
                                                        ),
                                                        Map(
                                                            [],
                                                        ),
                                                    ),
                                                ],
                                            ),
                                        ),
                                        (
                                            Text(
                                                "logtag",
                                            ),
                                            Integer(
                                                Integer(
                                                    2,
                                                ),
                                            ),
                                        ),
                                        (
                                            Text(
                                                "rest",
                                            ),
                                            Null,
                                        ),
                                    ],
                                ),
                            ],
                        ),
                    ),
                    (
                        Text(
                            "node_holes",
                        ),
                        Array(
                            [],
                        ),
                    ),
                    (
                        Text(
                            "edge_property",
                        ),
                        Text(
                            "directed",
                        ),
                    ),
                    (
                        Text(
                            "edges",
                        ),
                        Array(
                            [
                                Array(
                                    [
                                        Integer(
                                            Integer(
                                                1,
                                            ),
                                        ),
                                        Integer(
                                            Integer(
                                                0,
                                            ),
                                        ),
                                        Map(
                                            [
                                                (
                                                    Text(
                                                        "Placeholder",
                                                    ),
                                                    Text(
                                                        "hiinp",
                                                    ),
                                                ),
                                            ],
                                        ),
                                    ],
                                ),
                            ],
                        ),
                    ),
                ],
            ),
        ),
        (
            Text(
                "attach_to_logs",
            ),
            Bool(
                true,
            ),
        ),
    ],
)

serialized JSON of the same object

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

In v0.2.1 the signature of ciborium::de::from_reader() changed from

pub fn from_reader<'de, T: de::Deserialize<'de>, R: Read>(reader: R) -> Result<T, Error<R::Error>>

to

pub fn from_reader<T: de::DeserializeOwned, R: Read>(reader: R) -> Result<T, Error<R::Error>>

At first sight the new bound appears to be a looser bound than the original one, because in the serde crate we have

impl<T> DeserializeOwned for T where T: for<'de> Deserialize<'de> {}

so apparently anything that implements Deserialize also implements DeserializeOwned.

That is not a correct interpretation of the above blanket implementation, however, because it actually says that only types that implement Deserialize<'de> for any lifetime 'de also implement DeserializeOwned. Hence the signature change of ciborium::de::from_reader() is not a bound loosening and should therefore be considered a major change according to SemVer.

All this is to say that I think v0.2.1 should have actually gotten the version number 0.3.0.

Expected Behaviour

The signature change of ciborium::de::from_reader() is released under a new major version of the crate.

Environment Information

/

Steps To Reproduce

No response

I'm fuzzing this crate with a harness that checks that decoded objects, when encoded and decoded again, become the original object.

Looks like

#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|data: &[u8]| {
    if let Ok(v) = ciborium::de::from_reader(data) {
        let old: ciborium::value::Value = v;
        let mut writer = Vec::new();
        ciborium::ser::into_writer(&old, &mut writer);
        let new: ciborium::value::Value =
            ciborium::de::from_reader(&writer[..]).expect("round trip");
        if old != old {
            // contains NaN, can't compare.
            return;
        }
        eprintln!("old: {data:x?}");
        eprintln!("new: {writer:x?}");
        assert_eq!(old, new);
    }
    let _: Result<ciborium::value::Value, _> = ciborium::de::from_reader(data);
});

And it found this case

old: [195, 95, 255]
new: [195, 64]
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `Tag(3, Bytes([]))`,
 right: `Integer(Integer(-1))`', fuzz_targets/fuzz_target_1.rs:17:9

Is the harness correct here?

Is it possible to push an update of this project to crates.io? The version there hasn't been updated in a while and a few features that are useful have been added in the meanwhile, such as tags.

Is there an existing issue for this?

• I have searched the existing issues

Description

While some parts of the high-level ciborium crate by their nature require alloc (in particular, Value doesn't work without), most of it should manage well without.

It'd be great if structs with serde based serializations could be used on embedded targets where alloc is typically unavailable.

Acceptance Criteria

The encode_vec test works when replacing the Vec with a heapless::vec in a build configuration that does not use extern crate alloc.

The crate is added to the category no-std::no-alloc in its Cargo.toml.

Suggestions for a technical implementation

ciborium could introduce a crate feature alloc that is in the default set.

Then, parts of the library that depend on alloc are marked #[cfg(feature=alloc)].

As the crate is already no-std, the relevant parts should be trivially discoverable by grepping for alloc, and further grepping for the names of all types that get disabled without it.

Is there an existing issue for this?

• I have searched the existing issues

Description

The Value type in serde_json has convenience accessors of Index trait and .get().

Basically how they work is that .get() will try to index the value with the given type, returning an Option<&Value> as result. Using Index trait however never fails but just returns objects that are "undefined" in JSON terms. What this means in practice is that I can do something like this:

val["networks"][0]["ip_addresses"][0].as_string()

This makes it really convenient to access free form deeply nested JSON data which would be inconvenient to map to nested structures.

In order to implement this, however, the Map variant of Value should probably be changed to something which supports fast random access, as suggested in #81.

Acceptance Criteria

No response

Suggestions for a technical implementation

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

Long Japanese strings (>4095 bytes) can be normally serialized but deserialization fails with error. Alternative implementations deserializae correctly.

Expected Behaviour

Deserialization should succeed

Environment Information

Darwin Gleb-Pomykalov's-Macbook-Pro 21.6.0 Darwin Kernel Version 21.6.0: Thu Sep 29 20:13:56 PDT 2022; root:xnu-8020.240.7~1/RELEASE_ARM64_T6000 arm64
fish: Unknown command: enarx
fish:
uname -a; enarx --version; enarx platform info
          ^
fish: Unknown command: enarx
fish:
uname -a; enarx --version; enarx platform info

Steps To Reproduce

MRE:

        let mut s = String::new();
        while s.len() < 4095 {
            s.push_str("ボ");
        }
        println!("len = {}", s.len());
        let val = json!({
            "title": s,
        });
        let mut v = Vec::new();
        ciborium::ser::into_writer(&val, &mut v).unwrap();
        let _res: serde_json::Value = serde_cbor::from_reader(Cursor::new(&v)).unwrap();
        let _res: serde_json::Value = ciborium::de::from_reader(Cursor::new(&v)).unwrap();

        s.push_str("ボ");

        println!("len = {}", s.len());
        let val = json!({
            "title": s,
        });

        let mut v = Vec::new();
        ciborium::ser::into_writer(&val, &mut v).unwrap();
        let _res: serde_json::Value = serde_cbor::from_reader(Cursor::new(&v)).unwrap();
        let _res: serde_json::Value = ciborium::de::from_reader(Cursor::new(&v)).unwrap();

Investigation as a part of #54 seems to suggest that ciborium links, if not to libstd, then at least to liballoc, regardless of the presence of the std feature flag. I'm unsure whether or not this is intentional; I assume that disabling the std flag in this crate is intended to disable both liballoc and libstd. I'm unsure when this regression happened (if indeed this is a regression at all), but we should see about fixing it before issuing the next release. We may also want to consider adding an alloc feature.

We are trying to use ciborium to ser/de some complicated data. Part of the data invokes the Deserializer::deserialize_str and fails with an error: invalid type: string, expected str. This is because that ciborium limit the deserialze_str size to a hard coded 4096.

See:

Would it better to export a configurable limitation or alloc the internal buffer adaptively?

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

Given the following code, I expect the unit test to pass

pub fn serialize<T: ?Sized>(value: &T) -> Result<Vec<u8>, Error>
where
    T: serde::ser::Serialize + Sized,
{
    let mut writer = Vec::new();
    into_writer(&value, &mut writer).unwrap();
    Ok(writer)
}

// Forward mc_util_serial::deserialize to bincode::deserialize
pub fn deserialize<'a, T: ?Sized>(bytes: &'a [u8]) -> Result<T, Error>
where
    T: serde::de::Deserialize<'a>,
{
    Ok(from_reader(bytes).unwrap())
}

#[test]
fn test_string() {
    let the_string = "There goes the baker with his tray, like always";
    let serialized = serialize(&the_string).unwrap();
    let deserialized: &str = deserialize(&serialized).unwrap();
    assert_eq!(deserialized, the_string);
}

the following error is displayed

called `Result::unwrap()` on an `Err` value: Semantic(None, "invalid type: string \"There goes the baker with his tray, like always\", expected a borrowed string")
thread 'test_string' panicked at 'called `Result::unwrap()` on an `Err` value: Semantic(None, "invalid type: string \"There goes the baker with his tray, like always\", expected a borrowed string")', ciborium/tests/no_std.rs:29:27
stack backtrace:

Expected Behaviour

unit tests shall pass

Environment Information

Darwin devxs-MacBook-Pro.local 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 arm64

Steps To Reproduce

No response

Hi all,

I saw that Nathaniel mentioned the possibility of exposing the Serializer in the public API in pyfisch/cbor#179 (comment). I was wondering what steps or contributions you would be looking for getting it into a state where that would be a possibility. Not sure how many other people share my use-case, but I have been using Serializer::serialize_seq() for outputting logs to disk (with some intervening compression) with serde_json and was curious about moving to CBOR. I know there is more involved than just dropping pub in front of

Let me know what you think and thanks for all your work on this crate.

-Tom

Is there an existing issue for this?

• I have searched the existing issues

Description

I would like to add CBOR support to a data conversion utility that I've created, which makes use of serde. To do so, I would preferably like to access Deserializer and Serializer directly, rather than through helper methods.

Acceptance Criteria

Ability to use ciborium::{Serializer, Deserializer} from other crates.

Suggestions for a technical implementation

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

On deserializing an Option containing a tag, Semantic(None, "required tag not found")' is encountered.

#[test]
fn option() {
    let  opt = Some(Required::<u64, 0xff>(2u32.into()));

    let mut bytes = Vec::new();
    ciborium::ser::into_writer(&opt, &mut bytes).unwrap();

    // 'called `Result::unwrap()` on an `Err` value: Semantic(None, "required tag not found")',
    let output = ciborium::de::from_reader(&bytes[..]).unwrap();
    assert_eq!(opt, output);
}

Expected Behaviour

The above test should succeed. The byte representation for Some(Required(2u32)) and Required(2u32) are identical, I'm not too familiar with the CBOR spec but I assume this is correct. In which case, Some should be implied by the presence of a tag in the encoding.

Might have something to do with the tag being discarded here

Environment Information

Linux SMP PREEMPT_DYNAMIC Wed A x86_64 x86_64 x86_64 GNU/Linux

rustc 1.56.1 (59eed8a2a 2021-11-01)

Steps To Reproduce

#[test]
fn option() {
    let  opt = Some(Required::<u64, 0xff>(2u32.into()));

    let mut bytes = Vec::new();
    ciborium::ser::into_writer(&opt, &mut bytes).unwrap();

    // 'called `Result::unwrap()` on an `Err` value: Semantic(None, "required tag not found")',
    let output = ciborium::de::from_reader(&bytes[..]).unwrap();
    assert_eq!(opt, output);
}