I'm building a custom IDL parser using this repository's spec and tests, but it seems like there is an error.

parse_type.rs

type broker = service {
  find : (name: text) ->
    (service {up:() -> (); current:() -> (nat32)}); <<< missing semicolon here, but compiles
};

IDL.md

<actortype> ::= { <methtype>;* } <<< semicolon is not optional

when dfx call takes an opt type, the opt subtyping rule essentially turns off the type checking. We need to disable or at least warn about the use of opt subtyping rules in the parser.

Can you please help me to understand a couple of things?

candid/IDL.md section Serialisation -> Types says:

We assume that the fields in a record or function type are sorted by increasing id and the methods in a service are sorted by name.

What is "sorted by increasing id" in these terms? Sorted in an order they appear in source .did file?

How exactly does the type definition table works (one which assigns uint to a type)?
Is this correct that expressions of kind type Example = opt record { nat; nat; } (in other words - type definition expressions) are added to the type definition table? And every other appearance of type "Example" (when used as datatype id) should be serialized as the index of "Example" in the type definition table?
Is this correct that the type in expression like

service A {
  foo : (vec opt text) -> () oneway;
             ^------^ this complete type
}

is also added to the type definition table if it didn't appear earlier? Even if there was no expression like type OptText = opt text; met before.

After parse of a .did file one can end up having a "complete" type definition table - which contains every type that appear in a file. Should she use this type definition table when serializing a method call or she should somehow filter this table so it would contain only types that are really needed for the serialization of this method call? Is the order of types in the type definition table matters? How do one assert the correct order?

Currently specifically says only Rust implementation, is that still accurate?

I started to do some baby steps in formalizing a “mini candid” in Coq, starting with the rules that seem safe.

But adding the “specialize to constituent type” rule is causing problems:

If the rule is

not (null <: t2)     t1 <: t2
--------------------------------
        t1 <: opt t2

as it is now, then this is not transitive. We have int <: opt int (by this rule), opt int <: opt opt int (by the normal rule), but not int <: opt opt int (because of the not (null <: t2) restriction).

This restriction was added in #135, upon my suggestion. Without that restriction, we have the problems described in #135, i.e. bad interaction with the “catch decoding rule”, and the problem that

    t1 <: t2
------------------
  t1 <: opt t2

doesn't make progress if t2 = opt t2.

Or, to respond to @rossberg says in #135 (comment)

Interesting suggestion. I suppose options of options are not something you would see often in an interface, so limiting their evolution might be okay. How confident can we be that this has no problems?

I wish we had spare cycles to mechanise this, and prevent further holes.

I am no longer confident that this has no problems.

One way to maybe! taper over the problem is to restrict the other rule as well:

not (null <: t2)     t1 <: t2
--------------------------------
     opt  t1 <: opt t2

so that you can’t go from opt int to opt opt int… but that essentially disallows evolution under opt opt and might be better served by disallowing opt opt… probably not nice. Or maybe this rule:

(null <: t1) iff (null <: 2)      t1 <: t2
---------------------------------------
             opt  t1 <: opt t2

which rules out opt int <: opt opt int without ruling out opt nat <: opt int nor opt opt nat <: opt opt int ?

Right now we have to use an IDLValue which isn't ideal (as then we have to match everywhere). There should be a native encapsulation, similar to how we do Nat and Int.

enum E {
  A(i8),
}

When encoding E, Candid thinks field A is of type record { 0: int8 }, but decoding is expecting i8 directly.

#[derive(CandidType)]
struct Empty {};
#[derive(Deserialize)]
struct A { foo: Option<String> }

let bytes = Encode(&Empty {})?;
Decode!(&bytes, A)?;

The last line should fail, but it deserialized to None. If foo is a non-Option type, it correctly reports missing field foo.

The spec currently says

Completeness: Subtyping covers all cases of successful deserialisation.

and I wonder what that precisely means, and if we really want it.

(Previous discussion at #128 (comment) )

Is your feature request related to a problem? Please describe.

When building CI workflows, we need a human-readable representation of how two Candid values relate when they are not equal.

This human-readable representation would be included in reports of CI failures (e.g., lists of "expected versus observed" test outputs).

To compute that representation meaningfully, we need some constructive math to define it.

Describe the solution you'd like

This issue proposes a new Rust crate, which would augment the existing candid crate with higher-level implementations of various relations over candid values and types.

Let's call it candid_relate.

It gives semantic operations over ASTs of Candid values and types, including:

• Structural comparison (in the sense of mo:base/Order.Order)
• Structural editing, with a representation of diffs/deltas.
• Equality checks, in a semantic sense that requires types (re-implement some semantics of Candid, but at a higher level of abstraction, with more debugging information when things "go wrong").

This crate is not essential for building candid services, but would be essential for any candid tool that tries to debug expected versus observed values of candid data.

Here's an example of some related uses, demonstrating diff, delta and equal, and how they inter-relate:

// suppose we have two Candid ASTs, and a type that classifies each:
let (x : candid::Ast, y : candid::Ast, t : candid::Type) = ... ;

// (note: candid values x and y each have a common candid super-type, t)
// At the common type, we can edit one value into the other
let d : candid_relate::Delta = candid_relate::diff(x, y, t);

// difference/delta is represented by d, and can be applied as follows:
let z = candid_relate::delta(x, d);

// equal at the common supertype t, but perhaps not generally:
assert_eq!( candid::equal(z, y, t) );

When two values are not equal, a CI script could use this crate to help pretty-print a report that summarizes ("usefully") how the two values differ, e.g., by implementing the Show trait, or something similar:

println!("{}", d)

Describe alternatives you've considered

• Standard equality test (gives limited insights when the answer is "false", hence this proposal)

Additional context

• This came up while trying to build useful tools (in Rust) for CI to run to generate reports, and reports about failures of IC services written in Motoko/Rust.

• Caching and cache repair: The proposed "incremental edits representation" is (potentially) related to a future effort that would make query results systematically incremental across the Internet Computer. Just as we prescribe a representation of data by designing Candid, the representation of edits to these values could also be shared/systematic/general.

Is your feature request related to a problem? Please describe.

I want to write an actor/service like this in Motoko, with a corresponding Candid service type:

actor MapThing<X,Y> {
  put : (X, ?Y) -> async ();
  get : X -> async ?Y;
}

Leaving out some inessential details, this MapThing actor provides a service whose API is parametric in two types, X and Y, for a generic key and value type, respectively. It stores a dynamic map from Xs to Ys.

More generally, I want to author Candid services that use generic type arguments, and as a user, instantiate services by providing these types over the network.

Describe the solution you'd like

The solution here involves extending Candid, Motoko, as well as (perhaps) the Internet Computer itself.

Describe alternatives you've considered

Many systems and languages lack polymorphic types. The usual workarounds map a polymorphic design into a monomorphic one, somehow. For instance, we could assume a single choice of type for X and Y above, and try to provide a way to map into and out of these single choices (e.g., use [nat8] for all types and do extra layers of serialization, the current solution).

Additional context

None.

From the point of view of the spec, this field's type does not matter; whatever the canister returns we send back in the same shape. But from the Candid file documenting the HTTP Query spec, it becomes a simple record {} because there's no way to communicate this properly in Candid.

Either Candid could implement a generic (in which case the token becomes T) or the any type, as the token could change type between two responses (and the spec does not actually care).

serde serializes SystemTime as Duration from std::time::UNIX_EPOCH. Having a CandidType implementation for it that does the same would be useful when sending SystemTime in a message.

I actually tried to implement this myself but failed because I don't understand how to generate named fields in candid. The serde implementation for SystemTime wants a record with secs_since_epoch and nanos_since_epoch fields, for the second and nanosecond parts, respectively. I don't know if that's possible to do in candid.

Currently, all the functions that do the serialization construct a new buffer from scratch (e.g., https://docs.rs/candid/0.6.17/candid/ser/fn.encode_one.html).

This might lead to unnecessary work when serializing large primitives (e.g., large blobs).

It would be nice if the API could work with any implementation of Write.
This way one would be able to write to the message reply buffer directly using something like

pub struct ReplyAppender;

impl std::io::Write for ReplyAppender {
  fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
    // We can be a bit smarter and have a buffer for very small writes
    // and call append directly for big writes, depending on the cost model.
    ic0::msg_reply_data_append(buf.as_ptr() as u32, buf.len() as u32);
    Ok(buf.len())
  }
  fn flush(&mut self) -> std::io::Result<()> { Ok(()) }
}

This will eliminate one extra copy when sending the data to the clients (which might easily be the main part of the work performed by the canister).

Is your feature request related to a problem? Please describe.
An example candid deserialization error message is:
"No more values to deserialize".
That can be hard to debug: what I want to know is:

• The full context -- that is the full sequence of bytes that were being deserialized.
• Why was a value expected? In other words, what part of the candid type is missing?

Describe the solution you'd like
I'd like something like:
"No more value to deserialize. While trying to deserialize into a , the end of the byte stream was reached, but a was expected."

Describe alternatives you've considered
I have none.

Additional context

can interpret its own output

Meaning that given an output representing a diff, the tool can perform this diff as an edit (new subcommand, joining echo and diff). The edit subcommand takes a value and a diff and it produces a second value, related to the first by the diff.

Wasm interface types are a way for wasm modules to have a type-rich interface without any inherent serialization overhead. They're mostly intended to support low-cost calls between multiple wasm modules, but I suspect that they could be used for dfinity as well. They also don't require the module to include any serialization/deserialization code internally, all that gets generated at instantiation-time at the boundary automatically.

Interface types aren't ready for prime time, but if my understanding of candid is correct, there appears to be a lot of overlap.

Plus, languages like Rust will eventually support generating exported functions with interface types automatically without additional tooling.

The Query/Call buttons are not type=submit, so they don't trigger when pressing Enter.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

#[serde(rename_all = "snake_case")]
pub enum CanisterStatusType {
    Running,
    Stopping,
    Stopped,
}

Discussed https://github.com/dfinity-lab/dfinity/pull/4812

Right now, if you pass in 0 as a number as is, the textual parser turns it into an Int.

A better experience would be to have the textual parser uses the smallest type that can represent the number (e.g. 0-255 uses Nat8, -257 - -65536 use Int16, etc.

Then, have the candid values officially support Nat8 <: Nat16 <: Nat32 <: Nat64 <: Nat and Int8 <: Int16 <: Int32 <: Int64 <: Int then NatN <: IntN and Nat <: Int, thus removing any forced typings when entering numbers through the textual parser.

The new user’s guide, added in #158, does not fully address the Rust-specific workflow and type mappings.

Story

As a rust canister developer, I want Rust-specific Candid documentation, so that I can build Rust canisters that use Candid.

Demo

• Go to https://github.com/dfinity/candid/blob/master/docs/modules/candid-guide/index.adoc
• Section “How do I use Candid as Rust canister developer?” has useful information
• In the reference section, all types describe their mapping to rust types.

We currently have this rule:

<nat> not in <fieldtype>;*
record { <fieldtype>;* } <: record { <fieldtype'>;* } ~> f
------------------------------------------------------------------------------
record { <fieldtype>;* } <: record { <nat> : opt <datatype'>; <fieldtype'>;* }
  ~> \x.{f x with <nat> = null}

This gives record {} <: record { foo : opt bool }.

We also have record { foo : opt bool } <: record { foo : reserved } (by virtue of opt bool <: reserved).

By transitivity, we should have record {} <: record { foo : reserved }.

So it seems we need

<nat> not in <fieldtype>;*
record { <fieldtype>;* } <: record { <fieldtype'>;* } ~> f
------------------------------------------------------------------------------
record { <fieldtype>;* } <: record { <nat> : reserved; <fieldtype'>;* }
  ~> \x.{f x with <nat> = null}

as well.

(This uses null as an arbitrary representation for a reserved value. Woudn’t mind adding an explicit reserved to the textual representation and data model, that would be handy in other contexts as well.)

We could combine the two into a single rule by saying

<nat> not in <fieldtype>;*
record { <fieldtype>;* } <: record { <fieldtype'>;* } ~> f
opt empty <:  <datatype'>
------------------------------------------------------------------------------
record { <fieldtype>;* } <: record { <nat> : <datatype'>; <fieldtype'>;* }
  ~> \x.{f x with <nat> = null}

Minimal repro:

use candid::{CandidType, Decode, Deserialize, Encode};
fn main() {
    #[derive(CandidType, Deserialize)]
    struct Wrapper(Vec<usize>);
    let w = Wrapper(vec![1]);
    let b = candid::Encode!(&w).expect("failed to encode");
    candid::Decode!(&b[..], Wrapper).expect("failed to decode");
}

fails with:

thread 'X' panicked at 'failed to decode: 
Message: "Type mismatch. Type on the wire: Vec; Provided type: Nat64"
States:
Trailing type: [I(-8)]
Trailing value: [01, 01, 00, 00, 00, 00, 00, 00, 00]
Type table: [[I(-20), U(1), U(0), I(1)], [I(-19), I(-8)]]
Remaining value types: []

I often need a little more than the signature definitions to get the right syntax for specifying Candid values on the command line (for example, do I need to use equal sign or colon, comma or semi-colon?).
Adding.a usage example for each type would make it easy to look up this information.

If you try and compile the following code in a rust project without serde as a dependency:

use candid::Deserialize;

#[derive(Deserialize)]
pub struct A();

You will get the following error

error[E0658]: use of unstable library feature 'rustc_private': this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via `Cargo.toml` instead?
 --> rust_canisters/dfn_http/src/lib.rs:3:10
  |
3 | #[derive(Deserialize)]
  |          ^^^^^^^^^^^
  |
  = note: see issue #27812 <https://github.com/rust-lang/rust/issues/27812> for more information
  = note: this error originates in a derive macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

For more information about this error, try `rustc --explain E0658`.

This is the case for any struct, but it is resolved as soon as you add serde as a dependency.

This is true for 0.25 and master at the time of writing.

The Candid repo currently contains two versions of the spec, one in .adoc the other in .md format. And the former seems out of sync with the latter.

We should either generate one from the other, or get rid of one of the copies.

Example: in a .did file I have:

type ProposalId = nat64;
service : {
  "change_proposal_deadline_for_tests": (proposal_id: ProposalId, timestamp_unix_epoch_nanos: nat64) -> ();
}

In the web UI, I get 2 input fields both labelled nat64 --- so it's hard to tell apart the 2 arguments. It'd be nice if the Web UI would also display the arg name, here proposal_id and timestamp_unix_epoch_nanos.

We have two structures;

    #[derive(Deserialize)]
    struct CallCanisterArgs1 {
        canister: Principal,
        method_name: String,
        args: Vec<u8>,
        cycles: u64,
    }

    // RUNNING OUT OF CYCLE
    #[derive(Deserialize)]
    struct CallCanisterArgs2 {
        canister: Principal,
        method_name: String,
        cycles: u64,
    }

The CallCanisterArgs2 runs out of cycle if we try to deserialize the same value CallCanisterArgs1 would deserialize, which points to a performance regression when skipping fields.

An actor reference inherently is represented by a principal.

This also unblocks dfinity/motoko#2264

We keep getting user questions about Candid, e.g., about its syntax. But all documentation we currently have is the "spec", which is not really targeted at users and is overly technical.

We should have some primary "user guide" that is accessible for programmers, not implementers.

Create a C API for Candid in Rust and compile it down to WASM to distribute with C SDK.

The “option type can be specialised to its constituent type” still worries me…

Here is one problem, maybe fixable, but maybe indication that this rule just doesn't fit well with the system.

With this rule, we certainly want

true ~> opt true : opt bool

But also

true ~> opt true : opt bool
true ~> opt opt true : opt opt bool
true ~> opt opt opt true : opt opt opt bool
…

So what does this mean for

type FixOpt = opt FixOpt

true ~> ? : FixOpt

Let’s try to apply the rules (with a logical variable ?X for the result).

    true ~> ?X : FixOpt
⇐ opt true ~> ?X : FixOpt      
⇐ opt true ~> opt ?Y : FixOpt -- ?X = opt Y
⇐ true ~> ?Y : FixOpt

and we are in a loop. The solution ?X = opt ?X is not allowed (our values are inductive, not coinductive – I hope!). And since we understand _ ~> _ : _ to be inductive, we should get

not (∃x. true ~> x : FixOpt)

But by the rule about “failed parses at type opt turn into null`”, this implies

opt true ~> null : opt FixOpt

which (because opt FixOpt = Opt) implies

opt true ~> null : FixOpt

which implies

true ~> null : FixOpt

by the “subtype to constituent type” rules.

This is a blatant contradiction! This means our _ ~> _ : _ relation is inconsistent!

How can that happen? Don’t all inductively defined relations well-defined? No, they are only well-defined when the relation appears only in strictly positive positoins in the rules. And our rule

not (<v> ~> _ : <t>)
-------------------------
opt <v> ~> null : opt <t>

breaks that. This would also be a major headache when trying to put this into a theorem prover.

The way to deal with that there (e.g. Coq) would be to define the relation as a well-founded function on the v (we can’t use t because the types themselves are coinductive). In all rules, the antecents only mention subterm of v. And it also (nicely) matches a real implementation which would traverse v.

Well, all rules but his one:

<v> ≠ null
<v> ≠ (null : reserved)
<v> ≠ opt _
opt <v> ~> <v'> : opt <t>
-------------------------
<v> ~> <v'> : opt <t>

Maybe we can patch around this issue (e.g. requiring <t> ≠ opt <t>) … but I am leaning towards just dropping it. It makes the formalism cleaner (a good sign) and it makes the implementation easier to get right and secure (they otherwise have to guard themselves against an infinite loop upon true ~> ? : FixOpt).

Do we have any compelling reason/use case for this rule?

not (null <: <datatype>)
<datatype> <: <datatype'>
-----------------------------
<datatype> <: opt <datatype'>

Finished by matthewhammer/motoko-bigtest#10

Originally:

I would like to show examples of using the Candid tools within Github Actions:

• download and install candid tools (e.g., build from source, or get binary from Rust crates cache, or from a GH release?)
• test script captures a comprehensive log of all relevant canister behavior, as a candid value stored in a file (pipe output of dfx canister call MyCanister getFullLog)
• test script uses candiff to compare the latest log to an earlier one that is human readable, saved in the repo; any differences are also human readable (thanks to candiff)

I suspect it's due to having r# at the start of the field name, which should be ignored by Candid in the proc macro.

Candid applies some logic to every element of a vector, which is unnecessary for blobs since they're a single slice of memory.

This is going to consume a lot of gas for users so it should be made more efficient.

Encoding could also use a memcpy improvement in this use case.

The usage of blob is frequent enough (e.g. Wallet installs and call forwarding) that it should be high priority.

Looks like we still haven’t found the grail yet…

TL;DR: Our relaxed coercion rules for opt (#110 and following) may lead to “hard” deserialization errors later on.

I seems that the following can’t all hold:

• We are sound in the higher-order case (as per IDL-Soundness)
• We have no subtyping checks while deserialization (one of our Design goals).
• Type Erasure (another of the design goal)
• Record Extensibility, at least in the form implemented right now.

Counter-example

Here is the counter example:

service A1 : { bar : () -> () }
service B1 : { baz : () -> () } 

service C : { 
  init : (service A1, service B1) -> (); // stores (a1, b1)
  run : () -> () ;  // runs a1.bar(b1.baz())
}

If we pass A1 and B1 to C.init, all is well. C.run works fine.

Now we upgrade A1 to A2 and B1 to B2, as follows:

type WantsBool = service : { foo : (bool) -> () }
type WantsInt = service : { foo : (int) -> () }
service wantsInt : WantsInt
service A2 : {  bar : (opt WantsBool) -> () } // calls `arg.foo(true)` if `arg` is not `null`
service B2 : { baz : () -> (opt WantsInt) }  // returns (opt wantsInt)

These upgrades are permitted by our :<, and actually desirable (i.e. restricting <: to avoid “obviously bogus” upgrades doesn’t help)

But now C.run will cause B2 to pass opt wantsInt to A2. This will succeed at deserialization (because “Function and services references coerce unconditionally”). Now A2 invokes wantsInt.foo(true), and the deserialization of (true) at expected type (bool) will fail in wantsInt.

Generic Soundness proof

In https://github.com/dfinity/candid/blob/master/spec/IDL-Soundness.md#proof-for-canonical-subtyping we have a generic soundness proof for “Canonical subtyping”, that says “A solution with canonical subtyping (transitive, contravariant) is sound.”. And our <: is that!

But it’s not “decompositional” any more. In particular,

If t1 <: t2 and s1 in t1 <: s2 in t2 then s1 <: s2.

(as required in the generic proof) doesn’t hold .

Concretely in our example above, we have opt WantsInt <: opt WantsBool (because all opt types related). But because the reference value is passed through, this also means WantsInt in opt WantsInt <: WantsInt <: opt WantsBool, but we don’t have WantsInt <: WantsBool.

So…

Dunno. No concrete suggestions. It’s sunday morning, and I just wanted to get this insight out of my head.

Probably something along the lines of impl<T: CandidType + ToOwned> CandidType for Cow<T> { ... }.

The obvious advantage is to be able to serialize and deserialize without copying the data if not needed.

The new user’s guide, added in #158, does not fully address the JS-specific workflow and type mappings.

Story

As a JS frontend developer, I want JS-specific Candid documentation, so that I can build JS frontends that use Candid well.

Demo

• Go to https://github.com/dfinity/candid/blob/master/docs/modules/candid-guide/index.adoc
• Section “How do I use Candid as JavaScript front-end developer?” has useful information
• In the reference section, all types describe their mapping to JS values

When a user mistakenly uses and invalid record field name, the following error message is observed.
Invalid data: Unable to convert IDL to bytes: field 2468089002 not found
This appears to reference the hash of the invalid record field name and not the invalid record field name itself, which is somewhat ambiguous. Would awesome if we could display the unhashed value so that the user sees:
Invalid data: Unable to convert IDL to bytes: field 'name' not found