matthewhammer / candid-spaces Goto Github PK

Simple CLI tool for importing and exporting data from a Candid Spaces canister.

The Rust crate may provide almost enough.

But, we need to "re-encode" and "re-decode" the Candid data as a Candid AST, as the API accepts and emits these ASTs, as to be generic and multi-purpose.

That is to say,

• for importing into the canister with the tool (from a local machine), we want to read ordinary Candid data as human readable text or in serialized binary form, and then represent and import this data as Candid value AST, sent to the canister (see put).

• for exporting, we want to reverse the process, converting a Candid value AST into either ordinary text or serialized binary form (see getFullImage and getSubImage, which respond with the same data types accepted by put).

Perhaps this is best done as a feature added to either the Candid tools, or ic-repl, since those are the tools that already exist?

Thoughts @chenyan-dfinity?

This issue tracks the core API of the initial service.

• put candid data into a public space, using a path to identify it.
• get candid data from a public space, using a template to identify it.

Additional issues will call for access control configuration, quasi-private spaces, and other features.

The event log permits re-generation of each other table/relation/map stored in the canister state.

Providing stable memory for the event log means that all events are stored in stable memory, and that upgrades need not transform it.

Consequently, a canister with this feature is a canister with a lossless representation of its past state. That has important correctness implications for upgrades, since all other information can be regenerated from it, should the other information (stored in auxiliary data structures that may change format) be damaged or lost in a faulty upgrade.

A query language would filter, join and transform candid data in one view/space, producing a new view.

Background

Candid Spaces was originally inspired by two concepts meeting:

• Candid data from/to the IC
• Tuple Spaces (aka, the "Linda Model") service for free-form distributed coordination between other services.

In #2 and #3 we implement some basic operations from Tuple Spaces' operations (See slide 61 onward for an overview of the operations, as distilled by John Reppy).

As the slides show, one of the main concepts for consuming data from Tuple Spaces remains unimplemented: "pattern-matching" (using the type called "templates" in Reppy's slides).

What we want

We want a query language to define new views based on filtering, joining, projection or other Candid-data-based transformations.

When I say "query language" I really mean an expression language whose principal operations consume and produce view targets (currently either a space with a path, or a view with a unique ID) and transform, filter and combine them (table joins, set operations, etc.). Within that language, there must be a sub-language for candid data, including pattern-matching and projection of variants and products, respectively.

For each query, we want a View that captures its answers, as an ordered sequence (just like our existing concepts of spaces and views).

That is to say, for each query, we want the database concept of a materialized view that can be used like an ordinary table, and be the subject of further queries, or joins, etc.

Related work

Datafun comes from the recent academic PL landscape looking at query language semantics.
It gives a good example of what our query language could compare to someday, perhaps with overlapping, but not subsuming features. The purpose of Datafun is to provide a strong termination property for all programs. Since canisters are naturally coinductive (giving answers in streams, not wholesale), the purpose of Datafun is fundamentally not the same as our query language, but it should be meaningful and very insightful to compare them as we consider extending the query language.