Revisit earlier proposal https://github.com/p2panda/handbook/blob/646776a0ffccb746889b9fc81c88d9737963a812/spec/ideas/migrations.md

It contains more than a design-document

Thinking about dark crystal.

Specify how, when and why we prioritize replicating specific logs over others

Have a background task which automatically removes payloads of entries which are “not used” anymore (is it deleted instances? Outdated MLS key packages? etc.), if a sequence between skiplinks is finally empty it should also delete the complete entry, only keeping the skiplink

I want to write something about built-in schemas. Which ones do we have?

schema

describes schemas

fields:

• name
• description
• fields (list of field definitions)

persona

fields:

• username
• relations (list of relations to other personas, e.g. "same user as..." / same-as or "bot controlled by...")

Create self-destructing, ephemeral instances (maybe using encryption / throwing away the key).

• It could be interesting to build a system where every piece of data has an expiry date (which may be far in the future)
• It may be interesting to allow setting an expiry date on data explicitly
• It may be interesting to let node operators configure a node to only store e.g. data that is less than a year old

A way to specify different permissions for the authors that make up a persona. Could be used e.g. to make bots part of your persona (and allow posting on your behalf) but not give them the right to add and remove authors, or just allow them to post using specific schemas.

An API for clients to get information about a node’s replication state, a library that creates a visualisation of that state (SVG?)

https://github.com/ssb-ngi-pointer/netsim

Very-random-vague-thinking-which-is-probably-mistaken: are there any interesting parallels between a chain of update messages which form an Instance, and bamboo / Magma / Lava append only logs? This is a raw thought I’ve never managed to follow through properls. But I’ve had it a few times now (maybe many, I can’t remember) so I’d like to discus it and put it to bed for good.

Proposal to add a new system schema with the name "application_schema" to the p2panda spec. The purpose of the application_schema schema is to represent user-defined data schemas. These can be created by developers of applications who want to transmit and store application data using p2panda.

Related: schema migrations

User Stories

Developers can define data schemas for their applications

Alex is a developer and wants to build a blogging application using p2panda. Users of the application will be able to publish blog entries containing text content. Other users can then read these blogs on their own computer.

Alex creates a new schema by publishing a create operation of the schema schema. The operation defines the name "blog", the description "markdown-formatted blog post" and a list of field definitions. The fields defined by Alex are:

• title
• body
• created

After publishing this operation, Alex has created the new blog schema. They can now program their blogging application to create blog instances by publishing operations of the blog schema. When Alex publishes the first blog operation, the p2panda node they use assigns Alex' first unused user log to the new schema. All blog operation created by Alex from now on will be appended to this log.

Alex can configure their p2panda node to track the new schema. Tracking a schema causes the node to materialise blog documents described by blog operation in a database table. The contents of this table can then be queried through the node's API.

Now, Alex can program their blogging application to retrieve blog documents created by all users of their application by querying the node's API. Users of the blogging software can now comfortably read blogs created by other users and write their own blogs.

Developers can reference data that uses another schema in their entries

[tbd]

Schema specification

Following is a CDDL definition of schema operations that builds upon the schema of OperationEncoded

operation-fields /= {
    name: schema-name,
    description: schema-description,
    fields: [+ field-definition]
}

; Every schema has a name, which is limited in length to ensure 
; readability when displayed in logs or listings.
schema-name = tstr .regexp "\w{1,80}"

; Schema descriptions are less constrained but also have a maximum
; length of 256 bytes to keep them concise.
schema-description = tstr .size (1..256)

; Field names have the same length restriction as schema names
field-name = tstr .regexp "\w{1,80}"

; Possible data types are chosen to ensure compatibility with storing
; the data in Postgres databases.
; `datetime` is a ISO8601-formatted string
field-definition /= {
    type: "text" / "int" / "float" / "boolean" / "datetime",
    name: field-name,
}

; The `relation` data type allows referencing documents described by 
; other schemas. This has implications for materialisation: All schemas
; referenced must also be materialised.
field-definition /= {
    type: "relation",
    name: field-name,
    references: hash
}

Checklist should include "i updated the docs"

A method to record statistics of global replication activity / network health to allow us to see how p2panda is growing

Let's discuss how we are naming stuff.

Instance

We created this name because instances are instances of schemas (which we had thought of before). However, developers coming in contact with p2panda instances for the first time will probably not share this context.

How about:

• object
• entity

We can also add more candidates to discuss above.

Think about a maximum payload size for entries / logs (blobs), is it configurable or defined on protocol level? Or are these actually two separate things?

This could be configured on a per-schema level so that schema designers can design them explicitly for a specific usage scenario including specific target devices (and schemas can be excluded from replication).

We might only need update!

What if I missed out on the CREATE message?

• content published is identified by being associated with a key pair
• when multiple devices are used to publish content a solution has been to place a copy of the key pair on each device which introduces all kinds of problems in key pair management
• instead we create a key pair per device and create another way of modeling identity:
  • when a user initially starts using p2panda using their device D1 they create a persona P, which is linked to their device's key pair - and publish it to the network
  • when they want to use a second device D2 (which creates its own key pair) they publish a message using D2 that declares "i am the same as persona P"
  • when D1 receives this message it notifies the user that there was a "log-in attempt from a new device" and gives them the option to confirm or reject this
  • if the user confirms, D1 publishes a message that says "i accept D2's declaration"
  • now, the clients of other users can lookup the persona associated with the key pairs of D1 and D2, which is the same persona P
• TBD:
  • users can remove associated devices
  • users can create multiple personas
  • a persona can be shared by multiple users
  • users can block devices from asserting identity with their device
  • users can assign metadata to persona-associations, e.g.
    • "this is P's mobile device"
    • "this device has limited rights"
    • "this device is not allowed to de-associate other key pairs from this persona"
    • "this device is a spokesperson posting on behalf of persona P"
    • "this is a bot posting on behalf of P"
    • "this is a bot controlled by persona P"
    • and others

Schemas

persona

• name: string

persona-authorization-request

• persona: hash

persona-authorization

• request: hash
• accepted: boolean

In order to be able to materialise instance while throwing away payloads / entries in between

Bamboo Entry Encoding

• Bytes encoded following the Bamboo specification: https://github.com/AljoschaMeyer/bamboo#encoding
  • We use BLAKE3 hashes wrapped in yasmf multi-hash format for backlink, lipmaalink (skiplink) and payload_hash
    • Specification here:
      • blake3 https://blake3.io
      • yamf https://github.com/AljoschaMeyer/yamf-hash
      • yasmf https://github.com/bamboo-rs/yasmf-hash
  • We use Ed25519 as a public-key signature system
    • Specification here:
      • https://ed25519.cr.yp.to/
  • Currently we encode, but do not make use of the tag marker. We reserve it for future protocol updates
    • It needs to be set to 0?

Log Integrity

TODO

• Specify a pointer at the last entry of an instance you’re applying the update on
• Specify how we materialize multi-writer instances (they form a DAG)

CBOR

• Bytes encoded following the CBOR specification: https://datatracker.ietf.org/doc/html/rfc7049
  • We do not make use of CBOR tags
  • SHOULD (Recommendation): Canonical CBOR encoding as per https://datatracker.ietf.org/doc/html/rfc7049#section-3.9
    • Note: If not followed, hashes will be different for the same content on different implementations (test vectors / examples will potentially not work across implementations)
    • Integers must be as small as possible
    • The expression of lengths in major types 2 through 5 must be as
      short as possible
    • The keys in every map must be sorted lowest to highest
    • Indefinite-length items must be made into definite-length items
    • IEEE Floats are used, it is recommended to follow the canonicalization rules by the CBOR specification
  • MUST We use CBOR in "strict" mode
    • Operations not following the rules of "strict" mode need to be rejected
    • This includes:
      • A map (major type 5) can not have duplicate keys
      • Incorrect formatting (value does not represent claimed type)
    • We do not have to worry about tags as we do not use them

Operation Format

• Map keys must match ^[A-Za-z]{1}[A-Za-z0-9_]{0,63}$
• Canonical format:
  • Similar to the canonical format of CBOR, p2panda recommends one for operations to assure same hashes across different implementations
  • All array values SHOULD be in sorted order (from lowest to highest)
    • as long as the ordering does not hold semantic value, this is true for:
      • previous_operations
      • document view ids in pinned relations (also the ones inside of lists)
    • Arrays with semantic ordering (not sorted):
      • pinned relation lists
      • relation lists
• All operations MUST follow the Operation format encoding (version, action, previous_operations, schema, fields), specified here: https://p2panda.org/handbook/docs/writing-data/operations

Hi,

What is the status of the project? Is it still planned to implement it?
I'm currently brain storming about a similar project with a slight different scope. I would like to have a tool for everyday use.

See an draft version/writeup here: https://github.com/shared-economy/share-your-stuff-api/blob/master/README.md

Specify how we do “gossip” discovery (one node told the other that a third exists), does it make sense to put this into a schema? The data inside of it might be very old (IP addresses change …)

How do we handle reconnection attempts? Is there a maximum? When do we try again?