stencila / dev-config Goto Github PK

Programmable, reproducible, interactive documents

👋 Intro • 🚴 Roadmap • 📜 Docs • 📥 Install • 🛠️ Develop

🙏 Acknowledgements • 💖 Supporters • 🙌 Contributors

Stencila is a platform for creating and publishing, dynamic, data-driven content. Our aim is to lower the barriers for creating truly programmable documents, and to make it easier to publish them as beautiful, interactive, and semantically rich, articles and applications. Our roots are in scientific communication, but our tools are useful beyond.

This is v2 of Stencila, a rewrite in Rust focussed on the synergies between three recent and impactful innovations and trends:

• Conflict-free replicated data types (CRDTs) for de-centralized collaboration and version control.

• Large language models (LLMs) for assisting in writing and editing, prose and code.

• The blurring of the lines between documents and applications as seen in tools such as Notion and Coda.

We are embarking on a rewrite because CRDTs will now be the foundational synchronization and storage layer for Stencila documents. This requires fundamental changes to most other parts of the platform. Furthermore, a rewrite allow us to bake in, rather than bolt on, new modes of interaction between authors and LLM assistants and add mechanisms to mitigate the risks associated with using LLMs (e.g. by recording the actor, human or LLM, that made the change to a document). Much of the code in the v1 branch will be reused (after some tidy-ups and refactoring), so v2 is not a complete rewrite.

Our general strategy is to iterate horizontally across the feature set, rather than fully developing features sequentially. This will better enable early user testing of workflows and reduce the risk of finding ourselves painted into an architectural corner. So expect initial iterations to have limited functionality and be buggy.

We'll be making alpha and beta releases of v2 early and often across all products (e.g. CLI, desktop, SDKs). We're aiming for a 2.0.0 release by the end of Q3 2024.

🟢 Stable • 🔶 Beta • ⚠️ Alpha • 🚧 Under development • 🧪 Experimental • 🧭 Planned • ❔ Maybe

The Stencila Schema is the data model for Stencila documents (definition here, generated reference documentation here). Most of the schema is well defined but some document node types are still marked as under development. A summary by category:

In v2, documents can be stored as binary Automerge CRDT files, branched and merged, and with the ability to import and export the document in various formats. Collaboration, including real-time, is made possible by exchanging fine-grained changes to the CRDT over the network. In addition, we want to enable interoperability with a Git-based workflow.

Interoperability with existing formats has always been a key feature of Stencila. We are bringing over codecs (a.k.a. converters) from the v1 branch and porting other functionality from encoda to Rust.

Format	Encoding	Decoding	Coverage	Notes
JSON	🟢	🟢
JSON5	🟢	🟢
JSON-LD	🟢	🟢
CBOR	🟢	🟢
CBOR+Zstandard	🟢	🟢
YAML	🟢	🟢
Plain text	🔶	-
HTML	🚧	🧭
JATS	🚧	🚧		Planned for completion. Port decoding and tests from encoda.
Markdown	⚠️	⚠️
R Markdown	🧭	🧭		Relies on Markdown; v1
Myst Markdown	🚧	🚧		In progress; PR
Jupyter Notebook	🧭	🧭		Relies on Markdown; v1
Scripts	🧭	🧭		Relies on Markdown; v1
Pandoc	🧭	🧭		Planned. v1
LaTeX	🧭	🧭		Relies on Pandoc; v1; discussion
Org	🧭	🧭		Relies on Pandoc; PR
Microsoft Word	🧭	🧭		Relies on Pandoc; v1
ODT	🧭	🧭		Relies on Pandoc
Google Docs	🧭	🧭		Planned v1
PDF	🧭	🧭		Planned, relies on HTML; v1
Codec Plugin API	🧭	🧭		An API allowing codecs to be developed as plugins in Python, Node.js, and other languages

Kernels are what executes the code in Stencila CodeChunks and CodeExpressions, as well as in control flow document nodes such as ForBlock and IfBlock. In addition, there are kernels for rendering math (e.g. MathBlock) and styling (e.g. StyledInline) nodes.

[TIP] Run stencila kernels (or cargo run -p cli kernels in development) for an up to date list of kernels, including those available through plugins.

Tools are what we call the self-contained Stencila products you can download and use locally on your machine to interact with Stencila documents.

Stencila's software development kits (SDKs) enable developers to create plugins to extend Stencila's core functionality or to build other tools on top of. At this stage we are planning to support Python, Node.js and R but more languages may be added if there is demand.

Language	Description	Status	Coverage
Python	Types and function bindings for using Stencila from Python	⚠️ Alpha PyPI
TypeScript	JavaScript classes and TypeScript types for the Stencila Schema	⚠️ Alpha NPM
Node.js	Types and function bindings for using Stencila from Node.js	⚠️ Alpha NPM

Making sure Stencila v2 is well tested, fast, secure, and accessible, is important. Here's what where doing towards that:

At this stage, documentation for v2 is mainly reference material, much of it generated:

• Schema
• Formats
• CLI

More reference docs as well as guides and tutorial will be added over the coming months. We will be bootstrapping the publishing of all docs (i.e. to use Stencila itself to publish HTML pages) and expect to have an initial published set in.

Although v2 is in early stages of development, and functionality may be limited or buggy, we are releasing alpha versions of the Stencila CLI and SDKs. Doing so allows us to get early feedback and monitor what impact the addition of features has on build times and distribution sizes.

curl --proto '=https' --tlsv1.2 -sSf https://stencila.dev/install.sh | sh

tar xvf stencila-*.tar.gz
cd stencila-*/
sudo mv -f stencila /usr/local/bin # or wherever you prefer

curl --proto '=https' --tlsv1.2 -sSf https://stencila.dev/install.sh | sh

tar xvf stencila-*.tar.gz
mv -f stencila ~/.local/bin/ # or wherever you prefer

docker pull stencila/stencila

docker run -it --rm -v "$PWD":/work -w /work --network host stencila/stencila --help

docker pull ghcr.io/stencila/stencila

python -m pip install stencila

poetry add stencila

npm install @stencila/node

yarn add @stencila/node

pnpm add @stencila/node

npm install @stencila/types

yarn add @stencila/types

pnpm add @stencila/types

This repository is organized into the following modules. Please see their respective READMEs, where available, for guides to contributing to each.

• schema: YAML files which define the Stencila Schema, an implementation of, and extensions to, schema.org, for programmable documents.

• json: A JSON Schema and JSON LD @context, generated from Stencila Schema, which can be used to validate Stencila documents and transform them to other vocabularies

• rust: Several Rust crates implementing core functionality and a CLI for working with Stencila documents.

• python: A Python package, with classes generated from Stencila Schema and bindings to Rust functions, so you can work with Stencila documents from within Python.

• ts: A package of TypeScript types generated from Stencila Schema so you can create type-safe Stencila documents in the browser, Node.js, Deno etc.

• node: A Node.js package, using the generated TypeScript types and bindings to Rust functions, so you can work with Stencila documents from within Node.js.

• prompts: Prompts for used to instruct AI assistants in different contexts and for different purposes.

• docs: Documentation, including reference documentation generated from schema and CLI tool.

• examples: Examples of documents conforming to Stencila Schema, mostly for testing purposes.

• scripts: Scripts used for making releases and during continuous integration.

Several Github Action workflows are used for testing and releases. All products (i.e CLI, Docker image, SKDs) are released at the same time with the same version number. To create and release a new version:

bash scripts/bump-version.sh <VERSION>
git push && git push --tags

Workflow	Purpose	Status
test.yml	Run linting, tests and other checks. Commit changes to any generated files.
pages.yml	Publish docs, JSON-LD, JSON Schema, etc to https://stencila.dev hosted on GitHub Pages
version.yml	Trigger the release.yml workflow when a version tag is created.
release.yml	Create a release, including building and publishing CLI, Docker image and SDKs.
install.yml	Test installation and usage of CLI, Docker image and SDKs across various operating systems and language versions.

Stencila is built on the shoulders of many open source projects. Our sincere thanks to all the maintainers and contributors of those projects for their vision, enthusiasm and dedication. But most of all for all their hard work! The following open source projects in particular have an important role in the current version of Stencila. We sponsor these projects where, and to an extent, possible through GitHub Sponsors and Open Collective.

	Link	Summary
	Automerge	A Rust library of data structures for building collaborative applications.
	Clap	A Command Line Argument Parser for Rust.
	NAPI-RS	A framework for building pre-compiled Node.js addons in Rust.
	PyO3	Rust bindings for Python, including tools for creating native Python extension modules.
	Rust	A multi-paradigm, high-level, general-purpose programming language which emphasizes performance, type safety, and concurrency.
	Serde	A framework for serializing and deserializing Rust data structures efficiently and generically.
	Similar	A Rust library of diffing algorithms including Patience and Hunt–McIlroy / Hunt–Szymanski LCS.
	Tokio	An asynchronous runtime for Rust which provides the building blocks needed for writing network applications without compromising speed.

We wouldn’t be doing this without the support of these forward looking organizations.

Thank you to all our contributors (not just the ones that submitted code!). If you made a contribution but are not listed here please create an issue, or PR, like this.