For proto to become the best toolchain manager in the ecosystem, it needs to support plugins. Plugins will allow consumers to implement custom languages (no need for it to be in proto itself), possibly implement non-language tools (like CLI binaries), hook into life-cycles, and more.
Enabling a plugin
Plugins can be enabled on a per-directory basis through the .prototools
configuration file. This allows companies and projects to pin the plugins required for their repository to function correctly.
Plugins can also be enabled in ~/.proto/config.toml
for the current user.
Since both of these configuration files are TOML, the plugins will be defined in a [plugins]
table, which requires a mapping of tool name to plugin locator. A plugin locator is a string that combines a unique protocol to a location in which to fetch the plugin.
[plugins]
kotlin = "protocol:location"
The following protocols are supported:
schema:
protocol
The schema:
protocol defines the location of a TOML plugin schema. The location can be a file path (relative from the parent config), or an https URL.
[plugins]
# By path
kotlin = "schema:../../tools/kotlin.toml"
# By url
kotlin = "schema:https://raw.githubusercontent.com/kotlin/kotlin/master/proto-schema.toml"
# By url using a specific version
kotlin = "schema:https://raw.githubusercontent.com/kotlin/kotlin/9ec2503ce3324d637065f8f657b6efea0bbb81b7/proto-schema.toml"
source:
protocol
The source:
protocol defines the location of the compiled WASM plugin file. The location can be a file path (relative from the parent config), or an https URL.
[plugins]
# By path
kotlin = "source:../../tools/kotlin.wasm"
# By url
kotlin = "source:https://github.com/kotlin/kotlin/releases/latest/download/proto-plugin.wasm"
# By url using a specific version
kotlin = "source:https://github.com/kotlin/kotlin/releases/download/v0.5.0/proto-plugin.wasm"
github:
protocol
The github:
protocol defines the GitHub repository to download the WASM plugin file from. This protocol requires that the WASM file is added as an asset to each release, using a specific naming scheme.
[plugins]
# Default to latest
kotlin = "github:kotlin/kotlin"
# Using a specific version
kotlin = "github:kotlin/[email protected]"
Implementing a plugin
Because proto is written in Rust, supporting a runtime based plugin system is difficult. The only native solution to this problem is WASM plugins, which have the following concerns:
- Tooling to compile and build the WASM plugins must be stable.
- Plugins should be able to be written in any language that supports WASM, not just Rust.
- WASM is a foreign/unknown concept to many. Will most likely have a learning curve.
To mitigate these concerns and to ease the plugin implementation process, we’ll actually be supporting 2 types of plugins. The standard WASM plugin, and a TOML configuration schema plugin.
Configuration schema plugin
The schema plugin is a TOML file that defines all the necessary information for downloading, installing, and executing a tool. Since this is a configuration file, and all settings are defined literally, it only provides the bare minimum of functionality, and only supports pre-built tools and not build from source.
If more advanced functionality is required, then a WASM plugin will need to be used instead.
Here’s an example of what this schema may look like, using Node.js:
# PROOF OF CONCEPT, NOT FINAL!
name = "Node.js"
bin-name = "node"
[platform.os]
linux = "linux"
macos = "darwin"
windows = "win"
[platform.arch]
arm = "armv7l"
aarch64 = "arm64"
x86_64 = "x64"
x86 = "x86"
[resolve]
versions-url = "https://nodejs.org/dist/index.json"
version-selector = ".version"
[install]
archive-prefix = "node-v{version}-{os}-{arch}"
download-url = "https://nodejs.org/dist/v{version}/node-v{version}-{os}-{arch}{ext}"
checksum-url = "https://nodejs.org/dist/v{version}/SHASUMS256.txt"
[install.ext]
linux = ".tar.xz"
macos = ".zip"
windows = ".tar.xz"
[detect]
version-files = [".nvmrc", ".node-version"]
[execute.bin-path]
linux = "bin/node"
macos = "bin/node"
windows = "node.exe"
[commands]
install-global = "npm install -g {dependency}"
WASM plugin
For WASM plugins, we’ll be using Extism, a universal plugin system for multiple languages. On the host side (proto), we import the .wasm
plugin file and execute functions provided by the plugin. On the guest side (the plugin), plugins can be written in multiple languages, like Rust, JavaScript, and Go, all of which define and export functions that compile to .wasm
.
An example of what this may look like in Rust:
use extism_pdk::*;
use serde::Deserialize;
use std::env::consts;
#[derive(Deserialize)]
struct DownloadParams {
version: String,
}
#[plugin_fn]
pub fn get_download_url(input: String) -> String {
let params: DownloadParams = serde_json::from_str(&input).unwrap();
format!(
"https://nodejs.org/dist/v{version}/node-v{version}-{os}-{arch}{ext}",
version = params.version,
os = consts::OS,
arch = consts::ARCH,
ext = ".tar.xz"
)
}
And again in JavaScript (not tested):
export function get_download_url() {
const params = JSON.parse(Host.inputString());
Host.outputString(
`https://nodejs.org/dist/v${params.version}/node-v${params.version}-${process.platform}-${process.arch}.tar.xz`
);
}
Extism is a good choice for the following reasons:
- It provides libraries/SDKs for both plugin authors and for plugin loader (proto).
- It supports multiple languages and doesn’t lock authors into Rust.
- We’re not reinventing the wheel.
- It’s maintained by a company and supports a community.
- Supports async functions.