Git Product home page Git Product logo

vmware-labs / wasm-workers-server Goto Github PK

View Code? Open in Web Editor NEW
505.0 14.0 33.0 14.1 MB

🚀 Develop and run serverless applications on WebAssembly

Home Page: https://workers.wasmlabs.dev

License: Apache License 2.0

Shell 1.05% Makefile 0.24% JavaScript 6.35% Rust 75.61% Dockerfile 0.80% Python 0.76% Ruby 0.90% Go 7.76% HTML 0.11% SCSS 0.46% C 3.35% Zig 2.61%
wasm webassembly workers rust faas serverless

wasm-workers-server's Introduction

Wasm Workers Server

Develop and run serverless applications on WebAssembly 🚀

GitHub Workflow Status GitHub release (latest by date) GitHub GitHub contributors

workers.wasmlabs.dev


Wasm Workers Server (wws) is an open source tool to develop and run serverless applications server on top of WebAssembly. The applications are composed by multiple modules called workers. Each of these tiny modules is in charge of replying to a specific HTTP endpoint in your application.

When you start wws, it loads the existing workers in the given path or remote repository. You can write a worker in different languages like Rust, JavaScript, Go, Ruby and Python. The filenames and folders determine the final routes that will be served. For example, the index.js will reply to the / endpoint.

Getting started (5 minutes)

  1. Download and install Wasm Worker Sever:

    curl -fsSL https://workers.wasmlabs.dev/install | bash
  2. Create an index.js file with the following content:

    addEventListener("fetch", event => {
      return event.respondWith(
        new Response("Hello from Wasm Workers Server!")
      );
    });
  3. Run wws:

    $ wws .
    ⚙️  Preparing the project from: .
    ⚙️  Loading routes from: .
    ⏳ Loading workers from 1 routes...
    ✅ Workers loaded in 141.613666ms.
        - http://127.0.0.1:8080/
          => ./index.js
    🚀 Start serving requests at http://127.0.0.1:8080
  4. Access to http://127.0.0.1:8080.

Congrats! You just created your first worker 🎉. From this point, you can explore the different examples and guides:

Run in a container

If you don't want to install anything locally you can just run wws from the ghcr.io/vmware-labs/wws:latest container image. You only need to mount your project in the /app folder:

docker run --rm -v $(pwd):/app -p 8080:8080 ghcr.io/vmware-labs/wws:latest

Documentation

All our documentation is available at https://workers.wasmlabs.dev.

Features

You can find all the available features in the documentation. It includes dynamic routes, an in-memory K/V store, etc.

Language Support

Wasm Workers Server focuses on simplicity. We want you to run workers (written in different languages) safely in WebAssembly. For interpreted languages, we add different interpreters:

Language Support Requires an external runtime Issue
Rust No -
JavaScript No -
Go No #95
Ruby Yes #63
Python Yes #63
Zig No #144
PHP 🚧 No #100

To get more information about multi-language support in Wasm Workers Server, check our documentation.

Development

Prerequisites

To work with this project you will need to install:

Run the project

After installing the different prerequisites, you can run the development environment with:

$ cargo run -- --help

Wasm Workers Server is split into different Rust crates. The root project produces the wws, while every crate provides specific features.

Contributing

See CONTRIBUTING.md and CODE_OF_CONDUCT.md.

License

Wasm Workers Server is distributed under the terms of the Apache License (Version 2.0).

See LICENSE for details.

wasm-workers-server's People

Contributors

angelmmiguel avatar assambar avatar carrycooldude avatar dependabot[bot] avatar dierbei avatar ereslibre avatar flavio avatar gzurl avatar kbwo avatar mnafees avatar mtt-artis avatar narayanbhat166 avatar sea-grass avatar ssssota avatar taisukef avatar vmwghbot avatar voigt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wasm-workers-server's Issues

Allow Rust workers to return an array of bytes too

Rust workers return a http::Response object that it's lately serialized into a UTF-8 JSON string. Then, the JSON is returned to the main wws service via WASI stdout. Note that the current seralized JSON includes the body and any other related metadata such as headers and status code.

This beahvior limits the capabilities of the workers as some tasks may return non UTF-8 entities. For example, we may want to generate a binary file that can be represented with a Vec<u8> in Rust, but not as a String. For this reason, we plan to add support for an array of bytes (Vec<u8>) in Rust workers too. This will introduce a breaking change as the Rust kit assumes the worker will return a String.

Supporting multiple values in Rust is simple, although we still need to make it serializable to JSON. To accomplish this, we plan to encode non-String values with base64. This is an initial iteration as base64 will introduce a performance penalty. In the future, we will stop using JSON in favor of other formats that supports all the use cases without base64.

Define a common policy for dependency versions

Currently, we're pinning dependecy versions in different ways:

[dependencies]
wasmtime = "2.0.0"
wasmtime-wasi = "2.0.0"
anyhow = "1.0.66"
wasi-common = "2.0.0"
actix-web = "4"
env_logger = "0.9.0"
# ...

Some of them are pin to a patch while others pin to the major version. We should establish a common policy so the cargo update command behaves the same for all dependencies.

cc @ereslibre

Decouple language specifics from the runner

Currently, all the code related to run a worker in wws is defined in the src/runner.rs file. At this point, we only have two supported workers: Wasm and JavaScript. For this reason, maintaining this logic in a single file is doable. However, we plan to extend this support in the future.

The goals for this task are:

  • Split the current code in a separate runner module
  • Provide a clear API to defien and initialize a runner type (JsRunner, WasmRunner, etc)
  • Define the language specific in separate entities
  • Simplify the addition of new languages in the future

The KV data is flushed when a module fails to process a request

The KV store works with snapshots. Every worker gets a fresh version and returns an edited state. Then, the KV state gets overwritten by the new one.

When a worker run returns an Error, wws generates a default WasmOutput object with a generic error. This default output includes an empty kv state that may flush the original state.

To avoid this, wws should ignore states of failed workers run or provide create aWasmOutput instance with the initial KV value.

Related code

.unwrap_or_else(|_| {
WasmOutput::new(
"<p>There was an error running this function</p>",
HashMap::from([("content-type".to_string(), "text/html".to_string())]),
StatusCode::SERVICE_UNAVAILABLE.as_u16(),
HashMap::new(),
)
});

Add support for mounting folders in Wasm modules

Allow developers to mount folders in their Wasm workers. For that, we can use the available WASI features about filesystem. In Rust, this behavior is easy to implement as it will natively use the WASI library to interact with the filesystem.

This feature is useful for many different use cases as we may want to inject libraries or any other arbitrary file.

Design

By default, wws will ignore any file / folder that starts with _. This will prevent miss-configuration issues in which wws identifies files that are supposed to be mounted as workers.

Then, developers will configure the mounted folder as part of the worker configuration. This approach provides flexibility on:

  • Folders to mount in an specific worker
  • Mount point

The format of the worker configuration file will be:

name = "counter"
version = "1"

[[folders]]
  from = "_lib" 
  to = "/src/lib"
[[folders]]
  from = "_assets"
  to = "/src/assets"

JavaScript

For JavaScript, the current runtime we use (QuickJS) doesn't include any library for fs. Even we mount the folder inside, the JavaScript code won't be able to access it. This will require to research and discuss the best approach to bring these features to javascript workers. We will do it as a separate task.

Add the prefix option to the CLI

The --prefix option allows users to set a prefix for all the routes. This is useful when a specific application is mounted in a subpath of a server. Currently, this is reproduced by using a folder wws ./examples, but this will change after we fix #30.

The --prefix will be optional and expect a String as type.

Allow dynamic parameters in URLs

The parameters will be provided based on the filename. As other projects using this approach do (like NextJS), we will consider [id].wasm files as dynamic URLs with an :id parameter.

We also need to consider different scenarios. For example, a /a/b route may be replied by:

  • /a/b.js
  • /a/[id].js
  • /[id]/b.wasm
  • /[id]/[other].wasm

We need to establish a priority. The lower of the returned number, the more priority it has. This number is calculated based on the number of used parameters, as fixed routes has more priority than parameted ones.

To avoid collisions like [id]/b.wasm vs /a/[id].js. Every depth level will add an extra +1 to the score. So, in case of [id]/b.wasm vs /a/[id].js, the /a/b path will be managed by [id]/b.wasm.

This parameter information will be injected to modules as part of the requests snapshots, allowing to create a new set of applications.

Add automation to build the different binaries in every release

For every release, we should build and upload the different binaries. With this automation we ensure people see where those files come from and how they were built. We will upload the binaries and also the SHA256 of every of them. We will print these values in the job too.

For now, we want to compile it for these different platforms:

Required

  • (macOS) Darwin x86
  • (macOS) Darwin ARM64
  • Linux x86

Optional

  • Linux ARM64
  • Windows 64
  • Raspberry Pi ARM8
  • RISC-V (if possible)

Support static assests in handlers

Some handlers return HTML code that may require static assets such as JavaScript (client-side) and CSS. Currently, it can be injected in the handler, however is much more convenient to provide a way to set static files.

We can provide this feature using two approaches:

  • Mount generic folders (using WASI) in the handler and provide libraries to manage those files
  • Add a new data.folder configuration property that will serve the files directly using the Rust server

The final approach is still under discussion.

Add support for SQL queries

Allow workers to interact with a SQL database via host functions. The worker will be able to connect to a DB that it's configured in the TOML file. This will allow to have a great data persistance and enable new use cases.

We want to ensure the capability-based model is respected here too. So, in the configuration we may state what are the operations a certain worker can do. For example, read, update, etc.

For now, this is a proposal. We will refine it over the time.

Provide full application examples

Currently, we provided a set of examples in the project. These examples show the basics of Wasm Workers Server, although we don't include yet complete applications.

It would be great to provide more complete applications based on existing frameworks such as Astro, NextJS or Fresh. This will help to understand the current limitations and improve the project in general.

This issue is also created to get new ideas. If you have any specific application you would like to see here, feel free to drop it!

Check required lang runtimes when running `wws` (from `.wws.toml`)

As part of the wws initialization, it will retrieve the data from the .wws.toml file (#67). Then, it will use the runtime manager (#66) to check if the required runtimes are installed. If they are missing, it will install them by default in the current folder.

This step is required as runtimes may add more supported extensions to wws. For example, by default wws cannot support workers written in Python. However, they may add a Python runtime and wws will start managing .py files as workers.

If this step fails for any reason, wws will show a warning and will continue with the remaining supported extensions.

Configure CORS headers globally

Cross-Origin Resource Sharing (CORS) is a series of security policies to avoid a web browser fetching resources from a different domain. By default, CORS will block any request that a website makes to a different domain.

However, servers can set the CORS HTTP headers to indicate the browser they are fine to process the request. CORS can be configured using several headers. Some of them provides a global policy while others helps to add granularity to the requests:

  • Access-Control-Allow-Origin: http://example.com
  • Access-Control-Allow-Methods: POST, GET, OPTIONS
  • Access-Control-Allow-Headers: Content-Type
  • Access-Control-Max-Age: 10000

I recommend you to check the Cross-Origin Resource Sharing (CORS) documentation about the different headers and behaviors.

On wws, we plan to provide two different approaches to set the CORS policy:

  • With a global --cors option. This option will configure the Access-Control-Allow-Origin header. It will be applied to all requests
  • Every worker can set its own headers. In this way, we allow workers to customize the response. This approach has preference to the --cors option. If both are set, the worker value will be sent

Create a container image and build it

We want to simplify the usage of wws as much as we can. Since containers are quire popular these days, we want to provide a related wasm-workers-server container. This container will include the Rust binary and the minimum extra packages required.

We will compile this container for the following platforms:

Required

  • amd64 / x86
  • arm64

Optional

  • armv7

The container will use the /app folder as the entrypoint for the diffrent worker files. So, the goal is that the following command will spawn the server directly:

podman run --rm --name wws -v ./my-workers:/app projects.registry.vmware.com/wasmlabs/containers/wasm-workers-server 

index files are not properly mapped to API paths

Files with an index.* filename should be map to the current folder when retrieving the routes. For example:

./index.js => /
./api/index.wasm => /api/

Currently, these files are not properly mounted and wws is returning a 404 when trying to access to those routes.

Create a struct to manage runtime metadata files

The purpose of this struct is to act as a "Runtime Metadata Manager". The main tasks of this struct will be:

  • Load metadata files for new runtimes from the local filesystem or a remote one
  • Download related resources and store them using the runtime::Data struct. We will need to add new methods to this one
  • Validate the different files using the associated checksums

This can be split in multiple structs / modules if required.

Add an option to ignore certain folders

When running wws, it will fetch all the files in the given directory and subfolders. Sometimes, we may run the CLI in a folder, but we don't want to check all the folders inside it. For example, we want wws to be able to ignore node_modules folders in certain cases.

To make it configurable, we plan to add a new option to the CLI called --ignore. This --ignore option can be used multiple times to ignore several folders:

wws --ignore node_modules --ignore tmp .

Clap supports already supports this feature. The complexity of this issue comes from:

  • The support of Windows and Unix systems
  • Usage of Glob crate

Note that if we use the same approach we do for skipping the public folder, Glob will still list all the files in those folders. Folders lik node_modules can cause a performance issue as they may contain a big number of supported files (.js).

Fix wrong routes when passing a path as an argument

From #3 and #29

When passing a path to wws (like wws ./examples), it should load the files starting from that route and assign the HTTP endpoint from it too. Currently, this is not the case and running wws ./examples when ./examples/handler.js exists will assign the /examples/handler route instead of the desired /handler.

The issue is located at:

let parsed_path: String = api_path

Here we are removing base_path from api_path, but in this case the string doesn't match properly:

api_path: examples/handler.js
base_path: ./examples/

In addition to that, in #3 @assambar described an issue when the folder does not include the trailing /. The trailing slash should be optional, so we need to fix that issue too.

Add the new command to install a language runtime

With the addition of this feature, we plan to add a new command for wws. Currently, it doesn't include any command as it assumes a default "run" command.

This issue will include a new command which it's still under discussion. We're thinking on a runtimes command and several subcommands like install, uninstall, etc.

Regarding the main command, we still want wws to behave the way it works now. So, we don't plan to add an explicit run command at this point.

Required commands

  • Install a new runtime
  • Uninstall a runtime
  • List installed runtimes
  • Check local installation

Parameters

These commands will include a parameter to refer to a specific runtime metadata file.

Load and run workers with installed runtimes

Now that developers can install new runtimes (#68), wws should pick the installed ones and run workers based on it. For that, it will identify the new file extensions to look for files and then, prepare the worker and configure the WASI context.

If a language runtime is present in the .wws.toml config (see #67) and it's not installed, wws will show a warning message indicating they are not installed and some workers won't run.

This will require a new src/runtimes/runtime/external.rs struct that properly configures the installed language runtime and the WASI context. All this must come from metadata as developers may create their own set of runtimes.

Add support for environment variables

Environment variables will be configured in TOML files. Every worker will define its own set of environment variable, which will be accessible only for them.

As a tentative API, we will include a new .env resource in the TOML config file:

name = "test"
version = "1"

[env]
MY_ENV_VAR = "Testing!"

Then, the environment variables will be passed to the inner modules via WASI. The environment variables will be accessible via default env methods from the different languages.

Implement the runtime metadata file (toml)

These kind of files will be managed by wws, so it requires a representation in the source code. For consistency with the project, we chose the format as TOML.

The purpose of them is to "teach" wws how to use certain runtime and which files can be run with it. For this reason, it must include information about the name, version and binary, but also polyfills, arguments and environment variables.

Content

Property Type Description
spec_version u32 The version of this file
name String The runtime name
version String The runtime version number
binary String URL to download the binary
binary_checksum String Binary checksum
extensions String[] Associated file extensions to this runtime
polyfill (optional) String An associated source file with polyfills. It will be mounted inside lib folder using WASI
polyfill_checksum (optional) String The checksum to validate the polyfill file
template (optional) String An optional template in case the source code needs to be modified. For example, to load a library or to add a function call. The provided code will be available in the {code} key
template_checksum (optional) String The checksum to validate the template file
args String[] Arguments to pass in the WASI context
envs (optional) HashMap<String, String> Default environment variables
folders (optional) HashMap<String, Object> A list of folders to mount with this runtime. It includes URL to download a compressed file, mount point and if decompression is required.

We may change these names or add some of them in the future.

It's related to #63

Rename Rust kit attribute macro to worker

In the initial versions of this project, we used handler as the attribute macro for the Rust kit. However, our documentation references worker as the right term for this kind of Request / Respose functions.

This will be a breaking change. Since we are still defining the final API, it's fine to apply those at this moment.

To complete this task you will need to:

  • Rename the kits/rust/handler project to kits/rust/worker. This includes all references in Cargo.toml, kits/rust/Cargo.toml, kits/rust/handler/Cargo.toml and any place in which we import it.
  • Change the macro attribute name and all its references in kits/rust/worker
  • Update the exported name in kits/rust/src/lib.rs
  • Update all the examples in the documentation
  • Update the examples in the examples folder

Error building the binaries after adding reqwest

The GitHub Action automation is failing to build the binary due to missing dependencies. After adding reqwest on #74, the project now requires an OpenSSL installation. This is required by the openssl-sys crate, required by reqwest.

Here you can see the error: https://github.com/vmware-labs/wasm-workers-server/actions/runs/4014946928/jobs/6896087632

run pkg_config fail: "pkg-config has not been configured to support cross-compilation.\n\nInstall a sysroot for the target platform and configure it via\nPKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH, or install a\ncross-compiling wrapper for pkg-config and set it via\nPKG_CONFIG environment variable."

  --- stderr
  thread 'main' panicked at '

  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

In their documentation, they provide certain commands to install the required dependencies. I will reuse this information and test the build process works properly.

Create the documentation site

Documentation is a critical piece of every project. wws will have its own documentation site based on docusaurus. For the initial version, this is the information we want to include:

  • Introduction to the project
  • Getting started
  • K/V store (data plugins)
  • How to write your first JavaScript worker
  • How to write your first Rust worker

The site will be deployed as a subdomain of wasmlabs.dev.

Adding multi-language support (multiple language runtimes)

This issue tracks an ongoing effort. The implementation will be done in the related issues you can find at the end

Wasm Workers Servers aim to create serverless applications based on different languages. As a starting point, we added support for Wasm modules (compiled from Rust) and JavaScript (by embedding the QuickJS runtime). All these workers run inside the WebAssembly sandbox.

For v1.0.0, the goal is to extend the number of languages by allowing different language runtimes. Even though the size of other runtimes like Ruby and Python are not big (see our python and ruby releases), including them in the main wws is not an option.

In addition to that, we want to make wws extensible. Any developer may implement new language runtimes and use them. For this reason, wws will allow to import runtimes from different sources.

CLI experience

In this section, I will explore the desired CLI experience for this feature. All the examples assume a project has two or more workers. One is a "native" worker (.wasm) file that doesn't require any runtime. The other file requires a runtime to run (.rb or .py). 

$ tree ./examples/multi-lang 
./examples/ multi-lang 
├── hello.rb 
├── basic.toml 
└──  basic.wasm 

Repositories

A repository is an index file that points to different runtimes. It can be hosted in a git repository or in any service that can expose files. wws will retrieve the index file from the given repository. This index file contains pointers to the available runtimes. The repository is configurable via arguments, so anyone can develop their own repository. If no --repo parameter is provided, the WebAssembly Language Runtimes will be the default repo. 

The CLI experience would be: 

$ wws runtimes list 
NAME            VERSION         EXTENSION       BINARY          STATUS 
ruby            3.2.0           .rb             https://...     not installed 
ruby-std        3.2.0           .rb             https://...     not installed 
python          2.3             .py             https://...     not installed 

$ wws runtimes install ruby-std 3.2.0 
Pulling the runtime binary... 
Installing it in ./wws/runtimes/wlr/ruby-std/3.2.0/... 
Validating checksum... 
Done!

$ wws ./examples/multi-lang 

To use a different repository, you can use the --repo flag:

$ wws runtimes list --repo=https://my-repo... 
NAME            VERSION         EXTENSION       BINARY          STATUS 
ruby            3.2.0           .rb             https://...     not installed 
ruby-std        3.2.0           .rb             https://...     not installed 
python          2.3             .py             https://...     not installed 

$ wws runtimes install ruby-std 3.2.0 --repo=https://my-repo... 
Pulling the runtime binary... 
Installing it in ./wws/runtimes/my-repo/ruby-std/3.2.0/... 
Validating checksum... 
Done! 

$ wws ./examples/multi-lang 

You may point by default to a specific repository by setting up an environment variable. If the --repo argument is also present, it will take precedence to the environment variable. 

Runtime metadata files

wws will read runtime metadata files. These files include all the information required to install and use a language runtime in wws. For example, they will have a name, version, extension they can manage and a pointer to the Wasm binary. It may include other configuration parameters to ensure wws knows how to run it. You have the full list available in #65.

These files can be hosted anywhere. We will provide the default set of runtimes from the WebAssembly Languages Runtime project. However, any other file repository can be used instead in wws.

Index file

To simplify the installation process, wws will rely on an index file that will live in the repository the CLI is configured to point to. This file points to the different runtimes available on that repository, so wws can list them and install easily. By default, wws will point to the WebAssembly Languages Runtime project, although developers can change it via flag and environment variable.

Project metadata

For reproducibility, wws will create a local .wws.toml file with all the installed runtimes. This file contains all the details to retrieve the runtimes in the future. When it's committed to the application repo, wws will install the required runtimes automatically.

Issues

This list will be updated during development as well as referencing GitHub issues

Add cargo fmt and clippy to every PR

We should check the PRs follow Rust best practices and a consistent code style. This can be done by the popular Rust projects fmt and clippy. These two tools will be run against the project on every PR. In fact, the PR that integrates this feature may require code changes to make the code compliant with their expectations.

Vars must be optional in the configuration file

As part of #34, we introduced a new [vars] configuration section to set environment variables for a worker. This information must be optional, although it's required now. We should make it optional and avoid failing to load the configuration if it's not present.

Create the initial repository index

From v1.0.0, wws relies on external repositories to fetch the available language runtimes. By default, this project will host the initial metadata:

  • Index
  • Polyfills
  • Wrappers

We will host it as a static assets, while relying on GitHub releases for the different artifacts. Initially, we will generate the index manually based on the WebAssembly Language Runtimes project. We will automate this process in the future by adding several automations in both Language Runtimes and Wasm Workers Server projects.

For the URLs, we plan to segment it by version even it's included in the index spec. This will allow in the future to add redirections and any required change to maintain a better backward compatibility in case we need to change it. So, the final URL will be:

wws will use this metadata unless the user specify a different repository. The name of this repo will be wasmlabs.

Versioning

In the repository metadata we have to version both runtimes and also other files such as polyfills and wrappers. These files have two different, but related versions. In general, a polyfill will work for the major version of a language runtime. However, we may add new features that requires to update these polyfills.

Since the runtime metadata is stored locally in the .wws.toml file, updating a related file (polyfill or wrapper) and the index.toml entries will break developer environments that have references to the previous file version. For this reason, we musn't change polyfills and wrappers after releasing them.

If we need to do so, we will:

  • Create the new files under a different revision (for example, /repository/v1/files/ruby/3-1/poly.rb)
  • Create a new release pointing to the new files
  • Mark the previous one as yanked

This will ensure new installation gets the latest version and existing ones work properly. We will add a warning message asking the users to update to the latest version if this happens.

Add tags to the runtime metadata definition

Tags will simplify the installation of language runtimes. It allows repositories to set aliases to the runtime versions to simplify the naming. For example, you may name a specific version as latest or by the major version (3).

Aliases must be unique per repository + name, although they may change. You may have multiple latest for different runtime names. You can also update aliases when a new version is released. We will provide tooling in the future to validate it. For now, if an alias is duplicated, wws will use the first one.

Note that when a runtime is installed, the metadata is stored in the .wws.toml file. This locks the version of the runtime at that time. If a developer wants to use the latest version for a given alias, they may need to reinstall the runtime it.

New field

Property Type Description
tags String[] Version aliases to simplify installation

I also updated #65 with the new field

proposal to add an Extism runtime

Extism supports a number of languages quite well, and has a low-level interface which could be adapted to the Reqeuest/Response pattern of wws.

Essentially, the wws HTTP server layer would:

  1. Handle incoming request
  2. Copy fully buffered HTTP body as string/[u8]
  3. Pass it to Extism.plugin.call("request", body)
  4. Use the return value from that call as the Response body

Then any Extism plugin could work as a worker. The plugin would only need to export a request function for wws to call, reading the input Request, and setting the output as the Response.

Since wws is in Rust, our Rust SDK would be the implementation path: https://extism.org/docs/integrate-into-your-codebase/rust-host-sdk/

And then any of our official PDKs (in 7 languages) could be used to write wws Workers:
https://extism.org/docs/concepts/pdk/

If there is interest, maybe we can collaborate on this. Thanks!

Create the container image in GitHub Actions

wws is also provided as a container imager for convenience. However, this image is still generated manually. Since we plan to start using GitHub registry to publish them, we want to start using GitHub actions to publish it. These are the tags that should be created:

  • A preview tag on every commin in main
  • A X.X.X tag per new released version
  • A latest tag pointing to the latest released version

Fix inline arguments for println! and format!

After upgrading to the latest version of Rust and clippy, I'm getting the following warning in println! and format! statements:

error: variables can be used directly in the `format!` string
  --> src/runtimes/metadata.rs:39:13
   |
39 |             println!("Err: {:?}", err);
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
help: change this to
   |
39 -             println!("Err: {:?}", err);
39 +             println!("Err: {err:?}");
   |

This comes from a new linter rule that was introduced in Rust 1.66: uninlined format args. The warning is an error in the example as we're forcing this via parameters (cargo clippy -- -D warnings). The fix is quite simple and the code is more readable when adding the arguments inline.

Improve path management in wws

The wws server relies on the local filesystem heavily. The way we are managing paths at this point may cause issues when running the server in Windows environments. In addition to that, the way we check for multiple extensions is quite complex and error prone.

The goal for this issue is to:

  • Define available extensions in a constant (in the future we may mek this configurable)
  • Limit the conversion of paths to String and &str to printing data
  • Rely on Path and PathBuf for any path management requirement

Improve JavaScript error management

JavaScript handlers are run in WebAssembly using a module based on QuickJS. When an error appear, it's complex to debug as we are running an interpreter inside a Wasm module while adding a compatibility layer.

To improve this experience, the JavaScript error must be captured on the rust server and printed.

It's still under discussion if we want to show these error traces always or behind a CLI flag.

Skip any file in the public folder when loading routes

In #7, we added support for static assets using a special folder called public. This is causing an unexpected behavior when public contains static .js files for the UI. wws is loading those as workers when it should be considered static.

When loading routes in initialize_routes, we should ignore any file starting with public.

Update Json5 documentation dependency

Json5 is required by an internal docusaurus dependency:

~/W/o/w/docs ❯❯❯ npm ls json5
└─┬ @docusaurus/[email protected]
  ├─┬ @babel/[email protected]
  │ └── [email protected]
  ├─┬ @docusaurus/[email protected]
  │ └─┬ @mdx-js/[email protected]
  │   ├─┬ @babel/[email protected]
  │   │ └── [email protected] deduped
  │   └─┬ [email protected]
  │     └─┬ @babel/[email protected]
  │       └── [email protected] deduped
  └─┬ [email protected]
    └─┬ [email protected]
      └── [email protected] deduped

We should update this library to >=2.2.2 to fix the CVE-2022-46175.

Optimize the size of the container image

Currently the image is based on debian:bullseye-slim which is around 100MB. This is a huge overhead just for the sake of the wws binary.
We have to investigate if we can do an image based on something smaller, say alpine.

Cannot build examples/rust-basic

Sorry for the boring point, when running cargo build, the errors below was occurred.

$ cargo build --target wasm32-wasi --release
   Compiling rust-basic v0.1.0 (/workspaces/wasm-workers-server/examples/rust-basic)
error: invalid format string: expected `'}'`, found `'m'`
  --> src/main.rs:19:16
   |
19 |         body { max-width: 1000px; }
   |              - ^ expected `}` in format string
   |              |
   |              because of this opening brace
   |
   = note: if you intended to print `{`, you can escape it using `{{`

I think { and } should be replaced to {{ and }}.

    <style>
        body {{ max-width: 1000px; }}
        main {{ margin: 5rem 0; }}
        h1, p {{ text-align: center; }}
        h1 {{ margin-bottom: 2rem; }}
        pre {{ font-size: .9rem; }}
        pre > code {{ padding: 2rem; }}
        p {{ margin-top: 2rem; }}
    </style>

Thank you.

can't start wss in android (termux).

Hi I'm try using https://termux.dev to run this, but I'm facing this...

/wws . 
⚙️  Loading routes from: .
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }', /home/ereslibre/.cargo/registry/src/github.com-1ecc6299db9ec823/wasmtime-jit-1.0.1/src/code_memory.rs:63:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace```

Implement the runtime metadata file (toml)

These kind of files will be managed by wws, so it requires a representation in the source code. For consistency with the project, we chose the format as TOML.

The purpose of them is to "teach" wws how to use certain runtime and which files can be run with it. For this reason, it must include information about the name, version and binary, but also polyfills, arguments and environment variables.

Content

Property Type Description
name String The runtime name
version String The runtime version number
tags String[] Version aliases to simplify installation
status Enum The status of the given runtime: active, yanked, deprecated
binary RemoteFile A remote file that points to the runtime binary
extensions String[] Associated file extensions to this runtime
polyfill (optional) RemoteFile An associated source file with polyfills. It will be mounted inside lib folder using WASI
wrapper (optional) RemoteFile An optional wrapper file in case the source code needs to be modified. For example, to load a library or to add a function call. The provided code will be available in the {code} key
template (optional) RemoteFile An optional file to bootstrap new workers from the wws CLI.
args String[] Arguments to pass in the WASI context
envs (optional) HashMap<String, String> Default environment variables

Still under discussion

Property Type Description
folders (optional) HashMap<String, Object> A list of folders to mount with this runtime. It includes URL to download a compressed file, mount point and if decompression is required.

We may change these names or add some of them in the future.

Implement the index metadata file

The purpose of the index.toml metadata file is to provide simplify the installation of new runtimes in wws. This file will live in a repository and point to the different runtimes available on it. By default, wws will rely on the WebAssembly Languages Runtime project. However, any other file repository can be used instead in wws.

The file format will be TOML to be consistent with the other configuration files.

Fields

Name Description
version The version of the index file
runtimes The list of runtimes. Every runtime contains all the fields defined at #65.

Improve overall error capturing

This issure requires a better definition once we plan to work on it

wws heavily relies on the anyhow for a simple error management. In certain cases, we're capturing the errors and mapping it to more appropiate messages. However, this focuses on the errors we want to show in the CLI.

We want to create a set of specific error types to capture and manage them in code properly. Instead of bubbling generic errors, we plan to throw a set of specific ones we can manage later.

Add an initial GH action to build the project

As a first integration with GitHub actions, we plan to add a basic build process for the main wws binary. This workflow will build the project on every main commit and pull request targetting this branch.

For this, we will use the default action suggested by GitHub

Implement the `.wws.toml` file

The .wws.toml file contains information about the current project and the required language runtimes. For now, this file will be auto-generated and we don't expect developers to modify it.

It includes the pointer to the metadata file and it's checksum. When this file is committed to the repo, wws will detect it, download the runtime metadata and install the missing ones.

We may add this as part of the default wws command or add a separate one just for installing missing runtimes.

Content / Format

For the v1, the file will contain only the basics about the installed runtimes / repositories. I wouldn't go further with the implementation as at this point, wws doesn't really need more data.

So, the format for the .wws.toml file will be:

# Refers to the version of this file
version = 1

[[repository]]
name = "wlr"

[[repository.runtimes]]
# Runtime metadata
name = "ruby"
version = "3.2.0"
binary = { url = "...", checksum = "...", filename = "..." }
# ...
[[repository.runtimes]]
# Runtime metadata
name = "python"
version = "3.2.0"
binary = { url = "...", checksum = "...", filename = "..." }
# ...

At this point, you may noticed similarities and differences with other popular project definition files like package.json or Cargo.toml. In both Rust and NodeJS, the project definition differs from their lock files (Cargo.lock and package-lock.json). The main reason is to provide an friendly file for developers (definitions) and a full spec with all the requirements for the tools (lock files).

However, in this case we don't expect you to install many language runtimes. Also, these runtimes are self-contained. A single language runtime will be defined by an unique block. They don't have dependencies you need to fetch and refer in a lock file.

For this reason, the .wws.toml will include both definitions (runtime names and version) and "lock" metadata (URLs, checksums, etc). If in the future this file grows and it's not maneagable anymore, we will open a discussion about splitting it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.