Git Product home page Git Product logo

firm's Introduction

Firm

Checks

The Firm is a collection of Nedryland components that adds the notion of a "function".

Developer Setup

Firm is a Nedryland project.

First, install nix.

A development shell for any component can then be obtained by running

$ nix-shell -A <component> # component can be for example avery

Names of all components can be found in project.nix in the repo root.

The Function

A function can be described as a self-contained unit that takes inputs and produces outputs. However, it is not a function like in most programming languages in that it takes a set of inputs and then execute until completion and then produces a single return value. Instead, a function in this sense has multiple asynchronous inputs and multiple asynchronous outputs. A better way to think about a function might be as a node in an asynchronous node graph.

Execution of functions

A function is executed in something called an "execution environment". The function itself specifies which execution environment it needs and Avery then tries to find a matching one. An execution environment is simply something that accepts the function code (does not have to be actual code) along with its' inputs and executes it. Note that a function itself can also be an execution environment for other functions. Examples of execution environments could be WASI (for running in the Web Assembly System Interface, Python (for running python code on the host), etc.

Components in the repo

Avery

Avery is the heart of the functional pipeline. It is responsible for housing the base execution environments that need to run on a host OS, downloading function code and starting the execution. Avery also has a local development registry used for quick iteration when working on functions.

Bendini

Bendini is a CLI interface to Avery. It has functionality for registering, listing and executing functions.

firmRust

A Rust library that houses utility functionality for Rust functions targeting the WASI execution environment. Contains helpers to get inputs, set outputs and set errors etc.

firmTypes

Rust and Python type library for communicating via protobuf.

Nedryland Extensions

extensions/nedryland/function.nix contains a Nedryland extension that adds the notion of a "function" microservice component. This component is added to the base set of components that Nedryland already supports.

extensions/nedryland/runtime.nix contains a Nedryland extension for creating runtimes for functions.

firm's People

Contributors

abbec avatar k3rps avatar sakarias88 avatar simonrainerson avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

firm's Issues

As a pipe2 developer I want my semantic versions to be enforced upon registring

Acceptance Criteria

  • Any potentially breaking change forces and update to major version.
    • Deleting an input or output
    • changing and input or output type (if they can’t be connected any more)
    • changing execution environment

Other changes should result in either a minor version or a patch version

  • if checksum of the code is equal and tags are equal (and everything else is equal) reject it?

Structure "Function Folder"

Currently the function folder is free form. Instead it should be a context object of some sort with support for

  • One folder for this particular execution
  • A shared space that can be used as a cache for downloaded attachments and other things

Proto registry Filter offset and limit fields have the wrong type.

message Filters {
...
uint32 offset = 3;
uint32 limit = 4;
...
}

They should be uint64 since we use it as an usize.

  • Just change the fields in firm/protocols/functions/functions.proto.
  • Make sure the updated type compiles on all projects that depend on the proto type.

Use system credential store for token cache

Currently, the token cache is written to a JSON file to disk. This is a reasonable fallback but as a first option we should try to use a system credential store if there is one (on Windows there always is for example).

Add flags to bendini to decide what to do about version conflicts

An example we have today when we deploy "new" functions where we first register and upload attachments before checking if we can actually register the function. This means we upload orphaned attachments just because function registry fails.

We must always check if the function exists but we need a flag to decide what to do in case it finds an already registered function.

Right now a duplicate counts as a function with same name and version. The code might still be different though. This flag is for deciding the strictness of the check. It could for example compare the checksums between the local version and the already uploaded version. Skip all duplicates, skip duplicates with matching checksum, skip no duplicates. --skip-duplicates all/matching-manifest/matching-checksum/none

As a Bendini user I want to be able to follow output when executing a function

Description

Currently, we use files to do stdout/stderr redirection in Avery. We should instead create a struct that implements WasiFile and can read stdout/stderr pipes and then write to file and stream to network.

Acceptance Criteria

  • Bendini has a --follow flag that shows output (stdout+stderr) of the running function
  • The --follow flag works over the network. I.e. the function is executed on another computer than bendini.

Replace order direction with reverse flag.

Today we have a field called order_direction in our ListRequest. It can be ascending or descending. This is not super nice for API users since they can decide what to sort on as well.

There are expectations on if things should be sorted ascending or descending depending on that field your sort on so it doesn't make sense to give the api user the choice.

For example name is expected to be ascending where version is expected to be descending.

We must decide a sensible default for asc and desc and instead provide the api user with an additional field to reverse the list results instead.

Avery with runtimes assumes package

Avery with runtimes assumes it should output two things, windows and package. This means that any other os needs to alias itself to either package or windows to work. The function should take the target directly and not make any assumptions of what is available.

Remove streaming upload

We've decided it's not really worth having two endpoints for uploading attachments. Kill the streaming grpc one.

Change how login works in Avery

Currently Avery is responsible for opening the browser window. This forces us to set the DISPLAY variable on unix platforms for something that is really a background service. The browser window should instead be opened by foreground clients like Bendini. Avery can send back an error code when acquiring credentials that need login with information on what browser page to open up. Bendini then opens the browser page and completes the login flow and after that tries to re-acquire the credentials.

Installer should not use slog to communicate with user

The user interface of firm-installer should not use a logging framework to print to the terminal. The buffered and asynchronous nature of it will make adding user features like progress bars or other interactive features harder than it should. It is also not what a logging framework is intended for. Writing down a log file (or using the event log) is of course ok.

Release shell changelog multiline comments are converted into separate comments.

In the case where a change is described in multiple lines such as the following example.

## Unreleased

### Added
- Added a seriously long line to the change log
  with a very descriptive description.

When we go into the release shell and prepare the changelogs and move it to the Firm changelog double lines are converted into separate items like in the example below.

## Firm [1.1.1]

### Added
- Added a seriously long line to the change log
- with a very descriptive description.

This is an unexpected behavior. We tracked the problem down to the keepachangelog python library only working on lines and not considering that a comment might be on several lines. https://github.com/Colin-b/keepachangelog/blob/3538c5b4beb433293352d084f4df7bcce4d02bf1/keepachangelog/_changelog.py

Suggested solutions:
1: We don't make newlines in comments.
2: Patch the keepchangelog parser, send them a pull request (https://github.com/Colin-b/keepachangelog).

release shell: makeRelease could be better

The required github token does not work as a relative path, also it should accept either content or filepath. Check [ -f "$1" ] and take it as a filepath otherwise assume $1 is the content.

Rename protocols lib from gbk_protocols to something cooler

Requirement is that it must not contain a company specific name.

Start by going to firm/protocols/rust/Cargo.toml and change the package name.
The same applies to our python version of the project at firm/protocols/python/setup.py.

Lots of other projects are using these types so make sure it builds with all of them. Preferably have fixes for other ready projects in other merge requests before merging the changes to main. It's just so it's ready for update with our internal projects once it gets merged.

Avery only supports authentication for a single machine per user

You are on computer 1 and start avery for the first time (ignoring socket activation step) avery will create id_ecdsa.pem and id_ecdsa_pub.pem. Avery will then upload the public part to rolodex. Avery does not care if there is already an entry and overwrites the pub key in rolodex.

Now I remote in to computer 2 where I've already done those steps previously.
From this machine I use bendini to list functions on another users machine. I will fail to authenticate because the new pub key in rolodex does no longer match my local one since it got overwritten.

Suggested solution:
Support a list of public keys per user in rolodex.
Could do this change when we move auth out from avery.

Structure WASI APIs

The WASI APIs that we add need to be structured into different pieces. The pieces that can be implemented outside Avery should be implemented as a separate library and then the symbols are just exported in Avery or any other host. This would allow other hosts to use these APIs, for example an alternative Python executable. In that case, the wasi Python shims that we have should assume that something will provide the symbols needed to implement things like sockets, threads, etc.

The pieces that can only be implemented inside Avery should of course be implemented there but be part of a separate import/export namespace. This is for example support functions for node graphs (allocating streams, setting connections, queueing function execution etc.).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.