suborbital / subo Goto Github PK

The Rust builder is quite slow when using third party crates because it needs to fetch them every time. We could cache the build directory to improve this.

https://docs.wasmer.io/ecosystem/wasienv

The 'build pipeline', such as the BuildContext, the template system, and the native/Docker toolchains should be refactored into an isolated module with a well-defined API so that it can be used in other non-CLI contexts.

This builder would just check that the wasm file exists, and if so, the build is done.

This would allow languages that subo doesn't support to specify language: wasm and have an external build system.

Once Reactr supports TinyGo, Subo should gain the ability to build a Runnable written in Go.

When building an app, Subo should attempt to build the Vektor router from the Directive's routes and if Vektor panics, Subo should return a build error.

Would love to be able to run subo create handler /foo and have it added to the directive.

Some potential flags: --method, --stream, --steps

I noticed that when I run subo build . --bundle and it fails to compile, it exits with 0.

More information:

...
error: build failed
🚫 failed to doBuild for helloworld: failed to RunInDir: failed to Run command: exit status 101
✅ DONE: helloworld was built -> /home/sven/.../important-api/helloworld/helloworld.wasm
✅ DONE: bundle was created -> runnables.wasm.zip

$ echo $?
0

When creating a Rust Runnable, a Cargo.toml needs to be added, otherwise tomltool uses the default file, which has the name hivew-rs, and then subo can't find the resulting wasm file and it fails.

Should be able to do subo dev release v0.1.0 to bump version, push tag, build Docker, create GH release, release notes, etc.

curl -Ls https://subo.suborbital.dev | sh

  ==> Downloading github.com/suborbital/subo/subo@master
  ==> Resolved version master to v0.0.21
  ==> Downloading binary for linux amd64

  Error downloading, got 500 response from server

The script is making a request to this URL: https://gobinaries.com/binary/github.com/suborbital/subo/subo?os=linux&arch=amd64&version=v0.0.21

I just followed https://atmo.suborbital.dev/getstarted and created an empty project.

When running subo dev I see the following:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "chdir to cwd (\"/home/atmo\") set in config.json failed: permission denied": unknown.
time="2021-03-07T13:57:15Z" level=error msg="error waiting for container: context canceled"
Error: 🚫 failed to run dev server: failed to Run command: exit status 126
Usage:
  subo dev [flags]

Flags:
  -h, --help          help for dev
      --port string   set the port that Atmo serves on (default "8080")

In the container configuration:

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/sven/.../important-api",
                "Destination": "/home/atmo",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

and my current directory:

total 8.0K
-rwxrwxr-x 1 sven sven  384 Mar  7 13:55 Directive.yaml
drwxr-xr-x 3 sven sven 4.0K Mar  7 13:55 helloworld

Steps to reproduce:

# Build a project
subo create project test

# Build the project docker image
subo build --docker

# Run the image
docker run com.surborbital.test:v0.0.1

# Try and fail to exit the container with CTRL + C
^C

Currently the Dockerfile is using the shell form for the ENTRYPOINT instead of the exec form (docs). This means the SIGINT get's picked up by the shell running Atmo and not by Atmo itself (more on this). If we switch to using the exec form then this issue will be resolved.

Happy to write up a PR for this.

Hello, I get an error when using the TinyGo template

Status: Downloaded newer image for suborbital/builder-tinygo:v0.2.0
⏩ START: building runnable: hi (tinygo)
go: downloading github.com/suborbital/reactr v0.13.0
error: could not find wasm-opt, set the WASMOPT environment variable to override
Error: failed to BuildWithToolchain: 🚫 failed to build hi: failed to RunInDir: failed to Run command: exit status 1

...

Error: failed to BuildWithToolchain: failed to dockerBuildForDirectory: failed to Run docker command: failed to Run command: exit status 255

All is ok with Rust, Swift, and Grain templates

Developers should be able to configure subo to execute Make targets as part of a build. This would allow for things like building static sites before creating a bundle.

For example, if I run subo build . --make app, I'd want subo to run make app before starting the actual build.

It is an amazing work you folks are doing here. I encountered an issue while following the walkthrough here.

When I got to subo build helloworld --docker, the following error shows up:

...
✅ DONE: helloworld was built -> /Users/.../Desktop/projects/important-api/helloworld/helloworld.wasm
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x13b1670]

goroutine 1 [running]:
github.com/suborbital/subo/subo/command.BuildCmd.func1(0xc000130780, 0xc0000ec4e0, 0x1, 0x2, 0x0, 0x0)
	/private/tmp/subo-20210905-4022-iy0t5p/subo-0.0.22/subo/command/build.go:81 +0x310
github.com/spf13/cobra.(*Command).execute(0xc000130780, 0xc0000ec4c0, 0x2, 0x2, 0xc000130780, 0xc0000ec4c0)
	/private/tmp/subo-20210905-4022-iy0t5p/subo-0.0.22/vendor/github.com/spf13/cobra/command.go:852 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000c0f00, 0x0, 0xffffffff, 0xc000096058)
	/private/tmp/subo-20210905-4022-iy0t5p/subo-0.0.22/vendor/github.com/spf13/cobra/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/private/tmp/subo-20210905-4022-iy0t5p/subo-0.0.22/vendor/github.com/spf13/cobra/command.go:897
main.main()
	/private/tmp/subo-20210905-4022-iy0t5p/subo-0.0.22/subo/main.go:10 +0x2a

No runnables.wasm.zip generated.

Subo CLI v0.0.22 (Installed via brew)
macOS Catalina v10.15.6

This would be useful for languages like Grain that reference versions in the prereq steps.

When running subo build inside a Runnable directory, we should not create a bundle. as it doesn't make sense to create a bundle unless running from a project root.

The builders folder link in the README is broken (https://github.com/suborbital/subo/blob/main/builders). Is it supposed to point to https://github.com/suborbital/subo/tree/main/builder/docker instead? I can update the README, just making sure that's the correct one!

I have a feature request for live reloading when using subo dev.

I'm coming from a "similar in function but different in scope" dev environment called architect and one of the best features I've been able to take advantage of in development is the ability to reload function (or in our case runnable) code without restarting the web server or manually triggering the build process. This includes editing the arc equivalent of Directive.yaml.

It's unclear to me if atmo would ever support websockets (arc does because it's emulating aws's api gateway which has that feature), but being able to modify message handlers on a connected web socket is an amazing development experience.

There should be a subo clean command that removes all of the target/.build folders for Runnables and deletes the .wasm files.

Existing templates should be deleted before attempting to update templates to prevent errors caused by replacing files

Fairly straightforward, should be able to build C++ into Runnables!

subo create project should include a Dockerfile that can be used to build an image which includes the project's bundle. Down the line, it would enable something like subo publish which would build that image and push it to a repo.

Replication steps:

subo create runnable foobar --lang somethingnotsupported

Which still creates a foobar folder even though it failed.

Would be great to have some Go files in a directory, and be able to add template markers in documentation markdown such that running subo docs build would build and run all of the Go code to ensure it's still correct, then insert the appropriate snippets into the documentation.

There should also be a subo docs test command wherein it tests the snippets but doesn't re-generate the docs, to be run in CI.

Blocked by swiftwasm/swift#1854

Replication steps:

mkdir foobar && subo create runnable foobar --lang rust                             

Output:

⏩ START: creating runnable foobar
Error: 🚫 failed to Mkdir: failed to Mkdir: mkdir /Users/odc/Documents/GitHub/foobar: file exists

I think the error message is clear for most, but for new users it may be unclear what the problem is.

We already have a nice check if runnables already exist, do we want one to check if the output dir already exist? Maybe something like the following:

⏩ START: creating runnable foobar
Error: 🚫 runnable output dir already exists: failed to Mkdir: mkdir /Users/odc/Documents/GitHub/foobar: file exists

bctx.RunnableExists would catch this if the dir already exist and it's being used for runnables. But it doesn't cover the case when a user reuses an existing dir.

It's a simple patch if we like it.

When creating a bundle, a Directive should be generated using the metadata from each Runnable's DotHive, and the Directive file should be included in the bundle

Replication steps:
subo compute create token [email]

Output:

A verification code was sent to your email address. Enter the code to continue: [your code]
Error: failed to CreateEnvironmentToken: failed to Do: failed to Do request, received status code 401

We should run a Wasm bundle size optimization tool (like Wasm-gc) on modules built by Subo

Subo should be able to run native commands to run builds in various languages instead of using Docker.

Once this is implemented, the builder Docker images should be updated to use subo internally so that a Docker and non-Docker build is identical.

Rust for example provides a --release mode, and we should run that by default, with a --debug Subo flag for when that's not desired.

subo currently has subo create to create a new Runnable.

That should be updated to be subo create runnable, and then subo create project should be added to bootstrap a new project.

It should have a Directive, an empty Runnables directory, and perhaps a static and templates directory if those features get built.

If you have an existing Compute configuration setup:

 subo compute deploy core --local

And you try to change the version by re-running the command in the same folder:

 subo compute deploy core --local --version v0.1.0-rc1

It seems to use the existing version.

Initialization code for wasm runnables is now expected to appear in a module's _start function instead of init.

See also: suborbital/reactr#130

Adding a subo push and/or subo pull which packages the bundle as an OCI image and then pushes it to an image registry would be very useful.

Atmo could also be directed to use an image from a registry.

We can take inspiration from wasm-to-oci, though it will be different as we're pushing a bundle, not a single Wasm module.

spf13/cobra has built in support to automatically generate shell completion scripts for zsh, bash, and fish. CI would then be able to generate each script and package them for use in package managers (homebrew, etc.).

I think it would be beneficial for the subo create commands to also scaffold out a .gitignore file. I felt like this was an issue because when I ran subo create project my Rust target directory was being tracked.

These .gitignore files could be added at the project level, the runnable level, or both. Some things that might be nice to exclude would include:

• target directories
• The equivalent for Swift (not familiar with the language)
• Maybe runnables.wasm.zip. I'm not sure if tracking this artifact of the build process is idiomatic or not. Regardless, it's inclusion or absence in the .gitignore would be a good guide to new developers like me on whether it is idiomatic to track or not.

Related: suborbital/reactr#10

When a bundle exists, hivew serve {path_to_bundle} should spin up a server that loads the bundle and uses the (not yet existing) h.Serve() to automatically serve all of the Runnables in the bundle.

Should be able to build a Runnable from AssemblyScript!

I get a hint from the source that maybe the default used to be --version but a fresh install today has zero output with a successful error code. --help would be rad.

To reproduce:

• Install subo
• cd ~
• subo create project name

Returns:

Error: 🚫 failed to get CurrentBuildContext: failed to getRunnableDirs: failed to list files in .Trash: open /Users/fana/.Trash: operation not permitted

Expected result:
Project is created successfully

Hardware specs:

• 2.8 GHz Quad-Core Intel Core i7
• 16 GB 2133 MHz LPDDR3
• Docker Desktop 3.5.1

Remove the copy/paste step for getting an environment token by feeding it seamlessly through to the next command with a temporary file.

https://twitter.com/SuborbitalDev/status/1405275127428747274?s=20

I believe the Gobinaries service isn't compatible with Go 1.16... investigating if a rollback to 1.15 would fix it, but for now you can install subo by cloning this repo and running make subo !

subo create project command should initialize a new git repo for that project.

Essentially, using util.Run to execute git init after the new project has been created.

Right now, we only provide Homebrew binaries on macOS. It would be useful to have pre-built binaries for other systems.

Currently only listens on :80

If there is no provided Directive (or if the provided Directive doesn't include a version number), attempt to generate one using git (git describe --tag).