This can be used to make request/response handling between client and server easier and allow us to document expected handling in our client types. With request/response handled in the client, the CLI logic can focus on reading files, forming requests and formatting response.

Some of this work is already done in CLI last issue, this will mainly be abstracting some logic into a client interface.

See TODOs in cli/runner/client/client.go and feel free to rename to make more sense in your use case.

Related page from Effective Go: https://go.dev/doc/effective_go#interfaces_and_types

TODO

• implement NewClientFromConfig() helper function
• refactor existing code into interface implementation struct for Run() and Languages() commands
• use new client in CLI to make requests

Details

Short Term Goal

Track max number of concurrent code-runs and the uid and gid allowed to be assigned to these code runner instances. The uid and gid are passed in statically here. Instead, these values should be tracked centrally by the engine so concurrent never are using the same UID or GID.

This could probably be done by using a set number of "workers" to handle requests, where each worker is given an allowed Uid, Gid and working directory. There can be a central map that stores the workers and when a request comes in we could do a dumb (for now) linear search over the list to find the first available worker. Workers would be responsible for setting if they are "ready" or not.

Long Term Goal

Design a solution to have N CodeRunners as "workers", running user submitted code. This limit will help protect the "host" system and the server from overloading while letting us execute code concurrently.

The process of a user submitting code should have this flow:

1. client submits code to be executed
2. server calls the controller with request
3. controller checks for first available worker
   a. none found: return error/throttled response
   b. found: send to runner
4. runner executes code
5. runner sends response back to controller -> server -> client

This could also be approached by spawning as many threads up to limit N at one time and return 500 errors when at capacity.

Current Logic

We create temporary directories on-demand to store user code and use a static UID, GID. The goroutine handling a request will call CodeRunner, uses temporary directory to write user code to a file, compile and run it. This will likely be sufficient for now but we do not have any "levers" to control how many requests can be handled at a time. This will be important after load testing when we know the limits of our server.

Deployment strategy for runner project.

The project can use a container for deployment. Will need to identify providers that make this easiest. ECS Fargate is likely a reliable, familiar option but should investigate https://fly.io vs cheaper VPS providers with a VM using Docker compose.

Should make sure we have a "kill switch" to turn off project if we get DDOS'd or something.

• test if we can use containers for deployment
• check out and test (if possible) container service providers (AWS ECS Fargate, fly.io, etc)
• get domain, setup DNS
• deploy project, network load balancer
• write Ansible, Terraform, or AWS CDK for automated deployment (stretch)

• run tests locally for whole project (and maybe specific subtrees?)
• build server and CLI locally
• start mock server
• start actual server
• go lint entire project (?)

Will add more details to this issue but should be suitable for anyone to take up. This might be a good opportunity to get more familiar with the runtime part of the project.

The goal of this issue is to make sure that we use the "thread-safe" runtime to ensure that when concurrently running user requests we don't use the same UID and GID.

The coderunner should now use a Controller instead of the RuntimeAgent directly. We can make a coderunner struct (if there isn't one already) that will be initialized with the AsyncController when the server starts up. The server can have some default configuration as well that determines how many agents the server creates.

We can also make a more appropriate type to pass into SubmitRequest: https://github.com/camerondurham/runner/blob/e6362100fba794e9ea5626bf54da1f09592e7810/engine/controller/controller.go#L59

Instead of RunProps, the coderunner should not have to pass in any Uid or Gid since the Controller should handle the Uid/Gid since agents contain what values we should use. We can make some other wrapper type for the SubmitRequest like this:

type ControllerRequest struct {
    RunArgs: []string
    Timeout: int
    NProcs: int
}

TODO:

• create new props to pass to controller.SubmitRequest instead of RunProps
• remove Uid and Gid from RunProps
• use AsyncController in the coderunner instead of directly using a new runtime each time

It would be nice to have a doc that defines the basics of containers and how to do some basic process isolation in Linux. This should eventually become a detailed doc of how we handle isolating potential malicious user code from the machine.

Document should include resources to good articles/DockerCon/KubeCon videos that are helpful.

• document describes lower level container implementation
• document links to learning resources
• document includes how to use basic linux syscalls (i.e. unshare, pivot_root)

Details

Create unit tests to verify CLI functionality for command handlers. If possible, add integration tests to test against CLI binary.

Motivation

The purpose of doing this is to verify CLI functionality and make sure this handles all the error/success cases we think it does. Unit tests like this can speed up development and iteration because we don't have to manually verify the CLI works like we want it to, all we have to do is run go test ./....

Part of this involves making the code more test-able by breaking up commands into smaller functions that we can easily verify. This also means clearly identifying and separating dependencies to allow easier mocking so when writing a test for the functions in cli/runner/run.go we are only testing the code in that file, not also testing the code in cli/runner/client/client.go.

Possible Implementation Steps

1. Make Requester Mocks

Make a directory such as cli/runner/client/mocks to contain mocks for the Requester interface. Then you can generate mocks of the Requester interface by using the mockgen CLI. See an example from the Makefile: https://github.com/camerondurham/runner/blob/2d02bbcc19294dead678d9aa7b1ed6c885018d2c/Makefile#L80

Now you can create a requester and mock its behavior when testing the CLI.

2. Refactor CLI functions so the Requester is passed into function

This step is needed so we can make the CLI functions use the mocked requester created in 1 instead of making a "real" one.

To do this, we can refactor the CLI into the CliClient by making a struct with the CLI's dependencies. An example of this structure is the runner server Client struct: https://github.com/camerondurham/runner/blob/2d02bbcc19294dead678d9aa7b1ed6c885018d2c/cli/runner/client/client.go#L28-L31

What this allows is the ability to inject the Runner Server Client mocks.

Since the CLI really only needs the Client, the CliClient can be something like this:

type CliClient struct {
    client Requester
}

Then we can separate the runner CLI specific logic into refactor the the CLI logic into pointer receivers:

func (cli *CliClient) run(lang string, filename string) (*api.RunResponse, error) {
   // check language and load file 
   ...
   return cli.client.Run(&RunRequest{...})
    
}

3. Implement the unit tests

Now you can create unit tests that will test the CliClient structs created in 2. It will probably look something like this

cli := &CliClient{
   client: mocks.NewMockRequester()
}

See this snippet for how to make specific responses for the generated mocks:

https://github.com/camerondurham/runner/blob/2d02bbcc19294dead678d9aa7b1ed6c885018d2c/engine/coderunner/runner_test.go#L20-L33

Subtasks

• unit test for langs command (Run function)
• unit test for run command (Run function)

Project repository with dev environment guide and basic project structure.

Engine, API server and CLI should be runnable and have at least one unit test.

• README with dev environment instructions and PR process
• Engine module boilerplate
  • runs input commands and returns stdout/stderr
  • basic integration test
• CodeRunner module boilerplate
  • rough implementation of multiple language support
  • data structures setup to implement language specific features
  • unit test
• API module boilerplate
  • GET /api/v1/languages endpoint hard-coded response
  • POST /api/v1/run endpoint hard-coded response
  • basic integration test
• CLI module boilerplate
  • stub for languages subcommand
  • stub for run subcommand

This story is to add the data structures we will want to use to configure runner task/job resources. This config could be based on the OCI Runtime Spec for POSIX processes but open to different approaches.

The restrictions defined in this issue will be implemented in #38 with Golang wrappers around the mentioned setrlimit and getrlimit syscalls.

A sample configuration:

• Rootfs string: path to "root" filesystem, where container should think "root" (/) directory is
• [] RLimit, nprocs: the maximum number of processes the task can spawn
• Hostname string: the hostname of the container (this isn't in the spec)

Linux RLIMIT

Linux uses RLIMIT to restrict resources that a process can create. We can use the setrlimit syscall to set these limits in another issue. RLIMIT's are listed in getrlimit(2) pages. We will be interested in:

• RLIMIT_NPROC: number of processes process can spawn
• RLIMIT_FSIZE: max file size process can create
• RLIMIT_CPU: limit in seconds of CPU time the process can have

AC:

• decide new runtime props props
  • array of RLIMITs
  • container root path
  • timeout
  • hostname
• create new RunCmd function that will restrict the executed process, no need to implement restrictions yet

Create end-to-end integration tests for limiting behavior. Should create runner struct and send code that will test the num procs limit, timeout limit and other limits as needed.

Requirements:

1. verify runtime agent returns an error when program exceeds timeout limit
   1. and that execution does not exceed TimeoutLimit + epsilon
2. verify runtime agent returns an error when program exceeds max num procs limit
   1. (note: this will be hard to verify at an OS level, at least returning an error should be okay for now)
3. verify runtime agent returns an error when program exceeds max file size creation limit

Will add details later but this RLIMIT needs to be added.

https://man7.org/linux/man-pages/man2/getrlimit.2.html

       RLIMIT_CPU
              This is a limit, in seconds, on the amount of CPU time
              that the process can consume.  When the process reaches
              the soft limit, it is sent a SIGXCPU signal.  The default
              action for this signal is to terminate the process.
              However, the signal can be caught, and the handler can
              return control to the main program.  If the process
              continues to consume CPU time, it will be sent SIGXCPU
              once per second until the hard limit is reached, at which
              time it is sent SIGKILL.  (This latter point describes
              Linux behavior.  Implementations vary in how they treat
              processes which continue to consume CPU time after
              reaching the soft limit.  Portable applications that need
              to catch this signal should perform an orderly termination
              upon first receipt of SIGXCPU.)

Create named groups and users on container startup:

# the simplest solution from the command line is

# make group with gid 1234 named runner1
groupadd -g 1234 runner1

# make user assigned to group runner1 with uid 1234, "home" directory /tmp/runner1, named runner1
useradd -g runner1 -u 1234 -d /tmp/runner1 runner1

• install python3 and bash in dev container (can just use Debian or find equivalent in Alpine Packages)
• create user directory for single runner (we'll add more for other users later): /tmp/runner1
• make simple entrypoint script or setup users on server startup
• (stretch) add this functionality to a dedicated server struct or add to CodeRunner to run when we initialize the server

Make backend request to server with form values and display in the output box.

Should handle server timeout if needed. Can use mock server for basic development but should ideally test against actual running server.

Please submit screenshots of work when creating CR.

CLI makes valid requests to mock server endpoints created in #6.

Resources for working with the CLI:

• link to code in this repo: https://github.com/camerondurham/runner/tree/main/cli/runner
• Docs: Cobra Concepts
• Docs: Getting Started
• Examples:
  • adding a command line flag to CLI
  • reading what user set flag to

Basic requirements:

• CLI makes GET request to /api/v1/languages endpoint, displays a formatted response in terminal
• CLI makes POST request to /api/v1/run endpoint and displays a formatted response in terminal

Nice to have:
These can be implemented in another story

• CLI loads source code file from user
• CLI sends source code string in POST /api/v1/run request body

Validate client request has non-empty POST Body with all required fields.

So far this just includes language and source.

See api.md in docs branch for current API spec.

• validate client request has non-empty Body with required fields
• unit test for validation logic

Create a tool to generate and run synthetic load on the coderunner server. This is needed to test server performance and identify any scaling issues and determine how the application performs under high traffic.

Requirements:

• makes requests with different types of languages and source code input
• tool can be configured to make X requests per second

Nice to have:

• randomly selects from at least 5 examples of each language
• use GitHub Copilot or similar tool to generate language specific examples

Basic server wrapper around the CodeRunner API. Should respond to POST/GET request that provides user input and return the Stdout result.

Not expected to handle remote shell vulnerability, server crashes, etc but will be useful for initial testing.

Resources:

• examples on how to use CodeRunner module: engine/main.go

• where to implement server: server/main.go

• example of unit tests using mocks: engine/coderunner/runner_test.go

• server endpoint: GET /api/v1/languages returns list of supported languages, retrieved from the CodeRunner module (engine/coderunner/types.go)

• server endpoint: POST /api/v1/run lets user submit code to run and returns output

• unit test for server request handlers

The runtime should limit how many processes that can be created when the RunCmd function is used. Currently, the only limitation put on the code is how long it runs.

The Linux syscall setrlimit will be useful, exposed in os.syscall package with Setrlimit. Linux utility prlimit (used to get and set resource limits for a process) will likely be most helpful in debugging since it is a wrapper for setrlimit. NOTE this will require being run in a Linux box or at least a Linux Docker container.

Example:

runner@runner-vm:~$ prlimit --pid $$ --nproc=10:12
runner@runner-vm:~$ ps
    PID TTY          TIME CMD
  50411 pts/0    00:00:00 bash
  50438 pts/0    00:00:00 ps
...
runner@runner-vm:~$ sleep 100 &
[6] 50444
runner@runner-vm:~$ sleep 100 &
-bash: fork: retry: Resource temporarily unavailable

More Resources

RLIMITs we care about described here: getrlimit manpage
Process Resource Limit: prlimit syscall
Go wrapper for syscall: Prlimit
Golang package/wrapper for these syscalls: golang.org/x/sys/unix

Constants where RLIMITs are defined: pkg-constants

AC:

• runtime can set hard limits on how many processes can be spawned from a RunCmd
• unit tests verify processes are killed when over nproc limit
• fork new child process before applying limits

Go lint failing from not checking returned error from json encoding.

This was my code and just making this issue so I remember to fix it.

General repo improvements for faster development.

Merge with confidence! Let Github Actions run tests for you.

Should check the following:

• build server and CLI binaries successfully
• passes go lints/vets
• code coverage test (stretch)

Docker setup for local testing and easier deployment when we decide to do so.

• add Dockerfiles for running tests
• Docker compose for running mock server/real server
• add targets to Makefile for running containers
• run go ci lint tool

Git Hooks:

• run go fmt tool on repository before commits

This issue is to improve the resource limiting API by removing the process helper CLI or using another library to limit host resources.

An "external" process may not be needed and hopefully can find a way around this.

Tasks:

• test if limiting behavior can be accomplished in same process
• create new issue to try limiting using cgroups instead of rlimits (see containerd/cgroups)

Related Background

The prlimit/setrlimit mechanism being used will apply limits on the process if the user is not root and these limits will be for that user. Will need to manually test this implementation, this is possibly sufficient for now but using separate non-root users for each process is needed next.

prlimit --nproc a.k.a. RLIMIT_NPROC a.k.a. ulimit -u is the maximum number of processes¹ for the user as a whole. If you already have 20 processes running, and you set the limit to 20, you can't create any new process. What matters is how many processes are running as your user, it doesn't matter who their parent is or what their limit setting is.
[source]

See notes in this issue for more details: #42 (comment)

Originally posted by @camerondurham in #38 (comment)

Look into using containerd/containerd as a container runtime to run processes or using containerd/cgroups to enforce limits instead of Linux syscall setrlimit : https://github.com/camerondurham/runner/blob/9cc6b646cb85552122482783d44008383ea14966/engine/runtime/limits.go#L33

An example of using cgroups instead of resource limits could be setting the pids cgroup value to the proc limit (see the containerd/cgroups repo for what this library can do for cgroups, specifically: https://github.com/containerd/cgroups#create-a-new-cgroup).

Related links:

• containerd: Getting Started
• RedHat: cgroups
• Linux cgroup man pages

Note:

This will likely require setting up a custom Linux VM as these tools cannot be tested in a container. We'd have to adjust the project's deployment strategy accordingly to pursue these alternate libraries/runtimes.

This does not have to be limited to containerd. We can explore other container runtimes besides this, including Docker itself. Ideally using containerd would be preferred since it is a lower level runtime that can expose more customization for us. Docker is really meant for users and dev experience and has many extra features we don't need such as image building included in the Docker daemon.

AC:

If containerd is a good fit:

• determine dependencies required to run a containerd daemon
• create Ansible/Vagrant or some automated configuration to run containerd
• write module to start daemon and execute code in containers

If containerd is not a good fit:

• decide on syscalls and security strategy that our runtime should use

Engine can run basic commands from caller.

Example:

resp, err = engine.RunCommand(CommandProps{command: "echo hello world"})
fmt.Println(resp.CommandStdout)

• create engine module with RunCommand function
  • engine runs arbitrary command input from user
  • engine returns Stdout result from command
  • engine returns Stderr result from command
• unit test RunCommand and/or helpers

• types.go for server including expected request and response types
• export file extension map for reuse in CLI (can export/refactor this)

Design/Refactor CodeRunner module to support running compiled and interpreted languages.

Currently, to support compiled languages, the compile and run step have to be executed "together" (link). This issue is to create a PreRunHook interface that implements any steps required before running the code. An example of how this could be used is writing the sourcecode to file for Python or writing to file and compiling to binary for C++.

Requirements:

• "easy" to add another language (easy can be defined many ways, let's say adding support for Node shouldn't take more than a week?)
• supports compiled and interpreted code execution

Example interface:

type CodeRunner interface {
    PreRunHook(lang Language, filename string) (RunProps, error)
    Run(props RunProps) (RunResponse, error)
    PostRunHook(filepath string) (error)
}

Currently implementation is here.

Example execution structure:

func (r *CodeRunner) Run(req *RunnerProps) (*RunnerOutput, error) {
    // error handling ...

    // handles writing to filename.extension and compiling code if needed
    runProps, err = r.runner.PreRunHook(r.language, runDirectoryTmp)

    // error handling
    resp, err := Run(runProps)
    
    // cleanup
    // ...
    // return output
}

• CodeRunner function accepts variable language and code snippet

  • CodeRunner can internally handles compiling step if needed
  • CodeRunner may temporarily store user code in filesystem
  • CodeRunner deletes user code after running
  • Unit test for multi-language feature

• End to End test for basic code snippet

Use Sylvia's branch, should be able to use syscalls:

• chroot
• sethostname
• setrlimit

EDIT:

Unable to use syscalls requested with Docker settings. Recommend using VM, WSL, or native Linux install to run.

Have prepared an Ubuntu Server VM .ova file to import into VirtualBox:

• Setup guide Google Doc Link
• Virtual Machine Image runner-ubuntu-server.ova

Ideas on more languages to add:

• Go (very easy, we already have Go installed in the image)
• C (should be relatively easy, image already requires uses gcc)
• Ruby (honestly who uses this?)
• Rust
• Zig

See #82 for what files should be modified. Calling out in particular:

Backend

1. Server Dockerfile (for adding any dependencies required for the server image): https://github.com/camerondurham/runner/pull/82/files#diff-e3dc0276ffd991d263bbb04bce5d89d0773bb4069c96ef46282902681ec64ba3
2. language.go (for adding language config, how to compile (if it's compiled of course) and run): https://github.com/camerondurham/runner/pull/82/files#diff-3c34a65e1f5cf2827cbca5707c44a4301816d2c713a80ab9e868a6e221db08a3
   a. need to add language config
   b. add language to SupportedLanguages, SupportedLanguagesSet so server accepts new lang requests
3. Add new test cases in language_test.go as well

Frontend

1. set-code.js (for adding language starter code to the dropdown): https://github.com/camerondurham/runner/pull/82/files#diff-1d55dc40f31e290098a80b8785b23ffa804a4fbd7cc1f81aa95f08207cb6ff6e

This may be blocked by #15

Create test suite of "malicious" code to understand limits of the CodeRunner. This will be helpful for benchmarking and understanding how much progress we've made toward the Code Isolation Milestone.

These test cases can be wrapped in an environment variable check so they do not fail the build and just used in a temporary test container. Example:

func Test_MaliciousCode(t *testing.T) {
  if os.Getenv("TEST_MC") != "" {
    // test stuff that may intentionally break
  }
  // rest of the test
}

• add set of test cases for the runtime that run separately from unit tests (ideally in Docker)
• test failure modes
  • throw an unexpected exception
  • spawn many processes
  • "escape" the current directory
  • spawn a shell
  • ... (think of other ideas)

Create mock server for API endpoints described in design doc: Runner Project Details

This functionality currently exists. However it would be nice to have a separate mock server
for specifically testing frontend/CLI and so changes can be made on server without interfering with other development.

Acceptance Criteria:

• Mock GET /api/v1/languages (added in #11, can copy into mock server)
• Mock POST /api/v1/run (added in #11, can copy into mock server)
• Mock POST /api/v1/run with random response (stretch)
  • Return mock "syntax error" like response
  • Return mock "runtime error" like response
  • Return mock successful response (prints out some message to stdout/stderr, no errors)
• update Makefile commands to support make run-mock-api and run the corresponding mock server

Minimal frontend with a text input form, language selector, "Run Code" button, and ideally a space to display user output.

Should submit form selections but backend request/response handling can be handled in a separate PR.

Please submit screenshots with the CR. Only working to target Google Chrome on a ~1920x1080 screen. Any other screen is out of scope or stretch if you really want to make it work.

Some basic server unit tests exist here already: https://github.com/camerondurham/runner/blob/main/server/main_test.go

We don't need all of the tests below and can have different/more tests. This is just some basic test ideas to improve confidence in code changes that we're not breaking the server functionality.

• basic test for throwing 400 error function: https://github.com/camerondurham/runner/blob/main/server/main.go#L84
• basic tests that trigger runtime failure branch: https://github.com/camerondurham/runner/blob/main/server/main.go#L66-L69
• basic test that language endpoint has well formed response (just make sure we always serialize the response matching our api types): https://github.com/camerondurham/runner/blob/main/server/main.go#L23

Demos

Add these gifs to README

1. screen recording of writing/editing code
2. ascicinema recording of using CLI

Use (any) framework for frontend. It can be Vue, React, Alpine.js, whichever is up to whomever works on this can decide. Would be nice if we could use Typescript (Vue, React) but not a requirement if it makes development too difficult.

The current frontend uses webpack for building but this can be replaced by any framework packaging tools. create-react-app, create-vue comes with its own autogenerated webpack config and vue likely does as well.

Nice to have:
Feel free to take artistic liberty with the frontend but would be cool if we could use existing coding playgrounds as design examples:

• https://play.rust-lang.org
• https://go.dev/play
• https://elm-lang.org/try

Chat GPT didn't exist when this was written but it exists now. Use it to come up with a better project name.

https://www.reddit.com/r/golang/comments/16krmxd/comment/k10to73/

Ideas:

1. gobox
2. sandbox
3. codecanvas
4. langbox
5. langpage
6. codefusion (horrible wtf)
7. lang-playground

It would be nice to use a simple editor experience for a runner webapp/webpage.

This story is to look into existing libraries we could use for the editor and (if possible) create a minimal proof of concept where users can write code in the browser.

For example, the Monaco Editor from VSCode is an NPM library we could use.

Minimal requirements:

• syntax highlighting based on language

• frontend able to send input and log to console (show we can easily make a web request to backend)

• (timebox ~1-2 hrs) looking into existing libraries we can use for editor, syntax highlighting

• create mockup, single webpage

Currently, the runner uses the Linux timeout (from the coreutils homebrew formula on macOS) command to execute commands with a timeout.

This is to improve execution by using multiple channels and select to implement the timeout in Go. See gobyexample/timeouts for an example.

Acceptance Criteria (in progress):

• run command with configurable X second timeout
  • look at other options to implement timeout in Go
  • explore if we can use select with exec.Command instead of the timeout external process
  • test functionality in unit tests

The following Linux utilities may be helpful in implementing this:

• prlimit: get and set process limits
• timeout: run a command with a time limit
• pivot_root: change the root filesystem
• chroot: change root directory (also see https://superuser.com/questions/1575316/usage-of-chroot-after-pivot-root)

The solution to implementing these limits is still not certain. For now, we can attempt to use prlimit syscall to limit process resources. Our configuration for runner tasks/jobs should be based on the OCI Runtime Spec, specifically , the configuration for POSIX processes.

Acceptance Criteria (in progress):

• run command with configurable X second timeout (moved to #17)
• #38

Stringify the errors from the backend to prevent the [object Object]

https://github.com/camerondurham/runner/blob/016c7b907c718c5231d6cc0478eabf2af14f65f6/web-frontend/js/run-request.js#L43

Tasks

• stringify errors
• check for other un-stringified messages directly displayed on the frontend
• local testing / verification

Subtasks:

1. use webpack or other packaging tool to bundle frontend dependencies
2. update html/js for packaging
3. deploy to gh pages
4. (stretch) deploy to CloudFlare pages w/custom domain