Git Product home page Git Product logo

theia-workspace-builder's Introduction

theia-workspace-builder

Theia workspace builder (TWB) unifies the provisioning and building process of development environments featuring eclipse-theia, as docker containers. TWB consists of different modular setups, alongside the actual builder tool. In short, TWB allows to pack those modules together from a simple configuration file into a workspace of your needs.

How it works

Basically every workspace is broken down into a base system, and independant modules that take care of certain languages, or utilities. By utilizing templates, TWB merges the base and all selected modules into a single workspace setup, which is built as docker image.

Every module, and the base setup as well, consists of a Dockerfile template, and a package.json file. While all installation steps are placed in the Dockerfile, the package.json contains dependencies and plugins required by theia. By providing a Dockerfile for each base, it is possible to support multiple systems, where you can choose one of in your workspace configuration.

The builder tool is a utility, that does all the work for you. It bundles your workspace setup into a ready-to-use docker image.

Getting started

An example workspace setup can be found in example-ws.

Define a Workspace

To define a workspace, all you need to do is to create a directory, and place an application.yaml file in there. It is not important, where you create this directory, but recommended to place it inside your clone of this repository, as it allows the most easy use. All workspace specific files stay in this directory as well.

A workspace directory may contain a subdirectory named module, where a custom package.json and Dockerfile.j2 can be placed. This allows to add workspace specific setups.

The application.yaml

A minimal application.yaml looks like the following.

app:
  version: "0.0.1"
  org: my-org
  license: "Apache-2.0"
  title: "Example Theia Workspace"
  base: archlinux
modules:
  - cpp

To include a module in your workspace, just add it to the list of modules. It is also possible to set parameters for your workspace.

Prepare and build your Workspace

The tool that is used to provide workspace setups is found in builder-tool. It can either be invoked directly as python script, or installed via setuptools. Run it with --help to get more information about its usage.

Preparation

Run the builder tool with prepare command, to generate the final Dockerfile and package.json files inside your workspace directory. The prepare command must always be ran before build.

python3 builder-tool/main.py prepare example-ws/

If dependencies or plugins are defined multiple times in several package.json files, a warning will be emitted on value changes. The final value in the resulting package.json depends on module precedence. The precedences are as follows, where the last entry has the lowest precedence.

  • workspace-dir/module/package.json
  • modules/.../package.json
  • base/package.json

The Dockerfile templates are appended as is, in following order.

  • base/system/Dockerfile.j2
  • modules/.../system/Dockerfile.j2
  • workspace/module/Dockerfile.j2

Build

Run the builder tool with build command, to build the docker image. The build command might require root permissions, in order to talk to the docker daemon.

python3 builder-tool/main.py build example-ws/

Run your Workspace Container

In order to run the workspace as container, with support for git over ssh, run something like the following command.

docker run --init --security-opt seccomp=unconfined -dit --restart=always -p 3000:3000 -v "$(pwd)/my-project/:/home/project:cached" -v "$(pwd)/.ssh:/home/theia/.ssh:ro" my-org/example-ws

Complete application.yaml Schema

In the below schema, // is used as comment, and everything in () is optional.

app:
  (name): <docker image / theia app name>
  version: <version string>
  org: <organisation name>
  license: <license name>
  title: <application title>
  base: <base system name>
  (base_tag): <base system tag, default is 'latest'>
  (dep_version): <theia dependencies version, default 'latest'>
(parameters): // map module names to their params
  (<module name>):
    <key-value entries for params used in template>
(build):
  (registry): <docker registry name>
  (arguments): // args passed to docker build
    <key-value entries>
(modules):
  <list of module names>

If no app name is given, it will be generated from app title by replacing spaces with dashes (-) and convert characters to lower (a-z).

How to create Modules

Every module has its own directory under modules, where the directory name denotes the module name. A module may contain a package.json file, with dependencies and theiaPlugins. A module may contain subdirectories named after the base system, which may contain one Dockerfile.j2. This Dockerfile template defines all installation steps for this module. Ideally a module contains also a readme file, to describe its purpose, and parameters.

Supported Base Systems

  • archlinux
  • debian

theia-workspace-builder's People

Contributors

jarthianur avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar

theia-workspace-builder's Issues

Optional base image tag

Is your feature request related to a problem? Please describe.

Currently the base images tag is hardcoded. It could be great to be able to specify a certain tag in application.yaml, in order to select a different version.

Describe the solution you'd like

Add an optional field to application.yaml schema that denotes the base image tag to use. Default should still be latest.

Describe alternatives you've considered

Additional context

This could become more valuable, as soon as base images like debian, or centos are supported, because often different versions are used, and dev should match prod environment.
Altough some experimentation is necessary on how the scripts work independent from base version.

Project documentation/wiki with sphinx

This project needs a documentation/wiki about how to use, extend etc.
The doc should be created within docs/ folder and created with sphinx.

Subtasks:

  • create doc base layout with sphinx
  • layout and idea behind this project, and how it works
  • components:
    • builder-tool installation and usage
    • modules and how they work
  • how to define an application
  • how to extend/create modules
  • how to build and run an application

Move from manjaro to archlinux

Move from manjaro base to archlinux, as the latter one is better maintained and supported. Features and stability are quite the same. Apart from the base image name, there should be nothing to change.

bash-completion for terminal

Is your feature request related to a problem? Please describe.

There should be tab completion available in bash terminal.

Describe the solution you'd like

Install bash-completion in base image.

Describe alternatives you've considered

Additional context

Configurable theia meta version

Is your feature request related to a problem? Please describe.

Currently all theia packages are set to "next" which corresponds to the master branch.
As this may cause instabilities and theia is in a release version anyway, it would be better to change to latest or make it configurable.

Describe the solution you'd like

Add a parameter to the yaml, that allows to set the meta version for theia packages. It should default to latest.

Describe alternatives you've considered

None

Additional context

Split builder-tool sources

Is your feature request related to a problem? Please describe.

To make the code base more maintainable, separate source files by their concerns.

Describe the solution you'd like

Split the builder-tool into a proper directory/file separation.

Describe alternatives you've considered

None

Additional context

None

Go build not working

Describe the bug
Building a go workspace does not work due to build errors.

To Reproduce
Steps to reproduce the behavior:

  1. Build a go workspace

Expected behavior
Should build successfully.

Additional context
Probably related to gopkgs.

application.yaml validation

The builder-tool should validate the given application.yaml file, and handle missing - and maybe superfluous - fields, as well as incorrect values respectively.

Finalize generated package.json

Is your feature request related to a problem? Please describe.

The generated package.json file should not be just filled from template, but also sanitized to not contain duplicates and so on.

Describe the solution you'd like

Write the package.json content into memory instead of file directly and do some sanitation on it before finally writing to file.
Things to do there:

  • remove duplicate entries
  • detect version conflicts in case of duplicate (key is duped but value differs), and produce error
  • detect incorrect json syntax, or wrong keys

Describe alternatives you've considered

None

Additional context

None

Auto generate app name from title

Is your feature request related to a problem? Please describe.

Currently the app name is configured in yaml, but is actually never presented to the user, or used outside the build process and about page. Hence it could be removed from application.yaml and just generated from app title.

Describe the solution you'd like

Remove app.name from yaml schema.
Generate the app.name from app.title.

Describe alternatives you've considered

None

Additional context

None

Rust language support not optimal with rls

Rust langauge support with rls is not optimal. Intellisense and autocompletion are often not working. This seems to be a known issue with rls. Hence it should be evaluated whether rust-analyzer does a better job.

Cmdline flag to select docker API

Is your feature request related to a problem? Please describe.

It would be nice, if the docker API endpoint could be set with a cmdline option.

Describe the solution you'd like

Add a click option to set docker API endpoint. Still default to local unix socket.

Describe alternatives you've considered

None

Additional context

None

Ubuntu base module

Is your feature request related to a problem? Please describe.

It would be nice, to have an ubuntu based environment.

Describe the solution you'd like

Add an ubuntu base module, and extend all required modules to support ubuntu. This should be similar to debian module.

Describe alternatives you've considered

None

Additional context

None

Errors by built-in plugin versions too new

Describe the bug

The versions of built-in plugins are too new producing errors.

To Reproduce
Steps to reproduce the behavior:

  1. Build theia workspace
  2. Start container
    3 Find broken theia UI

Expected behavior

It should work without errors.

Screenshots

Desktop (please complete the following information):
Additional context

code safety and error handling

The builder-tool needs tough code safety and proper error handling.
Especially dict accesses, API use, and file IO require attention.
The builder-tool should not proceed to run if there are any problems happening that could produce unexpected results.
If an application cannot be prepared, or built exactly as it is defined by the user, no falsely "ready-to-use" state should be left behind, as it could lead to problems in case the user misses the logs.

write a proper README

This project needs a proper README.
The README should be short and appealing, containing only the most important information, but linking to the docs/wiki where appropriate.

Parts for README:

  • description what it is and for what purpose
  • description how it works
  • getting started
  • basic usage (apps and tool)
  • description on how to create modules

CentOS/Rocky Linux base module

Is your feature request related to a problem? Please describe.

It would be nice, to have a centos based environment.

Describe the solution you'd like

Add a centos base module, and extend all required modules to support centos.

Describe alternatives you've considered

None

Additional context

None

Missing Dockerfile.j2 in application module dir causes error

Describe the bug

Missing Dockerfile.j2 in application module dir causes error.

To Reproduce
Steps to reproduce the behavior:

  1. setup an application dir with custom module dir
  2. run prepare
  3. See error

Expected behavior

A missing Dockerfile.j2 should be just ignored.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Dockerfile syntax check

Is your feature request related to a problem? Please describe.

It would be nice, if the generated Dockerfile is checked for syntax errors. Maybe even before writing to disk.

Describe the solution you'd like

Check the Dockerfile in prepare step for syntax errors. Issue an error describing the invalid part, and do not cleanup the files - build step would then fail anyway.

Describe alternatives you've considered

If this is not possible with the docker API, discard it.

Additional context

None

golang module

Is your feature request related to a problem? Please describe.

A module for golang development is needed.

Describe the solution you'd like

Provide a golang module.

Describe alternatives you've considered

Additional context

builder-tool cleanup command

It would be great if the builder-tool offers a cleanup command.
This cleanup command should remove all generated files, and images for an application, but only related to the version currently defined in the yaml file.
If possible, also intermediate, then orphaned images should be removed as well.
In case any container requires the image, no deletion should happen.

parameterized application builds

An application definition should allow to inject parameters into Dockerfiles for builds.

This can be done in two ways.

1. prepare phase parameters:
Prepare time parameters allow to preprocess a module template before the docker build, so builds can be optimized for caching, and do not need conditionals and environment modifications.

Handle input Dockerfiles as jinja templates to allow prepare time parameter handling.
Allow a map in the application.yaml called parameters.
This map allows to define standard yaml key-value pairs, that are then passed into the template rendering.
The parameters map should contain nested maps named after the modules for distinct handling.

2. build phase parameters:
Build time parameters are basically docker args. They can be used when prepare phase parameters are not appropriate. This is actually only necessary if applications are somehow fixed in their prepared state, but still need some variable handling for builds.

These parameters are declared as ARG in Dockerfiles and simply passed in at the build command.
In the application.yaml a map can be defined under build, called arguments.
This map allows to define standard yaml key-value pairs, that are passed in as docker build args.

In both cases there should be NO option to set, or override parameters from command line using the builder-tool, as an application should be exactly as defined in the yaml file. Build arguments can still be used by utilizing docker build command manually instead of the builder-tools one.

Add sudo module

Is your feature request related to a problem? Please describe.

In certain scenarios it might be necessary to run commands as root. Hence a module needs to be added, that allows the theia user to run arbitrary commands with sudo. It is up to the user to include modules anyway, so the security risk also needs to be considered by them.

Describe the solution you'd like

Add a module that enables the theia user as sudoer without password.

Describe alternatives you've considered

None

Additional context

None

Debian base module

Is your feature request related to a problem? Please describe.

It would be nice, to have a debian based environment.

Describe the solution you'd like

Add a debian base module, and extend all required modules to support debian.

Describe alternatives you've considered

None

Additional context

None

Provide a readme for every module

Is your feature request related to a problem? Please describe.

It would be nice to have the content, purpose and use of every module documented.

Describe the solution you'd like

Create a descriptive, but short README.md file for every module, also for base.

Describe alternatives you've considered

None

Additional context

None

GitHub repository setup

The GitHub repository needs to be set up with following things:

  • GitHub community guidelines
  • Contributing and workflow description
  • Kanban projects for components
  • Issues for features and todos
  • Branch protection and CI setup
  • Project description and tags
  • GH pages setup for docs/wiki

Rename sys-manager module

Is your feature request related to a problem? Please describe.

The module sys-manager should be renamed, as it does not properly describe what it actually does.

Describe the solution you'd like

Rename the module to pkg-manager.

Describe alternatives you've considered

None

Additional context

None

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.