(EDITED - check edit history for prior revisions and discussion of the opening post)

For devfile 2.2.0. Check the proposal at https://github.com/devfile/api/blob/main/docs/proposals/devfile/outer-loop-build-and-deploy.md

A stack may require expressing the outer-loop build guidance ( source code to image ) in the form of a Dockerfile build, s2i build, etc.

    variables:
      myimage: myimagename
    components:
      - name: mydockerfileimage
        image:
          imageName: {{myimage}}
          dockerfile:
            buildContext: ${PROJECTS_ROOT}/build
            uri: Dockerfile
            args: [ "arg1", "arg2", "arg3" ]
            rootRequired: false

The other dockerfile src are available in the proposal doc.

Once the runtime image has been built, we can deploy it via the deploy group command:

components:
  - name: outerloop-deploy
    kubernetes:
      uri: deployment-manifest.yaml
commands:
  - id: deployk8s
    apply:
      component: outerloop-deploy
      group:
        kind: deploy
        isDefault: true

The build tool mechanism like kaniko or buildah will be up to the tools and hence will not be a part of the spec.

/kind user-story
/kind epic

Which area this user story is related to?

/area api
/area library
/area integration-tests
/area test-automation

User Story

As a stack/devfile author, I wish to provide deployment guidance in the form of a Helm Chart

---
components:
  - id: deploy-node-app
    helm:
      chart: http://helm-chart-url
      values: 
        image: quay.io/sample/hello-world # A chart may expose a well-known values.yaml parameter called "image".
        replicas: 3

If the helm component is associated with a dockerfile component (image build guidance) in a composite command, #51 , che/odo must associate the built image from the 'build' guidance using the image parameter in the chart's values.yaml .

- composite:
      id: build-and-deploy
      label: Build image and deploy nodejs app
      commands:
      - build-node-app
      - deploy-node-app
      parallel: false
      group: 
          kind: deploy

Acceptance Criteria

• Introduce new component 'helm' to Devfile API
• Introduce new component 'helm' to Devfile Parser
• Introduce calls related to new component 'helm' in Devfile Library
• Validation for new component 'helm'
• Guide for using component 'helm'

refined by: @michael-valdron

Polymorphic types makes devWorkspaces objects easier to validate and more close as a Kubernetes custom resources. But are we sacrificing the UX of the devfile? Was the 1.0.0 devfile spec easier to read and modify for humans?

devfile 1.0.0

components:
  - type: container
    image: maven
      ...
  - type: container
    image: nodejs
      ...
  - type: kubernetes
    reference: https://gist.../mongodb.yaml
       ...

current proposal

components:
  - container:
        image: maven
        ...
  - container:
        image: nodejs
        ...
  - kubernetes:
       reference: https://gist.../mongodb.yaml
       ...

alternative proposal

components:
  containers:
    - image: maven
        ...
    - image: nodejs
        ...
  kubernetes:
    - reference: https://gist.../mongodb.yaml
    ...

It should be published on https://github.com/redhat-developer/devfile/tree/master/docs on 2.0.0 branch

We should publish it to https://github.com/redhat-developer/devfile on branch 2.0.0

Adding optional metadata version and name:

schemaVersion: 2.0.0
metadata:
  name: petclinic-dev-environment                # optional
  version: x.y.z                                          # optional
projects:
   ...
components:
  ...
commands:
  ...

Since in the schema the metadata are optional they can even be extracted in a meta.yaml as it is today in the devfile-registry but that's not covered in the spec.

⚠️ That's about the version of the devfile and not to be confused with the schemaVersion. And the semantic is the semVer one.

On each Pull Requests, add a validation to check that the json schemas, and other generated files, have correctly been generated and committed back to the repository.

If not we should not allow merging the PR

(More of a question than a user story)

In devfile 1.0, we were able to set a dockerimage component entrypoint command and args with the command and args field. A simple example might be:

  -
    type: dockerimage
    image: docker.io/someimage:latest
    alias: build
    memoryLimit: 512Mi
    command: ['tail']
    args: [ '-f', '/dev/null']

While working on the Devfile 2.0 integration in odo, @adisky noticed there weren't equivalent fields for command and args in the container component, and I couldn't seem to find anything in the devfile 2.0 spec either:

Do we need those fields in devfile 2.0 (and therefore don't allow people to override a container's entrypoint) or was that an omission from the spec?

This a simple proposal to use the name manifest instead of kubernetes or openshift in the devfile components.

---
components:
  - container:
        name: component1
        image: maven
  - manifest:
        name: component2
       reference: https://.../mongo.yaml
  - plugin:
        id: java

Feature request: A mechanism to define/access external dependencies in the workspace. This would consist of links to external dependencies so that a developer could easily access those dependencies directly from the context of the workspace.

External dependencies could be url links to any resouce on the web. Common uses that I foresee are:

• links to Cloud services
• links to on-prem services
• links to related code repos
• links to project wikis
• links to deployments on other clusters

Proposed externalDependency component definition in the defile.yaml:

apiVersion: 1.0.0
metadata:
  name: My-Workspace
projects:
  - name: My-App
    source:
      location: 'https://github.com/triceam/my-sample-app-repo'
      type: git
components:
  - id: che-incubator/typescript/latest
    memoryLimit: 512Mi
    type: chePlugin
  - mountSources: true
    endpoints:
      - name: nodejs
        port: 3000
    memoryLimit: 512Mi
    type: dockerimage
    alias: nodejs
    image: 'quay.io/eclipse/che-nodejs10-ubi:7.8.0'
  - name: IBM Watson Visual Recognition
    url: http://cloud.ibm.com/watson/instance
    type: externalDependency
  - name: IBM Cloud Continuous Delivery
    url: http://cloud.ibm.com/devops/instance
    type: externalDependency
  - name: My on-prem Database
    url: http://url.to.my.database/foo
    type: externalDependency
  - name: My on-prem OpenShift cluster
    url: http://url.to.my.openshift/dashboard
    type: externalDependency

Proposed visualization in the Eclipse Che workspace:

Clicking on the external dependency link would open a new tab or window to access the external dependency.

• alias should not be used anymore
• name should be mandatory for kubernetes/openshift/containers components
• name should be optional for plugins components
• If name is not specified for a component of type plugin it should be inferred from its id/url field (id/url are used as name if its not specified explicitly)

### a Devfile spec

- container:
     name: tooling-container # <-- mandatory
     image: busybox
     mountSources: true
     endpoints:
       - name: petclinic-web-server

- kubernetes:
     name: mongodb # <-- mandatory
     reference: .../deployment.yaml # <-- already have a metadata.name: 'database1'

- plugin:
     name: vscode-java # <-- optional
     id: ms-vscode/vscode-java/1.1.1

Commands should have a group field that will specify in what point of application lifecycle the command should be executed (or type of the command like, build, incremental-build, run etc..).

• group field is optional
• Many commands of the same devfile can belong to the same group
• Only one command belonging to a group can be the default
• The list of groups should be: build, run, test, debug

commands:
 - exec:
     name: "package"
     commandLine: "mvn package"
     component: build-tools
     group: 
        kind: build
 - exec:
     name: "install"
     commandLine: "mvn install"
     component: build-tools
     group: 
         kind: build
         isDefault: true 

Currently, project source is mounted to either CHE_PROJECT_ROOT or /projects when mountSources is true. The CHE_PROJECT_ROOT value is configurable during Che setup via the che.properties file

If odo is to consume the devfiles, we would like it to have an alternate way to set the source path value in the component container, rather than having a fixed single value via either CHE_PROJECT_ROOT or /projects for all the devfiles.

This can be achieved by having sourceMapping when mountSources is true, which transfers/mounts the project to the specified path in the container.

Che can also use the property i.e.; when sourceMapping is present, it overrides the CHE_PROJECT_ROOT value if present and mounts the source to the path defined by sourceMapping

Example:

---
components:
  - container:
      image: "busybox"
      name: "mycontainer"
      mountSources: true
      sourceMapping: "/home/src"

We want to experiment how the Devfile format could be used to define a stack.

Requirements

1. Parent devfile

A parent devfile is a devfile published somewhere where odo or Che can read
it. Like a gist, a registry or a kubernetes resource (see referencing parents below).

A parent devfile is a regular devfile and includes components (for build and runtime), one or more code sample projects, events and commands to build and run the sample applications.

Any devfile that references a parent devfile will inherit its components, events and commands but won't inherit the code sample.

2. Referencing parents: uri, id or kubernetes

A parent devfile can be referenced in 3 different ways. Using its id if it has been published in a registry:

parent:
    id: redhat/nodejs/11.6  # <-- the id format is <publisher>/<stack-name>/<version>
    registry: https://devfile-registry.io/

Using the URI if it has been published on a static http server (like gist or pastebin):

parent:
    uri: https://raw.githubusercontent.com/eclipse/che-devfile-

Using a Kubernetes resource name, namespace if it has been deployed on a Kubernete cluster as a DevWorkspaceTemplate:

parent:
    kubernetes:
        name: mydevworkspacetemplate
        namespace: mynamespace

3. Customizing parent configuration

When referencing a parent, a devfile should be able to customize its configuration. The syntax to customize a parent configuration is similar to kustomize:

parent:
  id: redhat/theia-vsx-stack/latest     # <--- Parent referenced by id 
  components:                                  # <--- Parent configuration can be customized
    - name: vsx-installer                    # <--- Should match the name of an existing parent component
      container:
         env:
            - name: VSX_LIST
               value: java-dbg.vsix,java.vsix
  commands:
    (...)
components:                          # The main Devfile body provides new elements, not inherited from the parent
  - name: tooling                     # <--- Should *not* match the name of a parent component 
    container:                           # <--- Components are added to parent's components
      image: busybox
events:                                   # <--- Events are only allowed in the main Devfile body,
  (...)                                       #        and the resulting events is the merged of the these events with the events of the parent.  

The place where an customizable element (component, command or project) should be defined is clearly driven by the following rule. It is either:

• inside the scope of the parent element itself if it is an override of an existing parent element,
• at the toplevel of the current devfile if it is a new element not present in the parent.

As a consequence;

• If a command / project / component element is added inside the parent scope, but there is no element with the same key (name or id) in the parent, this will trigger an error.
• If a command / project / component element is added in the main Devfle body, at the top-level, and an element with the same key (name or id) exists in the parent or in any plugin, then this will trigger an error.

As for the events element it is not really customizable: Due to the structure of the Events object, we can only add command bindings to some event. Overriding an existing command binding doesn't really makes sense. That's why we only allow events in the main devfile body, and not in the overrides. And the resulting events are the merge of the event objects coming from the parent, the plugins, and the main devfile body.

An Example

The parent devfile is published on github repo: https://raw.githubusercontent.com/eclipse/che-devfile-registry/master/devfiles/nodejs/devfile.yaml

The following devfile references it:

schemaVersion: 2.0.0
metadata:
  name: nodejs-app
parent:
    uri: https://(...)/nodejs/devfile.yaml # <--- Parent referenced by `uri`, registry `id`
                                           #      or `kubernetes` devworkspace
  components:                              # <--- Parent configuration can be customized
    - name: vsx-installer
      container:
         env:
            - name: VSX_LIST
               value: java-dbg.vsix,java.vsix
components:                               # <--- components are added to parent's components
  - name: tooling                       # <--- should not match the name of a parent component
    container:
      image: busybox
commands:                                 # <--- commands are added to parent's commands
   (...)

The traditional way of customizing an existing container image is to create a new container by extending that existing container as a parent image. For example, the java-maven dev file uses quay.io/eclipse/che-java11-maven:nightly to pull in Maven, rather than using the standard Maven docker image (hub.docker.com/_/maven).

However, this approach requires creating a ‘child’ image for every parent image that is to be used in Che (or at least those requiring customization).

This ‘nice to have’ feature proposes a second option for customizing containers: when a container is initially created in Che (from a devfile), files that are specified by an environment value can be automatically copied (on startup) into the container at specific locations using an initContainer component.

As proposed, a devfile writer is free to either create parent image, or to customize an existing image using an initContainer. This feature is not a replacement, it is just another tool in the devfile-writers arsenal, to make it easier to customize individual files/directories in container without fully extending them with a new image.

Example

components:
- container:
    image: maven:3.6.3-jdk-11
    volumeMounts:
    - name: scripts
      mountPath: "/home/user/scripts"
- initContainer:
  name: cp
  image: quay.io/devfile/cp"
  volumeMounts:
      - name: scripts
        mountPath: "/scripts"
  env:
      - name: SRC_PATHS
        value: "/src/resources/scripts/build.sh, /scripts2"
      - name: DEST_PATHS
        value: "/scripts, /scripts"
- volume:
    name: scripts
    emptyDir: {}

In this example, the container will be based on the standard maven image, but additional an additional script file (build.sh) and an additional directory (/scripts2) will be copied into the container at startup.

The source for both of these files would be the root of the project sources (where the devfile is located).

For example: If the dev file were hosted here then build.sh would be hosted here

<repo root>
    ├── devfile.yaml
    ├── scripts2
    └── resources
        └── scripts
            └── build.sh

With this feature, it is now possible to use an existing container image in Che, while still retaining the ability to add additional required files.

Right now, there isn’t a way to specify the size of a component’s volume in a devfile, and Che will use the size specified in che.properties (default 1Gi). As different component-types may have different volume requirements (e.g. the issue seen in redhat-developer/odo#2455), Odo will have a need to specify the size of volumes it interacts with, and rather than having an odo-specific setting, it would be nice if we could store this setting in the devfile,

This change might also be nice for Che, as it would allow components to configure the size of volumes they need (such as a maven cache volume that requires more than 1Gi), rather than relying on global Che defaults.

Something like:

---
components:
  - container:
      image: "maven"
      name: "maven"
      volumesMount: 
      - name: my-data-volume
  - volume:
     name: my-data-volume
     size: 5Gi

As odo is working on consuming the v2 spec, i noticed that the command group kind is called command group type https://github.com/devfile/kubernetes-api/blob/master/pkg/apis/workspaces/v1alpha1/commands.go#L20

// CommandGroupType describes the kind of command group.
// +kubebuilder:validation:Enum=build;run;test;debug
type CommandGroupType string

Can we please rename this to CommandGroupKind to be consistent with the discussion here in #27 and also devfile actually uses kind instead of type:

commands:
  - exec:
      id: devBuild
      component: runtime
      commandLine: npm install
      workingDir: ${CHE_PROJECTS_ROOT}/nodejs-web-app/app
      group:
        kind: build
        isDefault: true

Currently the following syntax are used to reference the spec in a che-plugin registry or in a yaml file published on the internet respectively:

components:
  - chePlugin:
      registry:
        id: redhat/vscode-yaml/latest
        registryUrl: "external-registry-url"
  - chePlugin:
      url: https://gist.../meta.yaml

That may be confusing. We should find a better name.

If other go projects want to add the devworkspace as a dependency they should be able to import only the core API. For instance chePlugins or cheEditors components should not be part of it.

Example with proposed changes:

commands:
 - exec:
     commandLine: "./buildSchema.sh"
     workingDir: "~/schemaScripts"
     component: build-tools
     alias: buildSchema

The goal of introducing this component is to ONLY let the user specify whether the app is resilient as a Deployment / DeploymentConfig / Stateful Set / Knative Service / any pod-spec-able workload

The tool and user should pick up the hint and do the needful.

---
components:
  - workload:
        apiVersion: service.knative.dev/v1
        type: Service

part of #49

We want to replace chePlugin and cheEditor with a new plugin component type that:

• can be specified using the devfile syntax (opposed to 1.0.0 meta.yaml, a devfile-like format)
• is not Che specific, plugins can be specified for odo or even desktop IDEs

Use cases

These are the use cases of devfile plugins (basically Che use cases):

• Reference one or more vs code extensions and the sidecar container that will host them (e.g. the java plugin)
• Reference a container component that is re-used in multiple devfiles/stacks (che-theia is an example)

Proposal

devfile.yaml
---
components:
  - plugin:
      name: 'java language server'
      id: redhat/dependency-analytics/latest  # <= reference plugins published
                                              #    in a plugin registry
      registry: https://plugin-registry.io/
  - plugin:
      name: 'k8s workspace plugin'
      kubernetes:
        namespace: mynamespace
        selector: mykey=myvalue               # <== reference a kubernetes
                                              #     devworkspacetemplate resource
  - plugin:
      name: ...
      uri: https://..../java8-plugin.yaml     # <== reference external raw devfile
      components:                             # <== customise plugin's configuration
        - kubernetes:
            name: ...
            reference: ...
        - container:
            name: vscode-java
            memoryLimit: 2Gi
        - volume:
            name: m2
            size: 2G

java8-plugin.yaml
---
schemaVersion: 2.0.0
metadata:
  publisher: redhat
  name: java8
  version: 0.57.0
  displayName: Language Support for Java 8
  title: Language Support for Java(TM) by Red Hat
  description: Java Linting, Intellisense, (...)
  icon: https://(..)/logo.svg
  repository: https://github.com/redhat-developer/vscode-java
  category: Language
  firstPublicationDate: "2020-02-20"
  pluginType: che-theia-vsx    # <== mandatory for plugins. Valid types
                               #     are che-theia-vsx, che-editor, che-theia-plugin,
                               #     che-theia-extension, generic-service and 
                               #     generic-ui
parent:
  id: redhat/theia-vsx-stack/latest
  components:
    - container:
        name: vsx-installer
        env:
          - name: VSX_LIST
               value: java-dbg.vsix,java.vsix
components:
  - kubernetes:
      name: ...
      reference: ...
  - container:
      image: ...che-sidecar-java
      name: vscode-java
      memoryLimit: "1500Mi"
      volumeMounts:
        - path: "/home/theia/.m2"
          name: m2
  - volume:
      name: m2

Spec Details

• Plugins are defined in separated devfiles
• There are 3 ways to reference a plugin: id, kubernetes, uri. This is consistent with how a parent can be referenced (c.f. #25).
• chePlugin is replaced by non Che specific plugin
• There is no editor/plugin distinction (a cheEditor is a plugin with pluginType: che-editor)
• Every plugins configuration can be customised using a kustomize approach. This is consistent with how a parent can be customized.
• Devfile metadata should contains new fields needed to publish it in a registry (the list is shown in the sample)
• Plugins should have a mandatory pluginType metadata that allow tools to figure out if they supported the plugin or not. The list of supported types are che-theia-vsx/che-editor/che-theia-plugin/che-theia-extension/generic-service/generic-ui.

Plugin Registry vs Devfile registry

Plugins are devfile too now. Hence we should rename the devfile registry as stack registry.

As a consequence there will be 2 registries (as before) but name differently:

• the stack registry: a registry of devfiles to be used as parents
• the plugin registry: a registry of devfiles to be used as plugins

Alternatives (rejected for Devfile 2.0.0)

We do not need plugins because we have parents

We may consider to eliminate the plugin concept from the devfile and instead use devfile composition (a.k.a. parents).

registry/uri/kubernetes as different plugins types

For consistency with the Kubernete API spec we may want to use polymorfic plugin type (as we do for components):

  - plugin:
      name: ...
      registryReference:
        id: ...
  - plugin:
      name: ...
      kubernetes:
         ...
  - plugin:
      name: ...
      uriReference:
        uri: ...

instead of

  - plugin:
      name: ...
      id: ...
  - plugin:
      name: ...
      kubernetes:
         ...
  - plugin:
      name: ...
      uri: ...

using parameters in parents

Instead of the simple customisation of a plugin seen above (a la kustomize) we may use parameters to customize some some attributes of a plugin:

parent:
  id: redhat/theia-vsx-stack/latest
  parameters:
    - VSX_LIST: java-dbg.vsix,java.vsix

instead of

parent:
  id: redhat/theia-vsx-template/latest
  components:
    - container:
         name: vsx-installer
         env:
            - name: VSX_LIST
               value: java-dbg.vsix,java.vsix

In current devfile spec there it is not possible to specify if a container will be run as part of the main dev environment pod[1] or if it should run in its own separated pod. In the v2.0.0 spec we would like to add a new field (dedicatedPod ) that can be applied to container components.

components:
  - container:
        image: maven
        dedicatedPod: true          # <-- Optional. Default value is false

[1] In Che the main pod is the pod where the editor and its plugins run.

Composite currently only has a label. But that appears to be something that would be visible in the Che UI. It should have an id as well so that tools like odo can reference composite commands with the following requirements:

• 2 distincts attributes: label and id
• both can coexist
• id is mandatory
• id used to reference parent devfiles commands (not label)

Sample devfile snippet to show example of usage/format:

projects:
 - name: "devworkspace-spec"
   git:
       location: "https://github.com/che-incubator/devworkspace-api"
       branch: "master"
commands:
 - exec:
     commandLine: "./buildSchema.sh"
     component: build-tools
     id: buildSchema
 - vscodeTask:
     id: openDevfile
     inlined:
       json       
 - composite:
     id: buildAndOpen
     label: Build schema and open devfile
     commands:
       - buildSchema
       - openDevfile
     parallel: false
components:
 - container:
     image: some container image with required build tools
     name: "build-tools"

The new type is container:

components:
  - container:
      image: busybox
      endpoints:
        - name: petclinic-web-server

Possible alternative may be image, containerImage or ociImage...

We need to agree on a convention to specify the step(s) of the development lifecycle a command belongs to. For example a given command will be used during the build step and another one is used for the run step. Tools like odo and che will then leverage this information to infer the sequence of the commands that need to be run.

A possible convention we can agree on is to specify the lifecycleStep as command annotation:

commands:
  - exec:
      annotations:
        lifecycleStep: build
      commandLine: "./build.sh"
      component: build-container
  - exec:
      annotations:
        lifecycleStep: run
      commandLine: "./run.sh"
      component: runtime-container

For now the only type of command that allows running code in a container is the exec command.
Its semantic is really:

execute a command line in a already-running container.

That doesn't seem to fit with the case when we want to just start a terminating container (as defined in a container component), with its default entry-point, possibly before the start of the Main workspace POD.
Similarly, we could need to apply a Job defined in a kubernetes component at some point in the workspace lifecycle (workspace stop for example)

So we should probably introduce a new type of command, for example, named apply, that would allow applying a component at some point in the workspace history.

  commands:
    - apply:
        id: aCommandToStartAnInitContainer
        component: someContainerComponent
    - apply:
        id: cleanupJobCommand
        component: AKubernetesComponentWithAJob
  events:
    preStart:
      - "aCommandToStartAnInitContainer" # <== Start of the workspace happens
                        #     after this terminating container is completed  
    postStop:
      - "cleanupJobCommand"  

An application stack (represented by a devfile) might need to provide additional files to support certain use cases. Some examples that come to mind are:

• Dockerfile for converting app into a container image
• shell scripts for supporting commands for iterative development
• source code providing some value add for enriching the application.

Today, we can use $PROJECTS_ROOT as the location where the user's application code is made available in my container using mountSources: true definition on a component/container.

However, there is no way to express this for files that we need are not coming form user's application:

• files from component's Devfile
• files from parent Devfile

The current options are:

• create a custom container image with added files
• add a step to do git clone as part of initializing the container

Both these approach can work but cause additional overhead for stack maintainer. I feel almost every stack will need this capability and it will be a big improvement in the experience if we can simplify access to stack code.

It would be good to have all files from the stack (devfile's location or explicitly declared in the devfile) made available as a volume mount in the container at location $STACK_ROOT.

We can use the mountSources: true flag or create a new one mountStack:true (or a similar name) to trigger this.

I was wondering rather than specifying a custom image in component container like this:

commands:
  - exec:
      id: buildSchema
      label: Build the schema
      commandLine: "./buildSchema.sh"
      component: build-tools
      group:
        kind: build
        isDefault: true
components:
  - container:
      image: "maysunfaisal/myImageWithBuildSchema.sh"
      mountSources: true
      sourceMapping: /home/src
      name: "build-tools"

if we should support syncing of build artifact(s) buildSchema.sh to the component container with a base image, for example maven:latest rather than providing maysunfaisal/myImageWithBuildSchema.sh.

This way, a devfile creator would not require the need to build a custom image with these artifacts burned in.

https://github.com/kubernetes/api/blob/master/core/v1/types.go#L2116

• Tooling can implement only what is supported.
• Devfile authors needn't learn a new structure.
• Outer-loop can easily specify pod-spec based definitions.

Could we support the Kubernetes Container API in the components.container ?

When you use a devfile in an IDE, the IDE might want to be able to display a devfile-specific welcome page. Likewise, a command line tool might want to link to information on how to use the devfile. In both of these cases we want the devfile to provide the link and not pull other files into the user's project, which could be easily removed or annoying to clean up.

For cases like these it would be nice to have an optional doc url in the devfile metadata.

Devfile 2.2.0

The outer-loop guidance for tools would be added in devfile 2.2.0. This is a work-in-progress epic.

Package files with a stack

1. Introduce the STACK_ROOT variable #46

Image Build guidance

2. Support component: dockerfile #51

Image Deployment guidance

3. component: workload #57
4. component: podTemplate #52
5. component: helm Chart #53
6. component: objects ( Template-style ) #54

Examples in the doc

Odo will have a need for multiple components to mount the same volume. There isn’t an option in devfiles right now to define a volume once, and have multiple components mount that same volume.

To work around this, we’re proposing adding a way to the devfile spec that allows you to define a volume once (a top level ‘volumes’ field), then easily add it to other components via a ‘volumeMount’ field. Like how Kubernetes handles volumes in containers, there'd be a top level volumes field and a volumeMounts field in the component:

projects:
  - name: "my-project"
volumes: 
  - name: my-data-volume
    size: 5Gi
components:
  - container:
      image: "busybox"
      name: "mycontainer"
      memoryLimit: "128M"
      mountSources: true
      volumeMount:
        - name: my-data-volume
          path: /some/folder
  - container:
      image: "busybox"
      name: "mycontainer"
      memoryLimit: "128M"
      mountSources: true
      volumeMount:
        - name: my-data-volume
          path: /some/other/folder

In a devfile that is being used to define a stack, there could be multiple starter projects, each providing a separate starter application (simple app, app integrated with Kafka, reactive app etc.)

Today we only have a name field to distinguish between them. But for developer to make a choice between these starter applications, having more details information on the capabilities you get with each project will be necessary.

If we can add starterProjects section, devfile creators will be able to use it for starter projects (only one should be selected by the user).

starterProjects:
  - name: "kafka-project"
    description: "Use this app to get a nodejs application for working with kafka"
  - name: "simple-project"
    description: "Use this app to get a simple "hello world" nodejs application"

starterProjects would have the same syntax as projects but a slightly different behavior: instead of cloning the git repo, only the source code would be fetched, the git metadata would not.

Moreover we would like to add the description field to both project and starterProject to help developers understand the content of the project.

We might want to introduce a syntax that would allow binding commands to a given workspace event.

A few use cases

• Open a file in the IDE at workspace startup
• Run build after clone of the project
• Copy files before workspace startup
• Backup files at workspace shutdown

Lifecycle Bindings

The lifecycle binding could be provided by a new top-level construct that could look like the following:

events:
  <lifecycleBinding>:
    - <commandName>

where can be one of the following:

• preStart
• postStart
• preStop
• postStop

Spec Details

⚠️ lifecycleBindings are processed sequentially. For example preStart commands will be invoked after every postCreate command has completed.

⚠️ In the case of Che-Theia, the postStart bindings should happen after all plugins and extensions have started, including project cloning. This means that those commands are not triggered until the user opens the IDE in his browser.

⚠️ Implementation of preStart commands may need execution of a workspace pod init container. That means that the component associated to the preStart command should not be part of the workspace pod. The same applies to postStop bindings.

⚠️ Commands associated to a lifecycleBinding should provide a mechanism to figure out when they have completed. That means that the command should be terminating or having a readiness probe.

⚠️ Commands associated to the same lifecycleBinding are not guarantee to be executed sequentially or in any order. To run commands in a given order a user should use a composite.

Example

components:
  - container:        # <== This component is associated with a **preStart** event only.
      name: "copier"  #     That means it won't be included in the workspace pod but
      image: ''       #     will be run before it (as an initContainer for example).
      env:
        - name: ""
          value: ""
      volume: ....
  - container:
      name: "maven"
      image: ''
      env:
        - name: ""
          value: ""
      volume: ....
  - plugin:
      id: theia
commands:
  - exec:
      name: "copyNeededFiles"
      component: "copier"
      commandLine: "cp somefile"
  - exec:
      name: "buildAll"
      component: "maven"
      commandLine: "mvn ..."
  - vsCodeTask:
      name: "openFile"
      component: "theia"
      inline: |
        { 
          xxxx
        }
events:
  preStart:
    - "copyNeededFiles" # <== Start of the workspace happens
                        #     after this command is completed
  postStart:
    - "buildAll"  # <== The Order of execution of the commands
    - "openFile"  #     under a same binding is not guarantee 

Not included in this spec

• Custom lifecycleBindings of components

When running exec commands from a devFile within a container we may need to run the commands as a different user to the user that is the default user within the container. I am therefore proposing we have a runAsUser within the exec, for example:

commands:
  - exec:
      id: init-deps
      name: init image dependancies
      component: java-openliberty-build
      commandLine: /project/bin/init-stack.sh
      runAsUser: root 
 - exec:
      id: stack-prep
      name: stack-prep
      component: java-openliberty-build
      commandLine: /project/bin/run-stack.sh prep
      runAsUser: java_user 
  - composite:
      id: setupContainer
      commands:
        - init-deps
        - stack-prep
      parallel: false
      group:
        name: init
        isDefault: true

So in this example there is a need to run some setup that requires to be run as root, and then some other setup that needs to be run as java_user.

It is not easy to switch userids within a shell script (bash) then different shell scripts need to be run and need to be run as different userids.

Opening this issue to have a discussion about where devfile registries can/should be hosted, and whether we want to have any guidance or recommendations about this. IMHO neither devfile registry providers nor clients need to all support the same mechanisms, but it would probably be worth defining what access methods are core (assuming only url) or what kind of https/auth mechanisms are expected as a base. e.g. do we expect registries to be hosted with no auth, basic auth, on GitHub (with or w/o auth), GHE, or Kube? We don't need to list these, just generally what should be expected.

As a stack/devfile author, I wish to convey outer-loop deployment guidance in the form of a list of parameterized kubernetes objects.

The format draws inspiration from Templates https://github.com/sbose78/nodejs-rest-http-crud/blob/master/.openshiftio/application.yaml#L38

---

 components:   
   -  id: deploy-node-app
      parameters:
         - name: ${image}
           value: quay.io/quarkus-quickstarts/getting-started-knative ## Default
       objects:
         - apiVersion: serving.knative.dev/v1alpha1
            kind: Service
            metadata:
              name: greeter
            spec:
              template:
                spec:
                  containers:
                    - image: ${image}
                      livenessProbe:
                      httpGet:
                        path: /health/live
                      readinessProbe:
                      httpGet:
                        path: /health/ready

If the objects component is associated with a dockerfile component (image build guidance) in a composite command, #51 , che/odo must associate the built image from the 'build' guidance using the image parameter.

- composite:
      id: build-and-deploy
      label: Build image and deploy nodejs app
      commands:
      - build-node-app
      - deploy-node-app
      parallel: false
      group: 
          kind: deploy

Handling of resources that comes from the parent devfile. Things to do:

1. When a stack is being used, it should download the files in the stack in a given registry and put them under the local source folder.
2. Devfile library has functions to download the devfiles. The behaviour should go to each parent and download the files from the parent tree.
3. In case of the file conflict caused by the same file exists in both child and parent, the files in the child devfiles will be used.
4. This behaviour should also be documented in the specification on how to handle supporting files that comes with the parent.
5. Given that the supporting files will be downloaded, we will not introduce the ${STACK_ROOT} variable. (see old description below for details)
6. This item is still targeted as part of the devfile 2.2 release

Old description for record:

Could we support the ${STACK_ROOT} variable to be able to express that a specific artifact comes with the stack and not necessarily with the source code/project ?

Example, I would like to specify that a Dockerfile build be done using the ${STACK_ROOT}/Dockerfile and not the ${PROJECTS_ROOT}/Dockerfile.build .

Such artifacts/files would be packaged and served off the devfile directory, example

• https://che-devfile-registry.openshift.io/devfiles/angular/
• https://github.com/sbose78/stack-registry/tree/sbose78-patch-1/stacks/quarkus-serverless

EDIT:
This is how tools could implement it : #55

Target Date: 09-09-2022

This is something that I've been thinking about for a while, and haven't really been able to find concrete answers on. As there will be multiple consumers of devfile registries (Che, Odo, Codewind), we need to standardize/document what requirements we impose on both a devfile registry, as well as its consumers, so that a user could point Che, Odo, etc at one, and have it properly work across all of them.

Generally, a devfile registry consists of a simple HTTP server, devfiles, and an index.json linking to those devfiles. But there's some unanswered things that need to be documented:

• What exactly does the index.json need to contain?
  • Is there a defined schema for it? What fields are required and which are optional?
  • Can it link to devfiles on external servers (maybe a separate issue)
• Is there a restriction on where the devfiles need to be hosted on the server?
  • On Che's devfile registry, the index.json and devfiles are hosted under /devfiles
• What metadata is required?
  • The Che devfile registry has a meta.yaml for each devfile. What requirements are imposed there?
    • Is there a schema for it?
  • Devfile v2 will have an expanded metadata field. Will metadata be pulled from there instead?

As a stack author, I wish to provide outer-loop deployment guidance in the form of a podTemplate. The tools may offer different choices based on that : pod, Deployment, KnativeService, StatefulSet, DeploymentConfig, etc.

---
components:
  -  id: deploy-node-app
     podTemplate:
        containers:
          - command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600'] 
             readinessProbe:
               httpGet: ""
               path: /api/health/readiness
               port: 8080
               scheme: HTTP

When associated with the corresponding build guidance ( example #51 ), che/odo must make a smart kustomize-styled replacement of the image field with built image from the 'dockerfile' component.

- composite:
      id: build-and-deploy
      label: Build image and deploy nodejs app
      commands:
      - build-node-app
      - deploy-node-image
      parallel: false
      group: 
          kind: deploy

In previous version we used to have a apiVersion attribute. We would like to rename it schemaVersion. The main reason is because the devfile format versioning doesn't adhere to the Kubernetes versioning scheme and calling it schemaVersion will avoid confusions.

Should it be possible to set environment variables that apply for a specific command? The current way to handle this would be to set the environment variables inline with the command but this can be cumbersome and reduce readability.

Sample of current devfile snippet to show example of usage/format:

- exec:
     commandLine: "MAVEN_CONFIG=~/mvn/config JAVA_OPTS="-XX:MaxRAMPercentage=50.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
               -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
               -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom
               -Duser.home=/home/user" ./buildSchema.sh"
     component: build-tools
     alias: buildSchema

Example with proposed changes:

commands:
 - exec:
     commandLine: "./buildSchema.sh"
     env:
     - name: MAVEN_CONFIG
       value: "~/mvn/config"
     - name: JAVA_OPTS
       value: "-XX:MaxRAMPercentage=50.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
               -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
               -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom
               -Duser.home=/home/user"
     component: build-tools
     alias: buildSchema

The devworkspace status field is currently lacking for many use cases. I'd like to propose adding a few fields that are common to other k8s objects in order to support management better.

New fields:

• .status.conditions - Type corev1.Condition[], stores information about various aspects of workspace state (c.f. .status.conditions on Pods.)
• .status.phase - String valued, stores current "phase" of workspace (e.g. running, started, stopped, failed)

The struct for conditions would be the common Kubernetes condition struct type.

Use cases

Conditions can be used to share information about progress in a workspace start process -- e.g. "storage ready", "deployment created", "deployment ready". They can also be used to give more detailed information about the last observed state of a workspace (e.g. the workspace deployment isn't ready because we're pulling images)

Phase is useful to track the status of the workspace -- if e.g. a routing class that is unsupported is specified, a workspace can be marked as "failed". If another service is waiting to show mainIdeUrl to the user, it needs to know the workspace is in a running state.

Proposal

status:
  condition:
  - lastTransitionTime: "2020-05-06T23:21:55Z"
    status: "True"
    type: ServiceAccountReady
  - lastTransitionTime: "2020-05-06T23:21:55Z"
    status: "True"
    type: RoutingReady
  - lastTransitionTime: "2020-05-06T23:21:55Z"
    status: "True"
    type: DeploymentReady
    reason: "OneWordReasonForTransition"
    message: "User-readable message about transition"
  - lastTransitionTime: "2020-05-06T23:21:55Z"
    status: "True"
    type: ComponentsReady
  phase: Running
  mainIdeUrl: https://my-workspace.my-cluster.com
  workspaceId: workspace-id
  additionalInfo: {}

Supported Phases:

• Running: Workspace is fully operational and can be accessed
• Starting: Workspace is currently being started
• Stopped: Workspace exists but is not running
• Stopping: Workspace is currently stopping (potentially waiting on preStop actions, e.g.
• Failed: There is an issue preventing this workspace from entering the Running phase.

For conditions, I don't have a concrete proposal for categories. Currently, in the che-workspace-operator (which would adopt this API), we use

const (
	WorkspaceComponentsReady     WorkspaceConditionType = "ComponentsReady"
	WorkspaceRoutingReady        WorkspaceConditionType = "RoutingReady"
	WorkspaceServiceAccountReady WorkspaceConditionType = "ServiceAccountReady"
	WorkspaceDeploymentReady     WorkspaceConditionType = "DeploymentReady"
)

but I don't view this list as complete.

Devfile 2.0.0 (milestone)

Here is a list of proposals to include in the next version of the Devfile format. Proposals need to be reviewed and approved. We are flagging issues with 👍once they have been reviewed and approved.

1. Repositories Setup

We would like to split Devfile related source code in 2 distinct repositories:

che-incubator/devworkspace-api

The Devfile schema is directly generated from the DevWorkspace API. PRs to change the Devfile format spec (documentation included) should be done here. Issues related to the format should be also created here.

redhat-developer/devfile

The Devfile specification doc and examples are published here. Only PRs to add new examples should be done here.

Subtasks related to the repositories setup:

• Move devfile repository to the devfile organisation 👍
• Move devworkspace-api repository to the devfile organisation 👍
• Setup a GH action to generate and publish the Devfile OpenAPI schema 👍
• Setup a GH action to generate and publish the Devfile Spec Documentation 👍

2. Changes for ODO

• Specify the source mapping path (CHE_PROJECT_ROOT) for component containers 👍
• Id and label for Composite commands 👍
• Specify size of volume for component 👍
• Allow volumes to be shared across components 👍
• On startup, automatically copy files/directories from the Git location of a dev file, to a container superseded by #32
• Environment variables for a specific command 👍
• Renaming workdir into workingDir 👍
• Commands group (application lifecycle) 👍
• Out of main pod containers 👍
• Stacks/Devfile matching rules 👍
• Devfile metadata: add a link to an external website 👍
• starterProjects and projects description

3. Changes for Che / Codewind

• Polymorphic component type UX in a Devfile 👍
• Renaming dockerimage component type 👍
• Devfile should have a schemaVersion attribute 👍
• The Devfile schema/samples should have optional version and name attributes 👍
• Names mandatory for k8s/openshift/container but not for plugins 👍
• Add lifecycle bindings to bind commands to specific events👍
• Better plugin mechanism 👍
• Containers endpoints (routes/ingresses) 👍
• Find a better naming for chePlugin spec reference 👍
• New type of command to apply a component 👍
• Run exec commands as specified user 👍

4. Changes to support Stacks definition

• Parent devfiles 👍

Odo will have a need for multiple components to mount the same volume. There isn’t an option in devfiles right now to define a volume once, and have multiple components mount that same volume.

To work around this, we’re proposing adding a way to the devfile spec that allows you to define a volume once (a top level ‘volumes’ field), then easily add it to other components via a ‘volumeMount’ field. This is similar to how Kubernetes pods handle volumes.

Example:

---
projects:
  - name: "my-project"
components:
  - container:
      image: "busybox"
      name: "mycontainer"
      memoryLimit: "128M"
      mountSources: true
      volumeMount:
        - name: my-data-volume
          path: /some/folder
  - kubernetes:
      referenceContent: |
        kind: Pod
        metadata:
          name: sample-pod
        spec:
          containers:
            image: docker.io/my-sample-image:latest
            volumeMounts:
            - mountPath: /some/path
              name: my-data-volume
  - volume:
     name: my-data-volume
     size: 5Gi

We discussed 2 options:

• include a init group in the context #27
• use an initContainer, the same mechanism used to solve #20
• a new devfile top level event attribute as defined in #32

In a devfile that is being used to define a stack, there are certain files that the stack may be looking for to decide if a given project can be run by a given stack or not, e.g. a maven stack may look for the pom.xml and a node.js stack may look for package.json.

If we can provide a way to allow a given stack to define those required resources, we can use it in tools like Che or odo to do quick check on project to suggest the stacks on a devfile repo that may support the a given application. An example can be:

parent.yaml
---
schemaVersion: 2.0.0
metadata:
  publisher: my-org
  name: maven-stack
  version: x.y.z
  matchingRules:
    existingFiles: ['pom.xml', 'aaa.xml']

devfile.yaml
---
schemaVersion: 2.0.0
metadata:
  name: my-maven-project
parent:
  id: organization/maven-stack/x.y.z
...

There may be other types other than existingFiles that we can support in the future.

Current devfile 1.0 container.endpoints syntax (link to the doc):

components:
  - alias: maven
    type: dockerimage
    image: eclipe/maven-jdk8:latest
    endpoints:
      - name: maven-server
        port: 3101
        attributes:
          protocol: http
          secure: 'true'
          public: 'true'
          discoverable: 'false'

Current syntax of che plugins (link to the doc):

spec:
  endpoints:
   -  name: "theia"
      public: true
      targetPort: 3100
      attributes:
        protocol: http
        type: ide
        secure: true
        cookiesAuthEnabled: true
        discoverable: false
  containers:
   - name: theia-ide
     image: "quay.io/eclipse/che-theia:7.10.0"
     ports:
         - exposedPort: 3100