Git Product home page Git Product logo

build-definitions's Introduction

build-definitions

This repository contains components that are installed or managed by the managed CI and Build Team.

This includes default Pipelines and Tasks. You need to have bootstrapped a working appstudio configuration from (see https://github.com/redhat-appstudio/infra-deployments) for the dev of pipelines or new tasks.

Pipelines and Tasks are delivered into App Studio via quay organization redhat-appstudio-tekton-catalog. Pipelines are bundled and pushed into repositories prefixed with pipeline- and tagged with $GIT_SHA (tag will be updated with every change). Tasks are bundled and pushed into repositories prefixed with task- and tagged with $VERSION where VERSION is the task version (tag is updated when the task file contains any change in the PR)

Currently a set of utilities are bundled with App Studio in quay.io/redhat-appstudio/appstudio-utils:$GIT_SHA as a convenience but tasks may be run from different per-task containers.

Building

Script hack/build-and-push.sh creates bundles for pipelines, tasks and create appstudio-utils image. Images are pushed into your quay.io repository. You will need to set MY_QUAY_USER to use this feature and be logged into quay.io on your workstation. Once you run the hack/build-and-push.sh all pipelines will come from your bundle instead of from the default installed by gitops into the cluster.

Note

If you're using Mac OS, you need to install GNU coreutils before running the hack/build-and-push.sh script:

brew install coreutils

There is an option to push all bundles to a single quay.io repository (this method is used in PR testing). It is used by setting TEST_REPO_NAME environment variable. Bundle names are then specified in the container image tag, i.e. quay.io/<quay-user>/$TEST_REPO_NAME:<bundle-name>-<tag>

Pipelines

The pipelines can be found in the pipelines directory.

  • core-services: contains pipeline for the CI of Stonesoup core services e.g. application-service and build-service.
  • template-build: contains common template used to generate docker-build, fbc-builder, java-builder and nodejs-builder pipelines

Tasks

The tasks can be found in the tasks directories. Tasks are bundled and used by bundled pipelines. Tasks are not stored in the Cluster. For quick local innerloop style task development, you may install new Tasks in your local namespace manually and create your pipelines as well as the base task image to test new function. Tasks can be installed into local namespace using oc apply -k tasks/appstudio-utils/util-tasks.

There is a container which is used to support multiple set of tasks called quay.io/redhat-appstudio/appstudio-utils:GIT_SHA , which is a single container which is used by multiple tasks. Tasks may also be in their own container as well however many simple tasks are utilities and will be packaged for app studio in a single container. Tasks can rely on other tasks in the system which are co-packed in a container allowing combined tasks (build-only vs build-deploy) which use the same core implementations.

Shellspec tests can be run by invoking hack/test-shellspec.sh.

Release

Release is done by setting env variable MY_QUAY_USER=redhat-appstudio-tekton-catalog, BUILD_TAG=$(git rev-parse HEAD) and running hack/build-and-push.sh.

Versioning

When the task update changes the interface (eg. change of parameters, workspaces or results names) then a new version of the task should be created. The folder with the new version must contain MIGRATION.md with instructions on how to update the current pipeline file in user's .tekton folder.

Adding a new parameter with a default value does not require the task version increase.

Task version increase must be approved by Project Manager.

Testing

Script ./hack/test-builds.sh creates pipelines and tasks directly in current namespace and executes there test builds. By setting the environment variable MY_QUAY_USER the images will be pushed into user's quay repository, in that case creation of secret named redhat-appstudio-staginguser-pull-secret is required.

Script ./hack/test-build.sh provides way to test on custom git repository and pipeline. Usage example: ./hack/test-build.sh https://github.com/jduimovich/spring-petclinic java-builder.

Compliance

Task definitions must comply to Enterprise Contract policies. Currently, there are two policy configurations. The all-tasks policy configuration applies to all Task definitions, while the build-tasks policy configuration applies only to build Task definitions. A build Task, i.e. one that produces a container image, must abide to both policy configurations.

build-definitions's People

Contributors

arewm avatar brunoapimentel avatar chmeliik avatar dirgim avatar gabemontero avatar gbenhaim avatar hongweiliu17 avatar jduimovich avatar jencull avatar jinqi7 avatar joejstuart avatar jsztuka avatar lcarva avatar martinbasti avatar michkov avatar mkosiarc avatar mmorhun avatar psturc avatar rcerven avatar renovate[bot] avatar rh-tap-build-team[bot] avatar robnester-rh avatar sbose78 avatar simonbaird avatar sonam1412 avatar staticf0x avatar stuartwdouglas avatar tisutisu avatar tkdchen avatar zregvart avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

build-definitions's Issues

Simplify how images are built in this repo

This repo creates multiple images for delivery into the app studio platform.
Default pipelines are delivered into quay.io/redhat-appstudio/build-templates-bundle
Base Image for the ClusterTasks are in quay.io/redhat-appstudio/appstudio-utils

The versions for these images are maintained separately in each directory which leads to confusion
Options

  1. We change the release build to build both images at the same version. This requires the cluster tasks which are defined in the app studio environment to get version bumped every time any new release occurs.
  2. We write some update scripts which update the infra-deployment repository contents with the correct versions of the two (or more) images to prevent the wrong images updates.

I think we should do both -- and embed the automation in the .ci script for building and then installing new pipelines into app studio.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository.

  • WARN: No docker auth found - returning

⚠ Dependency Lookup Warnings ⚠

  • Renovate failed to look up the following dependencies: Failed to look up docker package registry.redhat.io/openshift4/ose-tools-rhel8, Failed to look up docker package registry.redhat.io/openshift4/ose-cli, Could not determine new digest for update (datasource: docker).

Files affected: .tekton/pull-request.yaml, task/init/0.1/init.yaml, task/s2i-java/0.1/s2i-java.yaml, task/s2i-nodejs/0.1/s2i-nodejs.yaml, task/summary/0.1/summary.yaml


Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

ansible
.tekton/tasks/yaml-lint.yaml
dockerfile
appstudio-utils/Dockerfile
syft.Dockerfile
  • registry.access.redhat.com/ubi8 8.7-1090
github-actions
.github/workflows/shellspec.yaml
  • actions/checkout v3
  • jerop/tkn v0.1.0
tekton
.tekton/pull-request.yaml
  • registry.redhat.io/openshift4/ose-tools-rhel8 v4.12
  • registry.redhat.io/openshift4/ose-cli v4.12@sha256:9f0cdc00b1b1a3c17411e50653253b9f6bb5329ea4fb82ad983790a6dbf2d9ad
.tekton/push.yaml
.tekton/tasks/buildah.yaml
.tekton/tasks/yaml-lint.yaml
  • docker.io/cytopia/yamllint 1.26@sha256:1bf8270a671a2e5f2fea8ac2e80164d627e0c5fa083759862bbde80628f942b2
task/buildah/0.1/buildah.yaml
  • quay.io/redhat-appstudio/buildah v1.28
  • quay.io/redhat-appstudio/syft v0.47.0
  • registry.access.redhat.com/ubi9/python-39 1-108
  • quay.io/redhat-appstudio/cosign v1.13.1
task/clair-scan/0.1/clair-scan.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/clamav-scan/0.1/clamav-scan.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/deprecated-image-check/0.1/deprecated-image-check.yaml
  • registry.access.redhat.com/ubi8/ubi-minimal 8.7-1085@sha256:dc06ba83c6f47fc0a2bca516a9b99f1cf8ef37331fd460f4ca55579a815ee6cb
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/fbc-related-image-check/0.1/fbc-related-image-check.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/fbc-validation/0.1/fbc-validation.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/git-clone/0.1/git-clone.yaml
task/init/0.1/init.yaml
  • registry.redhat.io/openshift4/ose-tools-rhel8 v4.12@sha256:253d042ecfad7b64593112a4aa3f528d39cb5fe738852e44f009db87964cf051
task/prefetch-dependencies/0.1/prefetch-dependencies.yaml
  • quay.io/containerbuildsystem/cachi2 sha256:bd8abcd9782af134d3c0d2f91cd469424ce413195dcfc050a7321ae0b29f5507
task/s2i-java/0.1/s2i-java.yaml
  • registry.redhat.io/ocp-tools-4-tech-preview/source-to-image-rhel8 sha256:637c15600359cb45bc01445b5e811b6240ca239f0ebfe406b50146e34f68f631
  • quay.io/redhat-appstudio/syft v0.47.0
  • registry.access.redhat.com/ubi8/python-39 1-105
  • quay.io/redhat-appstudio/cosign v1.13.1
task/s2i-nodejs/0.1/s2i-nodejs.yaml
  • registry.redhat.io/ocp-tools-4-tech-preview/source-to-image-rhel8 sha256:e518e05a730ae066e371a4bd36a5af9cedc8686fd04bd59648d20ea0a486d7e5
  • quay.io/redhat-appstudio/syft v0.47.0
  • registry.access.redhat.com/ubi9/python-39 1-108
  • quay.io/redhat-appstudio/cosign v1.13.1
task/sanity-inspect-image/0.1/sanity-inspect-image.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/sanity-label-check/0.1/sanity-label-check.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/sast-go/0.1/sast-go.yaml
task/sast-java-sec-check/0.1/sast-java-sec-check.yaml
task/sast-snyk-check/0.1/sast-snyk-check.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/sbom-json-check/0.1/sbom-json-check.yaml
  • quay.io/redhat-appstudio/hacbs-test v1.0.11@sha256:acf4e35adfbe16916d400f36b616236d872c2527c7618ffc6758ae930e353668
task/summary/0.1/summary.yaml
  • registry.redhat.io/openshift4/ose-cli v4.12@sha256:9f0cdc00b1b1a3c17411e50653253b9f6bb5329ea4fb82ad983790a6dbf2d9ad
task/tkn-bundle/0.1/tkn-bundle.yaml
  • quay.io/openshift-pipeline/openshift-pipelines-cli-tkn 1.10
task/update-infra-deployments/0.1/update-infra-deployments.yaml
  • quay.io/chmouel/github-app-token sha256:b4f2af12e9beea68055995ccdbdb86cfe1be97688c618117e5da2243dc1da18e

  • Check this box to trigger a request for Renovate to run again on this repository

Consume the Pipeline definitions for App Studio Components

Upload Snyk SAST results to snyk.io

For RHTAS prodsec compliance, we have been asked to provide links to systems or documents where it is shown how SAST findings are analyzed and processed by the team. With the current Konflux configuration, it is not straightforward to do this. Snyk Code results are stored as pipeline results, but are not shipped anywhere else during a pipelinerun. It would be ideal for us if the snyk results could be published to snyk.io using the --report option for snyk code test. This feature appears to be in private beta, but it has been for about a year. Perhaps we could use it?

Replace ClusterTask by something else

I am trying to use the build-definitions with https://github.com/openshift-pipelines/pipelines-service.

With some fixes in pipelines-service (https://github.com/openshift-pipelines/pipelines-service/compare/main...guillaumerose:builddef?expand=1), the PR is triggered but I hit an issue: ClusterTask doesn't exist on KCP and all definitions are using them.
How could we solve that?

Events:
  Type     Reason         Age   From         Message
  ----     ------         ----  ----         -------
  Normal   Started        29s   PipelineRun  
  Warning  Failed         27s   PipelineRun  Pipeline /noop can't be Run; it contains Tasks that don't exist: Couldn't retrieve Task "openshift-client": the server could not find the requested resource (get clustertasks.tekton.dev openshift-client)
  Warning  InternalError  27s   PipelineRun  1 error occurred:
           * Couldn't retrieve Task "openshift-client": the server could not find the requested resource (get clustertasks.tekton.dev openshift-client)

Build the Pipelines Catalog using tekton on Staging

Acceptance criteria:

Upon, merge to https://github.com/redhat-appstudio/build-definitions , everything inside https://github.com/redhat-appstudio/build-definitions/tree/main/pipelines/build-templates-bundle should be built into a new bundle as deemed relevant.

Implementation notes

RFE: way to set up a "install root" within a buildah task

In various contexts, we want to install packages into a sub directory of the root directory, and then use them as the root directory of a subsequent stage. Examples include:

In all of these cases, we really don't want to install into a bare directory - this will result in weirdness like redirections to /dev/null going into a actual file. The directory at least needs a skeleton /dev populated and would ideally have /proc and /sys as well.

There are two basic ways that this could be enabled:

  • Allow running buildah build with --cap-add=all --security-opt=label=disable ; this is sufficient to allow the Containerfile to create its own mount namespace and populate the root directory itself.

  • Instead specify the install directory as a task parameter, and have the buildah task add something like:

           --device=/dev/null:"$INSTALL_ROOT"/dev/null
           --device=/dev/random:"$INSTALL_ROOT"/dev/random
           --device=/dev/urandom:"$INSTALL_ROOT"/dev/urandom
           --device=/dev/zero:"$INSTALL_ROOT"/dev/zero
    

The advantage of this is that it is more flexible and satisfies some other needs for nested sandboxing1. It may the only way that actually works for the bootc case [@cgwalters - is that right?]. On the other hand, the second approach is more obviously secure. 2

Footnotes

  1. To delve some into this, if we were using buildah with user namespaces, then --cap-add=all would not be necessary since the Containerfile could create a nested user namespace where it had the SYS_ADMIN capability, and then create a mount namespace and set things up there. But with --isolation=chroot the run commands are being run in an active chroot, and the Linux kernel disables user namespace creation when a chroot is active, so we have to actually give the SYS_ADMIN capability to the subprocess so it can create a mount namespace

  2. --cap-add=all will not compromise overall system security, since we're counting on the pod running the task for that, and buildah --isolation=chroot is already not that strong. And of course, an actual malicious component could specify their own pipeline with its own build steps already. The main weakness it introduces is that it could make it easier for malicious content pulled into the Containerfile during the build process to mess with the build artifacts and build metadata.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.