sapcc / keppel Goto Github PK

View Code? Open in Web Editor NEW

101.0 44.0 7.0 16.56 MB

Regionally federated multi-tenant container image registry

License: Apache License 2.0

Go 99.02% Makefile 0.68% Shell 0.19% Dockerfile 0.12%

openstack container-registry kubernetes

keppel's Introduction

Keppel, a multi-tenant container image registry

In this document:

Overview
History
Terminology
Usage

In other documents:

Overview

When working with the container ecosystem (Docker, Kubernetes, etc.), you need a place to store your container images. The default choice is the Docker Registry, but a Docker Registry alone is not enough in productive deployments because you also need a compatible OAuth2 provider. A popular choice is Harbor, which bundles a Docker Registry, an auth provider, and some other tools. Another choice is Quay, which implements the registry protocol itself, but is otherwise very similar to Harbor.

However, Harbor's architecture is limited by its use of a single registry that is shared between all users. Most importantly, by putting the images of all users in the same storage, quota and usage tracking gets unnecessarily complicated. Keppel instead uses multi-tenant-aware storage drivers so that each customer gets their own separate storage space. Storage quota and usage can therefore be tracked by the backing storage. This orchestration is completely transparent to the user: A unified API endpoint is provided that multiplexes requests to their respective registry instances.

Keppel fully implements the OCI Distribution API, the standard API for container image registries. It also provides a custom API to control the multitenancy added by Keppel and to expose additional metadata about repositories, manifests and tags. Other unique features of Keppel include:

cross-regional federation: Keppel instances running in different geographical regions or different network segments can share their account name space and provide seamless replication between accounts of the same name on different instances.
online garbage collection: Unlike Docker Registry, Keppel can perform all garbage collection tasks without scheduled downtime or any other form of operator intervention.
vulnerability scanning: Keppel can use Trivy to perform vulnerability scans on its contents.

History

In its first year, leading up to 1.0, Keppel used to orchestrate a fleet of docker-registry instances to provide the OCI Distribution API. We hoped to save ourselves the work of reimplementing the full Distribution API, since Keppel would only have to reverse-proxy customer requests into their respective docker-registry. Over time, as Keppel's feature set grew, more and more API requests were intercepted to track metadata, validate requests and so forth. We ended up scrapping the docker-registry fleet entirely to make Keppel much easier to deploy and manage. It's now conceptually more similar to Quay than to Harbor, but retains its unique multi-tenant storage architecture.

Terminology

Within Keppel, an account is a namespace that gets its own backing storage. The account name is the first path element in the name of a repository. For example, consider the image keppel.example.com/foo/bar:latest. It's repository name is foo/bar, which means it's located in the Keppel account foo.

Access is controlled by the account's auth tenant or just tenant. Note that tenants and accounts are separate concepts: An account corresponds to one backing storage, and a tenant corresponds to an authentication/authorization scope. Each account is linked to exactly one auth tenant, but there can be multiple accounts linked to the same auth tenant.

Inside an account, you will find repositories containing blobs, manifests and tags. The meaning of those terms is the same as for any other Docker registry or OCI image registry, and is defined in the OCI Distribution API specification.

Usage

Build with make, install with make install or docker build. The resulting keppel binary contains client commands as well as server commands.

For how to use the client commands, run keppel --help.
For how to deploy the server components, please refer to the operator guide.

keppel's People

Contributors

Stargazers

Watchers

Forkers

steven-zou dddimcha brandongalbraith frewilhelm rgl zhurik andrewrothstein

keppel's Issues

Request for Feedback: OCI image-spec 1.1.0

OCI is approaching a 1.1.0 release on the image-spec and distribution-spec, so we are reaching out to implementations to see if you have any concerns implementing support for those changes that we should address before a GA release.

A full diff of the changes can be viewed at:

image-spec: opencontainers/image-spec@v1.0.2...opencontainers:image-spec:main
distribution-spec: opencontainers/distribution-spec@v1.0.1...main

I'm happy to take feedback here, you can raise an issue in OCI, or respond in our tracking issue at opencontainers/image-spec#1093.

Failed to push multi-arch image

Pushing an example multi-arch (aka multi-architecture) image fails with:

source_image='docker.io/ruilopes/example-docker-buildx-go:v1.10.0'
image='keppel.test:9006/ruilopes/example-docker-buildx-go:v1.10.0'
#platform='linux/amd64'
#platform='windows/amd64:10.0.20348.825'
platform='all'
crane copy --insecure --platform "$platform" "$source_image" "$image"

2022/08/27 16:14:20 Copying from docker.io/ruilopes/example-docker-buildx-go:v1.10.0 to keppel.test:9006/ruilopes/example-docker-buildx-go:v1.10.0
2022/08/27 16:14:22 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:22 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A460e86bd30d96aa1df7697b4fb8184599645dd748a9c473b3fa921aafe42bf56&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:22 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Aad526f14af1520ff0694848b88732134fbae909cdef95c02c49efcc7a2d6ab7b&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:22 pushed blob: sha256:ad526f14af1520ff0694848b88732134fbae909cdef95c02c49efcc7a2d6ab7b
2022/08/27 16:14:22 pushed blob: sha256:460e86bd30d96aa1df7697b4fb8184599645dd748a9c473b3fa921aafe42bf56
2022/08/27 16:14:23 pushed blob: sha256:1efc276f4ff952c055dea726cfc96ec6a4fdb8b62d9eed816bd2b788f2860ad7
2022/08/27 16:14:23 keppel.test:9006/ruilopes/example-docker-buildx-go@sha256:2ebebdde436cbbfea50bf5a4eb20b673029dbe7a68577b4fcf42aec122b5988a: digest: sha256:2ebebdde436cbbfea50bf5a4eb20b673029dbe7a68577b4fcf42aec122b5988a size: 740
2022/08/27 16:14:23 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Ab45a7254211aad549686c27ca941f0d3e9897a56c425fd2bafe3b19e9ea67c4c&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:23 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Aa9fe95647e78b5516c7e2327355b6996e2ea295cd76ae242cbfe87f016b4e760&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:23 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A79d4b8007a95f1971e12d2259d85c2e9f845a2c33541c4ea2893bc34d4f4b013&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:23 pushed blob: sha256:79d4b8007a95f1971e12d2259d85c2e9f845a2c33541c4ea2893bc34d4f4b013
2022/08/27 16:14:23 pushed blob: sha256:b45a7254211aad549686c27ca941f0d3e9897a56c425fd2bafe3b19e9ea67c4c
2022/08/27 16:14:24 pushed blob: sha256:a9fe95647e78b5516c7e2327355b6996e2ea295cd76ae242cbfe87f016b4e760
2022/08/27 16:14:24 keppel.test:9006/ruilopes/example-docker-buildx-go@sha256:ffb3b11112406f18c84308db56d2feabd09e88130671c2d23f44981fc8c2f047: digest: sha256:ffb3b11112406f18c84308db56d2feabd09e88130671c2d23f44981fc8c2f047 size: 740
2022/08/27 16:14:24 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A1dd75a3a9c893a7dc313f683dd62464b7eab6c6d522ee62c8a17022631830f32&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:24 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A220f1d0b5fbaa132ac1b5a1d18977092c24de291ecb3c673c2fa032084f0079c&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:24 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A714f60a76c9a842027926b69d9645c4a089c6f1bbfcbd0854d5de090bbc90251&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:24 pushed blob: sha256:714f60a76c9a842027926b69d9645c4a089c6f1bbfcbd0854d5de090bbc90251
2022/08/27 16:14:24 pushed blob: sha256:220f1d0b5fbaa132ac1b5a1d18977092c24de291ecb3c673c2fa032084f0079c
2022/08/27 16:14:24 pushed blob: sha256:1dd75a3a9c893a7dc313f683dd62464b7eab6c6d522ee62c8a17022631830f32
2022/08/27 16:14:24 keppel.test:9006/ruilopes/example-docker-buildx-go@sha256:0e5aef05e2cfc763739f1395de46a862dc21714a1258471c08e55be0d148db0b: digest: sha256:0e5aef05e2cfc763739f1395de46a862dc21714a1258471c08e55be0d148db0b size: 740
2022/08/27 16:14:25 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Ab5fa2ec8efc4e66fc023da0e4a2915a7eda3dc307c2aeb8b95eb60761d1fc204&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:25 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3Abbe151a29e8136b6d96996e67f266932d724cee589d14cc4ea21aa69dc305c7f&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:25 retrying without mount: POST http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/blobs/uploads/?from=ruilopes%2Fexample-docker-buildx-go&mount=sha256%3A896a00309dd817a9fda0ecf3ab240aa075f802dd203218cc10b9ec6bdd27aa75&origin=REDACTED: BLOB_UNKNOWN: blob does not exist in source repository
2022/08/27 16:14:25 pushed blob: sha256:896a00309dd817a9fda0ecf3ab240aa075f802dd203218cc10b9ec6bdd27aa75
2022/08/27 16:14:25 pushed blob: sha256:b5fa2ec8efc4e66fc023da0e4a2915a7eda3dc307c2aeb8b95eb60761d1fc204
2022/08/27 16:14:25 pushed blob: sha256:bbe151a29e8136b6d96996e67f266932d724cee589d14cc4ea21aa69dc305c7f
Error: failed to copy index: PUT http://keppel.test:9006/v2/ruilopes/example-docker-buildx-go/manifests/sha256:de7f83381ed864f79e21f2a7a80d89896f88ad8b55c64fbfbe20be6b232ad818: MANIFEST_BLOB_UNKNOWN: manifest blob unknown to registry; sha256:5d24e1a2f5c566b0afb1e46fc24e5cec821c8ebf44220276a95a2b91f44a2f2a

For reference, this is how the manifest looks:

crane manifest --insecure "$source_image" | jq

{
  "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:2ebebdde436cbbfea50bf5a4eb20b673029dbe7a68577b4fcf42aec122b5988a",
      "size": 740,
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:ffb3b11112406f18c84308db56d2feabd09e88130671c2d23f44981fc8c2f047",
      "size": 740,
      "platform": {
        "architecture": "arm64",
        "os": "linux"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:0e5aef05e2cfc763739f1395de46a862dc21714a1258471c08e55be0d148db0b",
      "size": 740,
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:de7f83381ed864f79e21f2a7a80d89896f88ad8b55c64fbfbe20be6b232ad818",
      "size": 1128,
      "platform": {
        "architecture": "amd64",
        "os": "windows",
        "os.version": "10.0.17763.3165"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:b91cc7beb040a3c607d8000d1f617a16fee2c59f112e66d8ed47cc85c119970a",
      "size": 1128,
      "platform": {
        "architecture": "amd64",
        "os": "windows",
        "os.version": "10.0.20348.825"
      }
    }
  ]
}

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Vulnerabilities

Renovate has not found any CVEs on osv.dev.

Detected dependencies

github-actions

.github/workflows/checks.yaml

actions/checkout v4

actions/setup-go v5

golangci/golangci-lint-action v6

golang/govulncheck-action v1

reviewdog/action-misspell v1

.github/workflows/ci.yaml

actions/checkout v4

actions/setup-go v5

actions/checkout v4

actions/setup-go v5

postgres 16

.github/workflows/codeql.yaml

actions/checkout v4

actions/setup-go v5

github/codeql-action v3

github/codeql-action v3

github/codeql-action v3

.github/workflows/oci-distribution-conformance.yml

actions/checkout v4

actions/setup-go v5

actions/upload-artifact v4

postgres 16

gomod

go.mod

go 1.23

github.com/alicebob/miniredis/v2 v2.33.0

github.com/aquasecurity/trivy v0.54.1

github.com/databus23/goslo.policy v0.0.0-20210929125152-81bf2876dbdb@81bf2876dbdb

github.com/dlmiddlecote/sqlstats v1.0.2

github.com/docker/distribution v2.8.3+incompatible

github.com/go-gorp/gorp/v3 v3.1.0

github.com/go-redis/redis_rate/v10 v10.0.1

github.com/gofrs/uuid/v5 v5.3.0

github.com/golang-jwt/jwt/v5 v5.2.1

github.com/gophercloud/gophercloud/v2 v2.1.0

github.com/gophercloud/utils/v2 v2.0.0-20240812072210-8ce1fc0f2894@8ce1fc0f2894

github.com/gorilla/mux v1.8.1

github.com/majewsky/schwift/v2 v2.0.0

github.com/opencontainers/go-digest v1.0.0

github.com/opencontainers/image-spec v1.1.0

github.com/prometheus/client_golang v1.20.0

github.com/redis/go-redis/v9 v9.6.1

github.com/rs/cors v1.11.0

github.com/sapcc/go-api-declarations v1.12.4

github.com/sapcc/go-bits v0.0.0-20240815085238-fce0691187a2@fce0691187a2

github.com/spf13/cobra v1.8.1

github.com/timewasted/go-accept-headers v0.0.0-20130320203746-c78f304b1b09@c78f304b1b09

go.uber.org/automaxprocs v1.5.3

golang.org/x/crypto v0.26.0

Check this box to trigger a request for Renovate to run again on this repository

Local storage driver

Is there a storage driver that could use the local filesystem? Or a remote S3 like one (e.g. minio)?

expose labels in manifest list endpoint

Docker images can have labels, as defined by the LABEL command in a Dockerfile. For instance, in Keppel's own Dockerfile, we have:

keppel/Dockerfile

Line 10 in 3ea36f3

LABEL source_repository="https://github.com/sapcc/keppel"

Right now, if a user wants to retrieve those label values for a certain image, they have to download the manifest, then find the image configuration blob within the manifest, then download the image configuration blob, then parse it. That's a lot of work, and what's worse, several of these actions count towards rate limits, which makes surveying a large repo for image labels impractically slow. Keppel should provide image labels in its own API in a more convenient spot.

In fact, Keppel already looks at the image labels to enforce the required_labels option that can be enabled on the account level, so it should be fairly straight-forward to retain the label values in Keppel's DB for quick access. For simplicity's sake, it will be sufficient to store the labels as JSON in a text column on the manifest object.

add a text field labels_json to the manifests table, and extend the keppel.Manifest struct accordingly
fill this field during func validateAndStoreManifestCommon (the existing testcase for the required_labels option can be used to validate that the field is populated correctly)
add a Labels field to struct keppelv1.Manifest (probably using the json.RawMessage type) and fill it in the handler method for GET /keppel/v1/repositories/:repo/manifests
adjust the unit test for the API, as well as the API docs, accordingly

No extra migration should be needed for existing images. validateAndStoreManifestCommon is called periodically on existing manifests by the janitor, so the new field should be slowly populated once the changes are rolled out.

Awesome Readme!

Hello there,

I liked very much how it was written.
Very clear and nice explanation. Congrats!

Sorry for abusing the issue system.
Please close this issue :D

Cheers!

checklist

This may be split out into multiple issues at some point. For now, let's keep the ceremony to a minimum.

functionality
- MVP: docker pull/push works for authenticated users
- GET /v2/_catalog
- orchestrate registry fleet via k8s (connect objects with annotations for reliable cleanup)
- allow account deletion (similar to swift-account-reaper: mark account as deleted, delete from DB only after all data has been purged by a background job)
- repository RBAC (enable docker pull/push for unauthenticated users)
- federation
code quality
- automated tests
  - Keppel v1 API
  - Registry v2 API
  - auth API
- automatically run golint/govet/gofmt
- resolve all of rg TODO pkg
release quality
- Prometheus metrics
- API spec for the /keppel/ parts
- install documentation (explain config options in more detail)
- operator handbook for recurring actions (issuer cert rotation)

Offloading accounts & permissions to external source

As it looks right now, Keppel seems to depend on operators creating accounts in the Keppel database and set permissions for them there. This does not really fit into many environments, as operators generally store their authentication & authorization data in a centralized store and distribute it at that layer.

Would there be a possibility to consider moving it into an interface so it can be extended? We'd love to handle all authz/n externally from Keppel.

JWT authentication and authorization

We've finally gotten to the point of experimenting with Keppel.

I started today with implementing Ory Hydra support, but quickly reverted and went on to supporting JWTs better. We plan to have our JWT token as part of the docker login password, with an empty username (or a single letter). This way we can make it a bit more custom and not rely on Dockers built in OAuth support, which isn't flexible enough for us.

When it comes to supporting authentication directly with JWTs, internal/auth/token.parseToken() basically does what we are trying to do, but perhaps with a few Keppel specific features.

Would it be a good idea to move the JWT support into it's own driver and make it more configurable? It would be great for example if we could select which fields in Claims should be included for scopes.

Happy to discuss any solution that might work!

Transport received Server's graceful shutdown GOAWAY

Hi guys,

Sometimes I'm having this problem using keppel. To work around this, you need to retry the job to build/push the image again.

How can I solve this?

error pushing image: failed to push to destination keppel.xxxxxxxxx:1.0.0.6: Patch "https://keppel.xxxxxxxxx/blobs/uploads/1a7f8560-6aca-4354-8adc-3e9c19fae0fa": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error