jenkins-x / bucketrepo Goto Github PK

An artifacts repository using cloud storage (a lightweight replacement for Nexus)

License: MIT License

Dockerfile 0.43% Makefile 3.75% Go 65.33% Smarty 2.76% Shell 26.38% Mustache 1.33%

artifacts nexus nexus-repository cloud hacktoberfest

bucketrepo's Introduction

Bucket Repo

This is a lightweight artifacts repository with low memory footprint which can be used as a minimal replacement for Nexus. It is able to cache artifacts from a remote repository on a local filesystem volume and also to store them on a cloud storage bucket via go-cloud.

It can be deployed either as a side-car container to a Kubernetes build pod or as a standalone service.

Getting Started

Configuration

The default configuration enables only the local file cache:

http:
    addr: ":8080"

storage:
    enabled: false
    bucket_url: "gs://bucketrepo"

cache:
    base_dir: "/tmp/bucketrepo"

repositories:
    - url: "https://repo1.maven.org/maven2"
    - url: "http://uk.maven.org/maven2/"

The header (for example Bearer token authentication) and timeout can be modified for every remote repository:

...

repositories:
    - url: "https://repo1.maven.org/maven2"
      timeout: "30s"
    - url: "http://uk.maven.org/maven2/"
    - url: "http://my.private.maven.repo"
      timeout: "10s"
      header:
          Authorization: "Bearer <Token>"

The cloud storage can be enabled by providing a bucket URL:

storage:
    enabled: true
    bucket_url: "gs://mybucket"
    # if necessary is possible to use Path prefix
    prefix: "myfolder"

Also the TLS and basic authentication can be configured with:

http:
    addr: ":8080"
    https: true
    crt: "/certs/domain.crt"
    key: "/certs/domain.key"
    username: "myuser"
    password: "mypassword"

Note that the basic authentication is turned off when HTTPS is disabled.

You can make artifacts not used in a while be removed from disk storage (cloud storage is not touched):

cache:
    base_dir: "/tmp/bucketrepo"
    clean_interval: "24h"
    cache_time: "720h"

The clean interval of 24 hours is the default, while the cache time doesn't have a default. This means that cleaning of the cache isn't enabled by default.

Note

For this to work the access times needs to be recorded in the file system used for caching. Typically it is.

If you do want cloud storage to be cleaned you can for example in the case of s3 add a lifecycle policy to the bucket.

Supported Artifacts

This repository has been tested with maven and helm tools, but it can also store other artifacts.

Installing

Kubernetes

The repository service can be installed in a Kubernetes cluster using helm. First, you need to add the jenkins-x chart repository to your helm repositories:

helm repo add jx3 https://jenkins-x-charts.github.io/repo
helm repo update

You can now install the chart with:

helm install jx3/bucketrepo --name bucketrepo

S3 compatible buckets

When using an S3 compatible bucket deployed locally (like Minio, Ceph...) you might need to configure bucketrepo to trust the SSL certificate for the bucket.

In order to that, you can add AWS_CA_BUNDLE to envSecrets with path of the CA file, and mount that file using extraConfig that looks like this:

extraConfig:
  ca-certificates.crt: |
    -----BEGIN CERTIFICATE-----
    ...

the certificate can be retrieved using:

kubectl -n minio get secrets minio1-tls -o yaml | ksd (public.crt)

the bucketUrl should look like this:

"s3://bucketrepo?endpoint=https://minio.minio.svc.cluster.local&s3ForcePathStyle=true&region=us-east-1"

Locally

The repository can be started in a docker container usinged the latest released image:

docker run -v $(pwd)/config:/config -p 8080:8080 gcr.io/jenkinsxio/bucketrepo:0.1.12 -config-path=/config

Or it can be built and run with:

make build
bin/bucketrepo -config-path=config -log-level=debug

Maven Configuration

bucketrepocan be configured as a mirror by adding the following in ~/.m2/settings.xml file:

<settings>
<mirrors>
    <mirror>
      <id>bucketrepo</id>
      <name>bucketrepo- mirror</name>
      <url>http://localhost:8080/bucketrepo/</url>
      <mirrorOf>*</mirrorOf>
    </mirror>
  </mirrors>
</settings>

And as a repository by adding the following in the pom.xml file:

<repositories>
    <repository>
        <id>bucketrepo</id>
        <url>http://localhost:8080/bucketrepo/</url>
    </repository>
</repositories>

<distributionManagement>
    <snapshotRepository>
        <id>snapshots</id>
        <url>http://localhost:8080/bucketrepo/deploy/maven-snapshots/</url>
    </snapshotRepository>
    <repository>
        <id>releases</id>
        <url>http://localhost:8080/bucketrepo/deploy/maven-releases/</url>
    </repository>
</distributionManagement>

Acknowledgments

This project is originally based on nexus-minimal. Thank you atsman for creating that project.

License

MIT

bucketrepo's People

Contributors

Stargazers

Watchers

bucketrepo's Issues

bucketrepo does not work well with GCS

Everytime my bucketrepo pod restarts, I get an empty index.yaml

I am seeing this error in my logs:

{"timestamp":"2020-02-06T14:10:51Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"failed to update cache for /tmp/bucketrepo/charts/index.yaml: reading file from cloud storage: blob (key \"tmp/bucketrepo/charts/index.yaml\") (code=NotFound): storage: object doesn't exist","severity":"ERROR","context":{"reportLocation":{"filePath":"internal/controller.go","lineNumber":295,"functionName":"(*FileController).chartReindex"}}}

It appears that bucketrepo is using the "cache.BaseDir" prefix in the cloud path as well. Whereas while updating the chart it uses the path without "cache.BaseDir".

As a temporary fix, I created a gs://repos/tmp/bucketrepo/charts/index.yaml with the expected contents manually, which results in bucket repo happily working with it, and updating the gs://repos/charts/index.yaml.

I am taking a copy of gs://repos/charts/index.yaml and pasting it to gs://repos/tmp/bucketrepo/charts/index.yaml every few days for now.

Wrong error handling preventing clean downloads

After d0458f7 I get an error message like ERRO[0002] Error when serving the file from cache: reading file from cache: <nil> for every download when actually no real error happened. Since the error message is also injected at the end of the HTTP response e.g. Maven reports numerous checksum mismatch errors.

Why does it happen? The above mentioned commit d0458f7 changed the semantics with the error mapping: places that before returned nil in case of no error now always return some non-nil wrapped value.

This happens at least due to d0458f7#diff-f7ebf5bb6047d56d88d9243d98113c4fL134

Error when upgrading version stream

I just tried upgrading version stream from v1.0.399 to v1.0.453

In that set of changes, bucketrepo has gone from 0.1.35 to 0.1.38

The pipeline failed, as well as running jx boot when patching bucketrepo.

Error from pipeline:

Error from server: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.38\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"**\",\"jenkins.io/version\":\"61\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"**\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"env\":{},\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.38\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"},"labels":{"chart":"bucketrepo-0.1.38","jenkins.io/version":"61"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"bucketrepo"}],"containers":[{"env":{},"image":"gcr.io/jenkinsxio/bucketrepo:0.1.38","name":"bucketrepo"}]}}}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "jenkins-x-bucketrepo", Namespace: "**"
Object: &{map["apiVersion":"apps/v1" "metadata":map["annotations":map["deployment.kubernetes.io/revision":"3" "jenkins.io/chart":"env" "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.35\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"**\",\"jenkins.io/version\":\"60\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"**\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.35\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"] "name":"jenkins-x-bucketrepo" "namespace":"**" "resourceVersion":"31907518" "generation":'7' "creationTimestamp":"2020-03-11T03:25:01Z" "labels":map["chart":"bucketrepo-0.1.35" "draft":"draft-app" "jenkins.io/chart-release":"jenkins-x" "jenkins.io/namespace":"**" "jenkins.io/version":"60"] "selfLink":"/apis/apps/v1/namespaces/**/deployments/jenkins-x-bucketrepo" "uid":"6d57ec82-3ddf-4ddf-8352-23401d98411f"] "spec":map["replicas":'\x01' "selector":map["matchLabels":map["app":"jenkins-x-bucketrepo"]] "template":map["metadata":map["creationTimestamp":<nil> "labels":map["app":"jenkins-x-bucketrepo" "draft":"draft-app"] "annotations":map["checksum/config":"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c"]] "spec":map["terminationGracePeriodSeconds":'\n' "dnsPolicy":"ClusterFirst" "serviceAccountName":"jenkins-x-bucketrepo" "securityContext":map[] "restartPolicy":"Always" "containers":[map["readinessProbe":map["timeoutSeconds":'\x01' "periodSeconds":'\n' "successThreshold":'\x01' "failureThreshold":'\x03' "httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"]] "terminationMessagePolicy":"File" "imagePullPolicy":"IfNotPresent" "name":"bucketrepo" "command":["/bucketrepo"] "ports":[map["protocol":"TCP" "containerPort":'\u1f90']] "volumeMounts":[map["name":"config" "readOnly":%!q(bool=true) "mountPath":"/config"]] "livenessProbe":map["httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"] "initialDelaySeconds":'<' "timeoutSeconds":'\x01' "periodSeconds":'\n' "successThreshold":'\x01' "failureThreshold":'\x03'] "terminationMessagePath":"/dev/termination-log" "image":"gcr.io/jenkinsxio/bucketrepo:0.1.35" "args":["-config-path=/config" "-log-level=info"] "resources":map["limits":map["cpu":"100m" "memory":"256Mi"] "requests":map["cpu":"80m" "memory":"128Mi"]]]] "serviceAccount":"jenkins-x-bucketrepo" "schedulerName":"default-scheduler" "volumes":[map["name":"config" "secret":map["secretName":"bucketrepo-config" "defaultMode":'\u01a4']]]]] "strategy":map["type":"RollingUpdate" "rollingUpdate":map["maxUnavailable":"25%" "maxSurge":"25%"]] "revisionHistoryLimit":'\n' "progressDeadlineSeconds":'\u0258'] "status":map["observedGeneration":'7' "replicas":'\x01' "updatedReplicas":'\x01' "readyReplicas":'\x01' "availableReplicas":'\x01' "conditions":[map["lastUpdateTime"
:"2020-03-19T18:17:53Z" "lastTransitionTime":"2020-03-11T03:25:01Z" "reason":"NewReplicaSetAvailable" "message":"ReplicaSet \"jenkins-x-bucketrepo-98667859f\" has successfully progressed." "type":"Progressing" "status":"True"] map["lastTransitionTime":"2020-05-12T15:16:14Z" "reason":"MinimumReplicasAvailable" "message":"Deployment has minimum availability." "type":"Available" "status":"True" "lastUpdateTime":"2020-05-12T15:16:14Z"]]] "kind":"Deployment"]}
for: "/tmp/helm-template-workdir-790200993/jenkins-x/output/namespaces/**/env/charts/bucketrepo/templates/part0-deployment.yaml": cannot restore slice from map'

Error from jx boot:

Error from server: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.38\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"jx\",\"jenkins.io/version\":\"1\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"jx\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"env\":{},\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.38\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"},"labels":{"chart":"bucketrepo-0.1.38","jenkins.io/version":"1"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"bucketrepo"}],"containers":[{"env":{},"image":"gcr.io/jenkinsxio/bucketrepo:0.1.38","name":"bucketrepo"}]}}}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "jenkins-x-bucketrepo", Namespace: "jx"
Object: &{map["apiVersion":"apps/v1" "kind":"Deployment" "metadata":map["annotations":map["deployment.kubernetes.io/revision":"3" "jenkins.io/chart":"env" "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"jenkins.io/chart\":\"env\"},\"labels\":{\"chart\":\"bucketrepo-0.1.35\",\"draft\":\"draft-app\",\"jenkins.io/chart-release\":\"jenkins-x\",\"jenkins.io/namespace\":\"jx\",\"jenkins.io/version\":\"60\"},\"name\":\"jenkins-x-bucketrepo\",\"namespace\":\"jx\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"jenkins-x-bucketrepo\"}},\"template\":{\"metadata\":{\"annotations\":{\"checksum/config\":\"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c\"},\"labels\":{\"app\":\"jenkins-x-bucketrepo\",\"draft\":\"draft-app\"}},\"spec\":{\"containers\":[{\"args\":[\"-config-path=/config\",\"-log-level=info\"],\"command\":[\"/bucketrepo\"],\"image\":\"gcr.io/jenkinsxio/bucketrepo:0.1.35\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"name\":\"bucketrepo\",\"ports\":[{\"containerPort\":8080}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080},\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":1},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"256Mi\"},\"requests\":{\"cpu\":\"80m\",\"memory\":\"128Mi\"}},\"volumeMounts\":[{\"mountPath\":\"/config\",\"name\":\"config\",\"readOnly\":true}]}],\"serviceAccountName\":\"jenkins-x-bucketrepo\",\"terminationGracePeriodSeconds\":10,\"volumes\":[{\"name\":\"config\",\"secret\":{\"secretName\":\"bucketrepo-config\"}}]}}}}\n"] "creationTimestamp":"2020-03-11T03:25:01Z" "generation":'7' "labels":map["chart":"bucketrepo-0.1.35" "draft":"draft-app" "jenkins.io/chart-release":"jenkins-x" "jenkins.io/namespace":"jx" "jenkins.io/version":"60"] "name":"jenkins-x-bucketrepo" "namespace":"jx" "resourceVersion":"31924380" "selfLink":"/apis/apps/v1/namespaces/jx/deployments/jenkins-x-bucketrepo" "uid":"6d57ec82-3ddf-4ddf-8352-23401d98411f"] "spec":map["progressDeadlineSeconds":'\u0258' "replicas":'\x01' "revisionHistoryLimit":'\n' "selector":map["matchLabels":map["app":"jenkins-x-bucketrepo"]] "strategy":map["rollingUpdate":map["maxSurge":"25%" "maxUnavailable":"25%"] "type":"RollingUpdate"] "template":map["metadata":map["annotations":map["checksum/config":"74d63f02b8b28520c65c19769b57227e01f1bd9dae4f950c1f93ea4e92787c2c"] "creationTimestamp":<nil> "labels":map["app":"jenkins-x-bucketrepo" "draft":"draft-app"]] "spec":map["containers":[map["args":["-config-path=/config" "-log-level=info"] "command":["/bucketrepo"] "image":"gcr.io/jenkinsxio/bucketrepo:0.1.35" "imagePullPolicy":"IfNotPresent" "livenessProbe":map["failureThreshold":'\x03' "httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"] "initialDelaySeconds":'<' "periodSeconds":'\n' "successThreshold":'\x01' "timeoutSeconds":'\x01'] "name":"bucketrepo" "ports":[map["containerPort":'\u1f90' "protocol":"TCP"]] "readinessProbe":map["failureThreshold":'\x03' "httpGet":map["path":"/healthz" "port":'\u1f90' "scheme":"HTTP"] "periodSeconds":'\n' "successThreshold":'\x01' "timeoutSeconds":'\x01'] "resources":map["limits":map["cpu":"100m" "memory":"256Mi"] "requests":map["cpu":"80m" "memory":"128Mi"]] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File" "volumeMounts":[map["mountPath":"/config" "name":"config" "readOnly":%!q(bool=true)]]]] "dnsPolicy":"ClusterFirst" "restartPolicy":"Always" "schedulerName":"default-scheduler" "securityContext":map[] "serviceAccount":"jenkins-x-bucketrepo" "serviceAccountName":"jenkins-x-bucketrepo" "terminationGracePeriodSeconds":'\n' "volumes":[map["name":"config" "secret":map["defaultMode":'\u01a4' "secretName":"bucketrepo-config"]]]]]] "status":map["availableReplicas":'\x01' "conditions":[map["lastTransitionTime":"2020-03-11T03:25:01Z" "lastUpdateTime":"2020-03-19T18:17:53Z" "message":"ReplicaSet \"jenkins-x-bucketrepo-98667859f\" has successfully progressed." "reason":"NewReplicaSetAvailable" "status":"True" "type":"Progressing"] map["lastTransitionTime":"2020-05-12T15:52:20Z" "lastUpdateTime":"2020-05-12T15:52:20Z" "message":"Deployment has minimum availability." "reason":"MinimumReplicasAvailable" "status":"True" "type":"Available"]] "observedGeneration":'7' "readyReplicas":'\x01' "replicas":'\x01' "updatedReplicas":'\x01']]}
for: "/var/folders/zc/g4gbg59j2cg_fq6vf75mbkyc0000gn/T/helm-template-workdir-569312494/jenkins-x/output/namespaces/jx/env/charts/bucketrepo/templates/part0-deployment.yaml": cannot restore slice from map'
error: failed to interpret pipeline file jenkins-x.yml: failed to run '/bin/sh -c jx step helm apply --boot --remote --name jenkins-x --provider-values-dir ../kubeProviders' command in directory 'env', output: ''

Implement latest api call

In many cases, especially with helm scenario in JX preview environments, we need to update a depedency in requirements file with the latest chart version of an app since it does not implement it.

with chart museum we used to invoke:
curl http://jenkins-x-chartmuseum:8080/api/charts/<ChartName>/latest --silent | jq -r .version

its true that helm search repo/chart -ojson would do kind of the same but this acts on the client side i.e. helm client interaction with the cached index file. in our case its not enough because latest JX builders include helm 2.14 which does not have -o/--output flag implemented.

we need the same with bucketrepo so we can use it.

Local Cache in Container Never Synced with S3 Bucket

I've deployed bucketrepo alongside an S3 Bucket, but no what config settings I try, the updateCloudStorage function never gets called and my bucket remains empty forever.

Config:

http:
  addr: ':8080'
  username: ''
  password: ''
storage:
  enabled: true
  bucket_url: 's3://my-cache-bucket'
cache:
  base_dir: '/tmp/bucketrepo'
repositories:
- url: https://myorg.jfrog.io/artifactory/maven
  header:
    Authorization: 'Bearer <token>'

When I request an artifact I get the following debug logs on the first try:

"GetFile, filename: /com/myorg/package1/1/package1-1.pom"
"Read file from cache: /com/myorg/package1/1/package1-1.pom\n"
"Read file from repository: /com/myorg/package1/1/package1-1.pom"
"Trying to download from repository: https://myorg.jfrog.io/artifactory/maven"
"Write file to cache: /com/myorg/package1/1.187/package1-1.pom\n"
"Read file from cache: /com/myorg/package1/1/package1-1.pom\n"

Subsequent requests generate the following:

"GetFile, filename: /com/myorg/package1/1/package1-1.pom"
"Read file from cache: /com/myorg/package1/1/package1-1.pom\n"

There's no mention of the debug line "Write file to cloud storage: <file>" which would indicate that it's never trying to take any of the files it pulled from the repo's to the cloud-cache.

Am I misunderstanding something?

support s3 bucket encrypted with kms

Relevant google/go-cloud#2802

Newest release errors with previous cache dir settings

The newest release is not backwards compatible.

Setting the following settings was required for bucketrepo to not delete all of your charts each time it started:

config:
  cache:
    dir: ./

This now results in the following error:

➜  ~ k logs jenkins-x-bucketrepo-7bb5944448-bzg4n
{"timestamp":"2020-10-06T14:51:14Z","message":"Configuration: http:\n  address: :8080\n  https: false\n  certificate: \"\"\n  key: \"\"\n  username: xxx\n  password: xxx\n  chartpath: charts\nstorage:\n  enabled: true\n  bucketurl: s3://repository-20200702162217040100000006\n  timeout: 5m0s\n  prefix: \"\"\ncache:\n  basedir: ./\nrepositories:\n- url: https://repo.maven.org/maven2\n  timeout: 1m0s\n  header: {}\n- url: https://repo1.maven.org/maven2\n  timeout: 1m0s\n  header: {}\n- url: https://services.gradle.org/distributions/\n  timeout: 1m0s\n  header: {}\n- url: https://plugins.gradle.org/m2/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.jenkins-ci.org/public/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.jenkins-ci.org/releases/\n  timeout: 1m0s\n  header: {}\n- url: https://jitpack.io/\n  timeout: 1m0s\n  header: {}\n- url: https://registry.npmjs.org/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.spring.io/milestone/\n  timeout: 1m0s\n  header: {}\n- url: https://repo.spring.io/release/\n  timeout: 1m0s\n  header: {}\n- url: http://uk.maven.org/maven2/\n  timeout: 1m0s\n  header: {}\n","severity":"INFO","context":{}}
{"timestamp":"2020-10-06T14:51:14Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"failed to initialise controller: failed to create charts dir charts: mkdir charts: permission denied","severity":"CRITICAL","context":{"reportLocation":{"filePath":"internal/main.go","lineNumber":39,"functionName":"main"}}}

use a version of go-cloud that supports AWS IRSA

Currently bucketrepo does not seem to work with IRSA for AWS s3

dockerhub image is not accessible

Trying to use the suggested command

docker run -v $(pwd)/config:/config -p 8080:8080 jenkinsxio/bucketrepo -config-path=/config

fails with

docker: Error response from daemon: pull access denied for jenkinsxio/bucketrepo, repository does not exist or may require 'docker login'.

Latest release (0.1.41) doesn't work

Error: failed to start container "bucketrepo": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/bucketrepo\": stat /bucketrepo: no such file or directory": unknown

From here:
https://github.com/jenkins-x/bucketrepo/blob/master/charts/bucketrepo/templates/deployment.yaml#L28

which makes sense cause it was removed here:
a2fb727

Error when downloading the file: writing file to cache: mkdir : no such file or directory

with no tekton pipeline changes, my release started failing today with

about to run: helm repo add 0 https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts in dir charts/kafka-cluster
Error: looks like "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts" is not a valid chart repository or cannot be reached: Get "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts/index.yaml": dial tcp 10.141.120.2:443: connect: connection timed out
error: failed to add repository: failed to run 'helm repo add 0 https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts' command in directory 'charts/kafka-cluster', output: 'Error: looks like "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts" is not a valid chart repository or cannot be reached: Get "https://bucketrepo-jx.preprod.myorg.com/bucketrepo/charts/index.yaml": dial tcp 10.143.120.2:443: connect: connection timed out'
�[31m
Pipeline failed on stage 'from-build-pack' : container 'step-build-helm-build'. The execution of the pipeline has stopped.�[0m

the linked url and the bucketrepo logs display this error

  Error when downloading the file: writing file to cache: mkdir : no such file or directory

bucketrepo-bucketrepo-7d89dccf64-2ttjx bucketrepo {"timestamp":"2022-08-04T20:04:28Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"Error when downloading the file: writing file to cache: mkdir : no such file or directory","severity":"ERROR","context":{"reportLocation":{"filePath":"internal/controller.go","lineNumber":98,"functionName":"(*FileController).GetFile"}}}

environment variables in config.yaml

I have BUCKET_HOST, BUCKET_NAME, BUCKET_PORT and BUCKET_REGION environment variables set.
Would it be possible to use then in the config.yaml in .storage.bucket_url. For example like this "s3://${BUCKET_NAME}?endpoint=${BUCKET_HOST}:${BUCKET_PORT}&region=${BUCKET_REGION}&s3ForcePathStyle=true"

These variables are created automatically by my "ObjectBucketClaim" and bucket name is a unique automatic generated id, so it's hard to know before you deploy it.

Instructions

Hi there, app sounds promising, but having trouble running this on docker.

Goal is to run this with my teamcity/maven infra and artifacts should end up in the bucket.

I specifically grabbed "gcr.io/jenkinsxio/bucketrepo:0.1.38", assigned a persistent volume "bucketrepo_config -> /config"

Added the config path parameter as "/config". On first launch there is config.yaml in the root. Just in case there's confusion about what "/" means, i copied it to /config also using the console shell. Restarted.

All i see in the logs is:

{"timestamp":"2020-05-11T16:28:28Z","message":"Reading config file","severity":"WARNING","context":{"data":{"error":{}}}}

Bucketrepo does not support apiVersion: v2 of helm charts

in v2, requirement are put directly in Chart.yaml (requirements.yaml is still supported) maybe we can add support for this.

All charts wiped out each release, PVC support needed

Each time bucketrepo pod recreates it wiped out all the charts and that break jx-git-operator functionality

KInldy provide more examples for self-hosted cluster

Great work but need working examples configurations for self-hosted K#S clusters:3

In case we are on k3s, how do we configure to use a storage bucket from GCS or S3? I see references to jx-requirements.yaml's storage section, but how does auth work in that case without the cluster itself running on GKE?
How do we use Minio? Lets say we configured minio backed by GCS, how does one authenticate with Minio?

Bucketrepo with S3 gives error on cache operations

bucket repo is set with s3 (via JX).
charts are stored properly on the bucket but regardless the bucketrepo has errors on cache like the following:
{"timestamp":"2020-04-10T16:53:12Z","serviceContext":{"service":"bucketrepo","version":"1.0"},"message":"failed to update cache for /tmp/bucketrepo/charts/index.yaml: reading file from cloud storage: blob (key \"tmp/bucketrepo/charts/index.yaml\") (code=NotFound): NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 7FB39D20F87ACD3B, host id: n5DXh6s30meT+d4lOiCAGu9mIpzK+BxxZJJbZVVDKSCA7sg0NxeU9W63mf0fexc9R3je5mQPxNA=","severity":"ERROR","context":{"reportLocation":{"filePath":"internal/controller.go","lineNumber":295,"functionName":"(*FileController).chartReindex"}}}

it seems that bucket repo treats /tmp/bucketrepo (default cache dir) as cloud storage (s3) and not local dir.

Bucketrepo resets index.yaml content on s3 on startup

Interacting with bucketrepo for charts works. charts are being pushed, index.yaml is being updated (you can see both via helm search and via s3 file content).
if deleting the pod, after the new pod coming up the index.yaml is being reset with entries: {} field being empty - also helm search return no results.

Expected behaviour: bucketrepo picks up the index.yaml from s3 and continuous from the same state.

Does bucketrepo support tar archives ?

I want to host tar and zip archives where the backend storage is AWS S3. I tried using nexus but it consumes a lot of resources.
As of now I have no plans of hosting all types of archives. Only looking at simple tar.gz and zip archives.
Does bucketrepo support them and can the repo be private so that only people with right credentials can access them ?