Pod Status

root@example-large0:~# kubectl get pod
NAME                                                      READY   STATUS    RESTARTS   AGE
all-in-one-database-postgresql-0                          1/1     Running   0          13h
all-in-one-minio-56d65664f8-znx6b                         1/1     Running   0          13h
all-in-one-redis-master-0                                 1/1     Running   0          13h
all-in-one-redis-slave-0                                  1/1     Running   0          13h
all-in-one-redis-slave-1                                  1/1     Running   0          13h
harborcluster-sample-harbor-core-669d75dd4-qh8fs          1/1     Running   0          2m3s
harborcluster-sample-harbor-jobservice-564cffbcb5-95dc9   1/1     Running   0          2m3s
harborcluster-sample-harbor-portal-bdd9d9b85-gjzjs        1/1     Running   0          2m3s
harborcluster-sample-harbor-registry-9f9f6b546-j7mf9      2/2     Running   0          2m3s

Core Component Health

kubectl exec -it harborcluster-sample-harbor-core-669d75dd4-qh8fs bash

curl localhost:8080/api/health

{"status":"healthy","components":[{"name":"redis","status":"healthy"},{"name":"registry","status":"healthy"},{"name":"jobservice","status":"healthy"},{"name":"core","status":"healthy"},{"name":"registryctl","status":"healthy"},{"name":"portal","status":"healthy"},{"name":"database","status":"healthy"}]}

Harbor Status

status:
  conditions:
  - lastTransitionTime: "2020-09-15T03:27:24Z"
    lastUpdateTime: "2020-09-15T03:27:24Z"
    status: "True"
    type: Applied
  - lastTransitionTime: "2020-09-15T11:09:17Z"
    lastUpdateTime: "2020-09-15T21:09:17Z"
    message: 'cannot get health response: unknown (get service harborcluster-sample-harbor-core)'
    reason: unknown (get service harborcluster-sample-harbor-core)
    status: "False"
    type: Ready
  observedGeneration: 1

The installation guide says the minimal version of cert manager is 1.11: https://github.com/goharbor/harbor-operator/blob/master/docs/installation/installation.md#minimal
But the highest version of cert manager is 1.0.1 for now

At present, different components using Redis will have different secrets that containing the connection info of different databases or Redis instances.

As allowing the user to configure the unique HA database service or sentinel Redis service and keep the current multiple secrets approach, we can introduce a global secret reference:

• If the component level secret is specified, then use component level secret reference
• Otherwise, use the global one as the default secret

// +kubebuilder:validation:Required
	// +kubebuilder:validation:Pattern="https?://.*"
	// The absolute url for .tgzs in index.yaml
	URL string `json:"url"`

Previous it is a config option with boolean value: AbsoluteURL: true // or false

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:
I follow installation doc to install harbor-operator. When I run make install-dependencies, I have a error

➜  harbor-operator git:(master) ✗ make install-dependencies
which: no controller-gen in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
which: no markdownlint in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
which: no golangci-lint in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
which: no pkger in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
which: no kubebuilder in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
which: no kustomize in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
which: no gomplate in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
which: no goreleaser in (/usr/local/bin:/usr/bin:/home/centos/bin:/usr/local/sbin:/usr/sbin:/home/centos/gocode/bin)
/usr/local/bin/helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories
/usr/local/bin/helm get notes core-database \
        || /usr/local/bin/helm install core-database bitnami/postgresql
Error: getting deployed release "core-database": release: "core-database" not found
Error: This command needs 1 argument: chart name
make: *** [install-dependencies] Error 1

Steps to reproduce the problem:

make install-dependencies

Versions:
Please specify the versions of following systems.

• harbor operator version: [0.5.0]
• harbor version: [x.y.z]
• kubernetes version: [1.18.2]
• Any additional relevant versions such as CertManager

TrivyStorage *HarborStorageTrivyStorageSpec json:"trivyStorage,omitempty"``

It is a temp storage for trivy to hold CVE DB and is more close to Trivy component itself. Maybe it's better and more clear to put it under the trivy component.

Cover:

• CODE (lint, vet, race )
• UT
• Verify if it can work well in cluster (kind/minikube etc.）

Harbor resource status should be based on dependencies status.
Suggestion: Ensure all deployments are ready to set Harbor to ready.

Reminder: for now, to check the Harbor readiness, the operator calls the /api/health (through the apiserver).
The new way should reduce resource consumption and will be finer status message.

I understand the current operator spec is referring to the harbor chart V1.4 templates.

My concern is anyway those two specs will be developed separately in the future. Some configuration may be introduced into the operator spec but not put into the chart template. And If the chart version is upgrading or templates are changed for fixing some bugs in the chart and maybe no influences to the operator, will the operator define a new spec object HarborHelm1_5_0Spec then?

• config/Manager

and no mutating-webhook but patch file defined 'webhookcainjection_patch.yaml'

Miss create Issuer for registry cert in Harbor CR namespace.

Need to use YAML to create Issuer.

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: pg
spec:
  selfSigned: {}

Currently in installation guide, step make install-dependencies will install 3 redis clusters for different components, actually we don't need so many redis instance, and in customer environment they may also only buy one redis cluster with different Database number, so we should also support such case.

redis-cli -n 1

install harbor with harbor-oprator in KIND k8s cluster, with 4G memory, and set nginx proxy-body-size=0
When push docker image golang, which is about 800MB, we can see the docker client failed retry for several times, and the registry pod restart several time.
it is ok that push image busybox.

Not sure if there is memory leak, or healthcheck timeout that leads to the restart

Split the Harbor CRD into multiple smaller one.
So every components can be reconciled independently.

Have a definition for following components

• ChartMuseum
• Clair
• Core
• JobService
• NotaryServer
• NotarySigner
• Portal
• Registry
• RegistryController

And an other definition Harbor which can handle all of them.

BTW with multiple controllers this would reconcile faster.

It would be a great addition to harbor operator to be able to have the operator create a repository and robot account based on custom resources

Currently I need to use the harbor API to create a repositories and robot accounts post install, store the robot token somewhere, create a dockerconfigjson secret for in cluster access to harbor.

All of these things could be automated within the operator allowing declarative deployment of these resources.

Add an extra description to enable the operator in local clusters like kind / minikube and try the sample.

non-load balancer scenarios

Error:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "mharbor.kb.io": the server could not find the requested resource
Makefile:117: recipe for target 'sample' failed
make: *** [sample] Error 1

Check endpoints of webhook SVC:

kubectl describe service -n harbor-operator-system harbor-operator-webhook-service
Name:              harbor-operator-webhook-service
Namespace:         harbor-operator-system
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"harbor-operator-webhook-service","namespace":"harbor-operator-sys...
Selector:          control-plane=controller-manager
Type:              ClusterIP
IP:                10.96.115.134
Port:              <unset>  443/TCP
TargetPort:        9443/TCP
Endpoints:         10.244.1.12:9443
Session Affinity:  None
Events:            <none>

It will cause kustomize build failed

Harbor resource definition requires a public exposed certificate (through ingresses) and an issuer to create internal certificates. This is unclear for new comers, these components should be documented.

Reproduce steps:

install harbor core components with make sample, harbor "sample" is deployed

install full harbor with make sample-harbor-full, no more components are deployed.

expectations: the optional components like notary, chartmuseum and trivy should be deployed.

What can we help you?

This may cause the login to fail.

- name: _REDIS_URL
          value: cluster-harborcluster-sample.pg.svc:6379,100,cpxgggod

In Harbor 2.0, we introduce a new default scanner Trivy (that is also an optional component).

The operator deploys many resources. Some are required for other.
Write a documentation about resources dependency.
The goal of this documentation is to have a better overview of the components dependency (core, registry, portal, ...) to prepare the 1.0 release with more Custom Resource Definitions.

Found issues in the cluster-operator: goharbor/harbor-cluster-operator#118

registry configuration references: https://docs.docker.com/registry/configuration/

similar configurations in helm chart: https://github.com/goharbor/harbor-helm/blob/fad6918c0fe3ca916d0948b98d433f6bffc96035/values.yaml#L243

When we enable notary (TLS must be enabled), shall we update the following customization files that are located at config/?

• harbor-full/harbor_notary_patch.yaml

apiVersion: goharbor.io/v1alpha2
kind: Harbor
metadata:
  name: sample
spec:
  notary:
    migrationEnabled: true
  expose:
    notary:
      ingress:
        host: notary.harbor.domain
     tls:  // add tls config and share the same root cert with core?
        certificateRef: sample-public-certificate

• harbor/https.yaml

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: sample-public-certificate
spec:
  isCA: false
  issuerRef:
    name: sample-public-certificate
  secretName: sample-public-certificate
  dnsNames:
  - core.harbor.domain
  - notary.harbor.domain # add notary domain here?

Some changes may need to be done for supporting Redis service with sentinel mode.

Get an overview of all components & Kubernetes resources dependency.

Please make a graph (maybe with plantuml?)

Example:

• Job service Deployment requires Core secret to be created

when I run
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.13.1
It return error
Error: This command needs 1 argument: chart name

I use helm v2.16.8.
The reason is that helm install only need one argument to specify chart name, any others are flags.
So I changed the command to
helm install --name cert-manager jetstack/cert-manager --namespace cert-manager --version v0.13.1
It works.

Error: container has runAsNonRoot and image has non-numeric user (nonroot), cannot verify user is non-root

Dockerfile use non numberic nonroot user nonroot

USER nonroot:nonroot

==>

USER 1000:1000

Add Helm chart template files in a chart/ folder. so it is easy to package and deploy the operator.

The yaml files under config/samples/ are out of date, they're not aligned with the latest harbor CR spec.

kcustomization.yaml

resources:
  - containerregistry_v1alpha1_harbor.yaml # wrong name
  - certificate.yaml
  - requirements.tmpl

goharbor_v1alpha_harbor.yaml

# ...
registryCtl: # wrong
      image: "goharbor/harbor-registryctl:v1.10.0"
    registry:
      storageSecret: registry-storage
      cacheSecret: registry-cache
      image: goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.10.0
# ...

notary:
      publicURL: 'https://{{ env.Getenv "NOTARY_DOMAIN" }}'
      notaryDBMigratorImage: jmonsinjon/notary-db-migrator:v0.6.1 # wrong
      server:
        databaseSecret: notary-server-database
        image: goharbor/notary-server-photon:v0.6.1-v1.10.0
      signer:
        databaseSecret: notary-signer-database
        image: goharbor/notary-signer-photon:v0.6.1-v1.10.0
# ...

Explain how to connect databases (with a single databases instances or with multiple instances) to Harbor components.

Please use a graphical representation.

Core         --> Postgresql
Core         --> Redis
Registry     --> Redis
Portal       --> x
JobService   --> Redis
RegistryCtl  --> x
NotaryServer --> Database
...

storage secret / configMap?

GithubTokenRef string json:"githubTokenRef,omitempty"``

run cmd make deploy IMG=goharbor/harbor-operator:v0.5.1
goharbor/harbor-operator:dev is pulled.

How about move sudo microk8s.kubectl cluster-info after microk8s.status

There will create a secret which contains an environment variable, REGISTRY_HTTP_SECRET.
But not mount the secret in deployment.

Cause:

the registry log:

{"go.version":"go1.12.12","harbor":"harborcluster-sample-harbor","instance.id":"5044caa1-a913-492e-94ce-044445eef03a","level":"warning","msg":"No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable.","operator":"harbor-operator","time":"2020-07-13T12:50:29.202771793Z","version":"v2.7.1.m"}

Expose more configuration options:

# log configurations
log:
  level: debug
  
# set proxy
proxy:
  http_proxy: 10.10.20.2
  https_proxy: 10.10.20.2
  no_proxy: 10.123.111.10
    components:
    - core
    - jobservice
    - clair

Use harbor version and a unified registry to define the images of Harbor components:

# source registry of images
imageSource:
  registry: harbor.com
  imagePullSecrets:
   - pSecret

k8s describe registry-pods

Events:
  Type     Reason       Age                    From                                               Message
  ----     ------       ----                   ----                                               -------
  Normal   Scheduled    <unknown>              default-scheduler                                  Successfully assigned harbor/sz-harbor-cluster-harbor-registry-665f8c845-27b7w to ip-10-0-1-218.us-east-2.compute.internal
  Warning  FailedMount  4m39s (x3 over 4m41s)  kubelet, ip-10-0-1-218.us-east-2.compute.internal  MountVolume.SetUp failed for volume "certificate" : secret "sz-harbor-cluster-harbor-certificate" not found

It may be because the certificate is not ready when starting mount operation as creations are handled by concurrent goroutines. And then the creating order cannot be guaranteed.

check pod log:

k8s logs pod/sz-harbor-cluster-harbor-registry-665f8c845-27b7w -n harbor -c registry

configuration error: error parsing /etc/registry/config.yaml: yaml: did not find expected node content

Usage: 
  registry serve <config> [flags]
Flags:
  -h, --help=false: help for serve


Additional help topics:

Clair updater interval is hardcoded to 0s and then the updater is disabled.

  updater:
    interval: 0s
    enabledupdaters:

Currently in installation guide, step make install-dependencies will install 3 databases for different components, actually we don't need so many database instance, and in customer environment they may also only buy one database with different schema, so we should also support such case.

When I push a 5.6MB size image. But failed.

The push refers to repository [test.harbor.local.com/library/socat]
ddc78141088d: Layer already exists
076f4a787bf9: Pushing [==================================================>]    5.6MB/5.6MB
error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.19.0</center>\r\n</body>\r\n</html>\r\n"

Add nginx.ingress.kubernetes.io/proxy-body-size: 1024m annotation to ingress may resolve this issue, if you use nginx ingress controller.
This annotation value should be configurable.

missing folder name installation

In harbor 2.0, we enable the TLS between harbor components

We need to verify the bitnami/redis chart does work well.