[fetch] support custom ca cert for image pulling about kapp-controller HOT 34 CLOSED

ca cert functionality is available in https://github.com/vmware-tanzu/carvel-kapp-controller/releases/tag/v0.14.0, controller wide.

from kapp-controller.

from @jessehu :

1. When executing 'imgpkg' on a local machine, we can add the self-signed CA certificate on the local machine to fix it.
2. When executing 'imgpkg' in a container (e.g. kapp-controller), does it work? If not, we might need to add an option in App CR (around fetch.image.url) for specifying CA cert.

This is simliar to #42

from kapp-controller.

Even i am facing this issue from private registry
Fetching registry image: Imgpkg: exit status 1 (stderr: Error: Collecting images: Working with registry.XXXXX.com/library/tkg-extensions-templates:v1.2.0_vmware.1: Get https://registry.XXXXX.com/v2/: x509: certificate signed by unknown authority

  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
  fetch:
  - image:
      url: registry.XXXXX.com/library/tkg-extensions-templates:v1.2.0_vmware.1
  serviceAccountName: prometheus-extension-sa
  syncPeriod: 5m
  template:
  - ytt:
      ignoreUnknownComments: true
      inline:
        pathsFrom:
        - secretRef:
            name: prometheus-data-values
      paths:
      - tkg-extensions/common
      - tkg-extensions/monitoring/prometheus

How do i add certificate or way to skip it

from kapp-controller.

@arfiqbal Thanks for pointing out this issue. As documented above, kapp-controller does not support custom ca certs, but the underlying tool (imgpkg) does support a --registry-ca-cert-path.

Unfortunately, we need a way to specify options for imgpkg pull to avoid scenarios like this. So I am not sure if there is a workaround with kapp-controller at this moment.

from kapp-controller.

@arfiqbal

How do i add certificate or way to skip it

you can do that today by either builder a new image that inherits from kapp-controller and adding your own CA bundle or figuring out a way to inject CA bundle at runtime before through init container.

this issue will track work for making this feature a first class feature and will be available in one of the future releases.

from kapp-controller.

@cppforlife @danielhelfand
Let me know the path where we can add the custom CA ? so that i'll try to mount the ca cert using configmap

from kapp-controller.

@arfiqbal assuming you have put the CA cert file of the private registry under /etc/ssl/certs/harbor-ca.crt (file mode 644) on your k8s worker nodes, you can make the following change to the kapp-controller deployment, then imgpkg inside the kapp-controller will be able to pull images from registry.XXXXX.com.

      containers:
      - command:
        - /kapp-controller
        env:
        - name: KAPPCTRL_MEM_TMP_DIR
          value: /etc/kappctrl-mem-tmp
        image: registry.yourdomain.com/kapp-controller:v0.9.0
        imagePullPolicy: IfNotPresent
        name: kapp-controller

        volumeMounts:
        - mountPath: /etc/kappctrl-mem-tmp
          name: template-fs
        - mountPath: /etc/ssl/certs/harbor-ca.crt
          name: certs
          subPath: harbor-ca.crt

      volumes:
      - emptyDir:
          medium: Memory
        name: template-fs
      - hostPath:
          path: /etc/ssl/certs
          type: ""
        name: certs

from kapp-controller.

@jessehu we tried the exactly the same thing but still no luck

    spec:
      containers:
      - command:
        - /kapp-controller
        env:
        - name: KAPPCTRL_MEM_TMP_DIR
          value: /etc/kappctrl-mem-tmp
        image: registry.xxxxx.com/kapp-controller:v0.9.0_vmware.1
        imagePullPolicy: IfNotPresent
        name: kapp-controller
        resources:
          requests:
            cpu: 120m
            memory: 100Mi
        securityContext:
          runAsGroup: 2000
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/kappctrl-mem-tmp
          name: template-fs
        - mountPath: /etc/ssl/certs/harbor-ca.crt
          name: cert
          subPath: harbor-ca.crt
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 3000
      serviceAccount: kapp-controller-sa
      serviceAccountName: kapp-controller-sa
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          medium: Memory
        name: template-fs
      - hostPath:
          path: /etc/ssl/certs
          type: ""
        name: cert

arif@de1gks1tkg:~$ oc get apps.kappctrl.k14s.io
NAME         DESCRIPTION   SINCE-DEPLOY   AGE
prometheus   Reconcile failed: Fetching (0): Fetching registry image: Imgpkg: exit status 1 (stderr: Error: Collecting images: Working with registry.xxxxx.com/library/tkg-extensions-templates:v1.2.0_vmware.1: Get https://registry.xxxxx.com/v2/: x509: certificate signed by unknown authority
)                          85s

from kapp-controller.

@arfiqbal /etc/ssl/certs/harbor-ca.crt must exists on every worker nodes, and the file mode must be worldwide readable. BTW I will try the Secret/ConfigMap approach later.

from kapp-controller.

@jessehu yup, I have copied the harbor-ca.crt in all the worker node and i also tried with configmap but its still not working

Thanks for your quick response :)

from kapp-controller.

Also make sure your harbor-ca.crt is correct. Running the following command on worker nodes should succeed if /etc/ssl/certs/harbor-ca.crt contains the correct CA of your Harbor Registry.

imgpkg pull --image registry.xxxxx.com/library/tkg-extensions-templates:v1.2.0_vmware.1 --output /tmp/1

from kapp-controller.

@jessehu
i tried from one of my worker node and its working perfectly

root@gks-rat-stg-5-md-0-6f77987df7-zvhd6 [ ~ ]# imgpkg pull --image registry.xxxxx.com/library/tkg-extensions-templates:v1.2.0_vmware.1 --output /tmp/1
Pulling image 'registry.xxxxx.com/library/tkg-extensions-templates@sha256:b9825280fd44c012b97b92ebf24b3b7fd2af4bbbe11c3901d350432ae828451e'
Extracting layer 'sha256:46c40ce88d5394ceb4350e8c2a3b8f9f160cd9b3bdea5d6317f2a14c244ed1e6' (1/1)

Succeeded

from kapp-controller.

How about the file mode 'ls -l /etc/ssl/certs/harbor-ca.crt' ? If it's worldwide readable, then I have no other idea.

from kapp-controller.

Here it is

root@gks-rat-stg-5-md-0-6f77987df7-zvhd6 [ ~ ]# ls -l /etc/ssl/certs/harbor-ca.crt
-rw-r--r-- 1 root root 2508 Dec  3 08:35 /etc/ssl/certs/harbor-ca.crt

from kapp-controller.

@jessehu
Now i am getting different issue

arif@de1gks1tkg$ oc get apps.kappctrl.k14s.io
NAME         DESCRIPTION                                       SINCE-DEPLOY   AGE
prometheus   Reconcile failed: Templating dir: exit status 1                  4m16s

from kapp-controller.

The certificate issue in kapp-controller is resolved. You can email me at [email protected] to debug this new issue.

from kapp-controller.

@jessehu sure mate :)

Can you let me know what are the information your are looking for ?

from kapp-controller.

HI @jessehu The error in detail is bellow . Some templating which is not able to predict the platform infra . btw we are using tkg on vpshere.

  inspect:
    error: 'Inspecting: exit status 1'
    exitCode: 1
    stderr: 'kapp: Error: App ''prometheus-ctrl'' (namespace: tanzu-system-monitoring)
      does not exist: configmaps "prometheus-ctrl" not found'
    stdout: Target cluster 'https://100.64.0.1:443'
    updatedAt: "2020-12-03T10:06:14Z"
  observedGeneration: 1
  template:
    error: 'Templating dir: exit status 1'
    exitCode: 1
    stderr: "ytt: Error: \n- cannot load values.star: Evaluating starlark template:
      \n    in <toplevel>\n      00-server-namespace.yaml:1 | #@ load(\"values.star\",
      \"values\")\n\n    reason:\n     - assert.fail: fail: infrastructure provider
      should be either aws or vsphere or azure\n         in validate_infrastructure_provider\n
      \          9 |   data.values.infrastructure_provider in (\"aws\", \"vsphere\",
      \"azure\") or assert.fail(\"infrastructure provider should be either aws or
      vsphere or azure\")\n         in validate_pushgateway\n           195 |     validate_func()\n
      \        in <toplevel>\n           255 | validate_pushgateway()\n"
    updatedAt: "2020-12-03T10:06:14Z"

from kapp-controller.

Thanks @sharmavijay86. You didn't set infrastructure_provider: vsphere in the values.yaml. Please email me for further questions or find another place like Slack to discuss it, as this issue not related to kapp-controller.

from kapp-controller.

@jessehu Just mailed you with all the information... Please check

from kapp-controller.

Here is the ConfigMap solution for supporting custom ca cert for image pulling:

1. Add the CA cert of your registry server into a file custom-ca-certs.crt

2. Create a ConfigMap with this file:
   kubectl create configmap custom-ca-certs --from-file=custom-ca-certs.crt -n kapp-controller

3. Modify the kapp-controller.yaml as below to mount the cert file from the ConfigMap, then kubectl apply it.

containers:
- command:
  - /kapp-controller
  ... ...
  volumeMounts:
  - mountPath: /etc/kappctrl-mem-tmp
    name: template-fs
  - mountPath: /etc/ssl/certs/custom-ca-certs.crt
    name: custom-ca-certs
    readOnly: true
    subPath: custom-ca-certs.crt
... ...
volumes:
- emptyDir:
    medium: Memory
  name: template-fs
- configMap:
    name: custom-ca-certs
  name: custom-ca-certs

from kapp-controller.

@jessehu its working man

Thanks for your help :)

Appreciated 👍

from kapp-controller.

We have a discussion on how to fix this issue on carvel slack channel: https://kubernetes.slack.com/archives/CH8KCCKA5/p1607008555143000

from kapp-controller.

It would be good to consider the secret structure of cert-manager created certs ( data.ca.crt: cert | b64enc ) in any examples as a way to store ca certs for kapp-controller as to be more widely applicable.

from kapp-controller.

Hi @ewrenn8 is there an ETA for this feature?

from kapp-controller.

Hey @jessehu, it was added in this commit dec56a3. We will cut a release with this included today.

from kapp-controller.

Thanks a lot @ewrenn8. The patch looks great! Will the new kapp-controller release ship with the coming imgpkg 0.3.0?

from kapp-controller.

Will the new kapp-controller release ship with the coming imgpkg 0.3.0?

nope, we havent cut imgpkg 0.3.0 so it wont be included. subsequent releases im sure will pick it up when it's available.

from kapp-controller.

If this is working, it would be nice if someone at vmware would update the tkg-extensions-manifests tarball.

from kapp-controller.

Here is the ConfigMap solution for supporting custom ca cert for image pulling:

1. Add the CA cert of your registry server into a file custom-ca-certs.crt

2. Create a ConfigMap with this file:
   kubectl create configmap custom-ca-certs --from-file=custom-ca-certs.crt -n kapp-controller

3. Modify the kapp-controller.yaml as below to mount the cert file from the ConfigMap, then kubectl apply it.

containers:
- command:
  - /kapp-controller
  ... ...
  volumeMounts:
  - mountPath: /etc/kappctrl-mem-tmp
    name: template-fs
  - mountPath: /etc/ssl/certs/custom-ca-certs.crt
    name: custom-ca-certs
    readOnly: true
    subPath: custom-ca-certs.crt
... ...
volumes:
- emptyDir:
    medium: Memory
  name: template-fs
- configMap:
    name: custom-ca-certs
  name: custom-ca-certs

The error"certificate signed by unknown authority" has been fixed after following your guidence, but new error occurs:"Stderr: Error: Syncing directory '0': Syncing directory '.' with image contents: Imgpkg: exit status 1 (stderr: Error: Collecting images: Working with 192.168.104.2/tkgs-cluster-ns/tkg-extensions-templates:v1.3.1_vmware.1: GET https://192.168.104.2/v2/tkgs-cluster-ns/tkg-extensions-templates/manifests/v1.3.1_vmware.1: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:tkgs-cluster-ns/tkg-extensions-templates Type:repository]]"

I using embed harbor in vsphere with tanzu, How can i pass "imagePullSecrets" to kapp-controller pod to authenticate with harbor?

from kapp-controller.

@boyfzb2018 could you make the project tkgs-cluster-ns as a public project?

from kapp-controller.

@boyfzb2018 could you make the project tkgs-cluster-ns as a public project?

It is diffcult because the embedded habor has no default admin account provided to user, I will try to recover the admin account , then create public project in harbor; many thanks

from kapp-controller.

BTW in TKG 1.3.1, the kapp-controller supports configuring custom CA certs via a configmap:

$ k get configmap -n kapp-controller     kapp-controller-config -o yaml
apiVersion: v1
data:
  caCerts: ""
  httpProxy: ""
  httpsProxy: ""
  noProxy: ""
kind: ConfigMap

from kapp-controller.

BTW in TKG 1.3.1, the kapp-controller supports configuring custom CA certs via a configmap:

$ k get configmap -n kapp-controller     kapp-controller-config -o yaml
apiVersion: v1
data:
  caCerts: ""
  httpProxy: ""
  httpsProxy: ""
  noProxy: ""
kind: ConfigMap

Thanks, It works, the issue has fixed

from kapp-controller.