The release of trust-manager 0.4.0 hit several issues relating to buildx. For go, it seems incredibly heavyweight since rather than cross compiling, say, the arm64 version of trust-manager with the amd64 version of go, it instead runs the native arm64 binary in qemu, inside a container.

There are a lot of things that can go wrong there, and during the build for 0.4.0 we saw a deadlock - the cause of which was very difficult to pin down.

We use ko in other projects and should consider using that here too.

When filing #163, I somehow didn't realize that the generated CRDs were not up-to-date 🦆, and had to fix this in #164. This is unfortunate, and I think we should ensure that the project is consistent in CI.

With some guidance from a maintainer, I was linked to the CI configuration: https://github.com/jetstack/testing/tree/master/config/jobs/cert-manager/trust-manager

Definition of done: CI that fails if the following generated artifacts are not updated:

• CRDs
• Deepcopy functions
• Helm docs
• Boilerplate header in source files Update: Already done by https://github.com/cert-manager/boilersuite (lint)
• Other?

In my setup the Trust controller runs in the same namespace as the cert-manager installation ( ns: cert-manager ).

I have a certificate that is issued by an Issuer running in the monitoring namespace (ECK deployment). This CA I like to replicate with the Trust operator, so I made a Bundle in the monitoring namespace, expecting this namespace to be the trusted namespace.

After reading some of the issues, it seems like only the Cert-Manager namespace seems trusted, which seems strange to me. The whole point for me to use this project is to provide CA bundles to other namespaces to they can connect to secured microservices.

Is there a way to override the trusted namespace with config? I don't like that my teams need to deploy the Trust Operator in their namespaces, that seams strange.

I have made a clusterrole so the SA in the cert-manager namespace has read/list access to the secrets in the namespaces that have Issuers. This can be scoped to specific secrets so you don't leak.

Love to hear your opinions.

I want to install Trust Manager without cert manager. For that I removed the certificate.yaml file from helm chart. I also saved ca.crt, tls.crt, tls.key files in trust-manager-tls secret. This corresponds to the certificate I got signed from AWS Private CA.
I then tried to install trust manager via helm chart. Trust Manager got installed successfully.
Now I applied a Bundle to sync the ca.crt file to example config maps. This gave me the following error:

Internal error occurred: failed calling webhook "trust.cert-manager.io": failed to call webhook: Post "https://trust-manager.cert-manager.svc:443/validate?timeout=5s": x509: certificate signed by unknown authority

Can someone please help me here. I just want to use a certificate issued by AWS Private CA and sync is CA certificate to other configmaps. For this, I don't want to use cert manager.

A common pattern to build a locally trusted CA with cert-manager is to create a self-signed issuer, use it to issue a CA certificate, then use that certificate with a ca issuer to issue leaf certificates. A common request with this configuration is to be able to supply a trust bundle to clients so they can trust the CA, and support rotation of that CA.

When a certificate is issued by cert-manager, it will create a CertificateRequest, whose status is populated with the public certificate that is issued. By default, this resource is not automatically garbage collected unless the Certificate object is deleted. The certificate requests are linked to the certificate via owner references.

By adding support for certificate requests as trust-manager sources, an end user could easily build a trust bundle for every certificate issued by the self-signed issuer, automatically extending it as the root certificate is rotated. If a specific issued root certificate is considered compromised, the user could delete the specific CertificateRequest and it would be removed from the trust bundle.

Suggested API:

spec:
  sources:
  - certificateRequests:
      certificate: "my-self-signed-ca"

The trust-manager would use owner references to identify the matching certificate requests (similar to how cert-manager does it in the issuing flow) , and use the certificate field from the status of the request to populate the trust bundle.

Alternatively, we could extend cert-manager to provide a CertificateRequest template option to be able to label requests, and select them that way.

We could also include the ability to select a specific CertificateRequest, but I'm not sure why we would do that.

PR #37 introduced the ability for users to specify a subset of namespaces to which a target should be synced. We propose that the namespaceSelector becomes a required field, replacing the current behaviour of a target being synced everywhere by default.

This would be a breaking change. We agreed in the 2022-07-21 standup that breaking changes are something we're willing to accept for trust given it's pre-v1.0.

Justification

Imagine a multi tenanted cluster.

In namespace foo, FooCorp has their root certificate with subject CN=FooCorp. This is added to FooCorp's bundle.

In namespace bar, BarCorp is a different company and does totally different things.

By default, FooCorp's bundle will by synced to BarCorp's namespace. BarCorp employees will see a ConfigMap with FooCorp's root certificate in it. They won't be able to use the root to sign certificates, but they will be able to see that they're tenanted on the same cluster as FooCorp.

Likewise, if someone at BarCorp erroneously used the FooCorp bundle it would enable anyone with access to the FooCorp root to MitM any affected BarCorp service.

Syncing to all namespaces by default introduces an absolutely critical security requirement for some users; they must ensure they always set a namespaceSelector. Effectively they need to manage a denylist from a default position of "allow all".

Making namespaceSelector required would change this to an allowlist. It's a little more work for the average user but it will present a significant security win.

Cert-Manager and Trust-Manager both use ValidatingWebhookConfiguration as part of their helm deployments, however cert-manager has specifically got support for AWS EKS, where these webhooks require hostnetwork: true to be set. This is because the EKS control plane sits apart from the rest of the cluster. Cert-manager documents the config changes needed to work in EKS but currently the Trust-Manager helm chart neither supports these changes nor documents how it can done. Without these changes, it is currently not possible to have a stable deployment of trust-manager on AWS EKS.

if deploying right now, you would see errors like the following:

cannot patch "cert-bundle" with kind Bundle: Internal error occurred: failed calling webhook "trust.cert-manager.io": failed to call webhook: Post "https://trust-manager.default.svc:443/validate?timeout=5s": Address is not allowed

The helm chart's webhook should be changed to allow a hostNetwork flag to be added so that EKS can be supported, which should also bring the chart in line with the cert-manager helm chart..

If trying to set source.useDefaultCAs to false the admission webhook prevents creation of the bundle.

How to reproduce:

cat  <<EOF | kubectl apply -f -
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: my-org.com-ca
spec:
  sources:
  - useDefaultCAs: false

  #- secret:
  #    name: my-org.com-ca
  #    key: "tls.crt"

  target:
    configMap:
      key: "certs.pem"
    namespaceSelector:
      matchLabels: {}
EOF

Error output:
admission webhook "trust.cert-manager.io" denied the request: spec.sources.[0]: Forbidden: must define exactly one source type for each item but found 0 defined types

Workaround is trivial comment the useDefaultCAs entry, but still one would expect the false is supported value.

The Certificate resource created for webhook does not have a reference to Issuer and logs a warning

Warning  BadConfig          40s   cert-manager-certificaterequests-issuer-selfsigned  Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients

Followed the instruction on installing the Helm chart. No success:

$ helm repo add jetstack https://charts.jetstack.io --force-update
"jetstack" has been added to your repositories

$ helm upgrade -i -n cert-manager cert-manager-trust jetstack/cert-manager-trust --wait
Release "cert-manager-trust" does not exist. Installing it now.
Error: failed to download "jetstack/cert-manager-trust" (hint: running `helm repo update` may help)

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...
...Successfully got an update from the "jetstack" chart repository
...
Update Complete. Happy Helming!

$ helm search repo jetstack
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
jetstack/cert-manager                   v1.6.1          v1.6.1          A Helm chart for cert-manager
jetstack/cert-manager-approver-policy   v0.1.0          v0.1.0          A Helm chart for cert-manager-approver-policy
jetstack/cert-manager-csi-driver        v0.2.0          v0.2.0          A Helm chart for cert-manager-csi-driver
jetstack/cert-manager-csi-driver-spiffe v0.1.0          v0.1.0          A Helm chart for cert-manager-csi-driver-spiffe
jetstack/cert-manager-istio-csr         v0.3.1          v0.3.0          A Helm chart for istio-csr

Affinities and topologySpreadConstraints can wait for nodeSelector(enforced by validatingwebhooks) and tolerations(where all nodes run with taints) are more important to support in the helm chart

It is possible that the trust bundle contains multiple sources which point to the same CA. It would be nice if we can dedup in the final trust bundle

This will be a somewhat manual task since the current image will need to be deleted (which is fine as we've not released a version of trust-manager yet which uses the images).

This is blocked behind #93

Hello,

I wonder if it would be possible to add functionality to the helm chart so that:

1. provide a image pullsecret (you already allow to provide a custom repo in the values, but if a pullsecret is required there's no option).
2. install the CRD

Thanks for your time..

In #90 @inteon implemented some changes to help with cache correctness, since trust-manager has to work around some limitations currently present in controller-runtime (see kubernetes-sigs/controller-runtime#1962 which should help).

We believe that PR was an improvement, but there might be other issues that need addressing which Tim is looking into.

It seems like my PR just makes Secrets the same as the other resources, but there is a fix necessary to make everything work as intended (minimal caching etc.).
[and to avoid] duplicate caching (see this comment for more info: https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/builder/options.go#L103-L132)
My current hypothesis is that we cache every resource 2 times (one time metadata only and one time fully), I still have to check by running an actual example.

This is a tracking issue for that work, which is likely a sensible prerequisite for a 0.4.0 release.

It would be great if it was supported to have certificates injected into existing ConfigMaps if they are labelled, similar to the approach that ist available in OpenShift https://docs.openshift.com/container-platform/4.11/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki

Something like the snippet belo would inject the bundle to configmaps with the provided label.

target:
  configMapLabel:
    trust.https://cert-manager.io/inject-bundle: "true"

Hi Folks,

I played around a bit and ran into an error when using targets with the same key. Unless there's a way, that satisfies the use case, I find it somewhat limiting.
Let's take the following example:

I want to distribute the root ca bundle for the webhooks in linkerd:

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: linkerd-webhook-trust-roots
spec:
  sources:
    - secret:
        name: "linkerd-webhook-trust-roots"
        key: "ca.crt"
  target:
    configMap:
      key: "ca-bundle.crt"

Also, I want to distribute the root ca bundle for the linkerd trust anchor:

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: linkerd-identity-trust-roots
spec:
  sources:
    - secret:
        name: "linkerd-identity-trust-roots"
        key: "ca.crt"
  target:
    configMap:
      key: "ca-bundle.crt"

This fails with admission webhook "trust.cert-manager.io" denied the request: spec.target.configMap.key: Invalid value: "ca-bundle.crt": cannot use the same target as another Bundle "linkerd-identity-trust-roots".

Is there a specific reason for not supporting that kind of behaviour?

HI, I might be missing something here:

I setup issuers and certificates for linkerd and I try to get the Root CA into the config map as required.
The error I get is: Bundle source was not found: failed to retrieve bundle from source: Secret "linkerd-identity-issuer" not found

--- # Linkerd CA 
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linkerd-identity-issuer
  namespace: linkerd
spec:
  secretName: linkerd-identity-issuer
  duration: 48h0m0s
  renewBefore: 25h0m0s
  issuerRef:
    name: linkerd-trust-anchor
    kind: Issuer
  commonName: identity.linkerd.cluster.local
  dnsNames:
  - identity.linkerd.cluster.local
  isCA: true
  privateKey:
    algorithm: ECDSA
  usages:
  - cert sign
  - crl sign
  - server auth
  - client auth
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: linkerd-identity-trust-roots
  namespace: linkerd
spec:
  sources:
  - secret:
      name: linkerd-identity-issuer
      key: ca.crt
  target:
    configMap:
      key: ca-bundle.crt

When I use the console and check the secret I can that the data itself is there.
Do you have a hint on what I am missing here?

main@server5:~$ microk8s kubectl describe secret linkerd-identity-issuer -n linkerd
Name:         linkerd-identity-issuer
Namespace:    linkerd
Labels:       controller.cert-manager.io/fao=true
Annotations:  cert-manager.io/alt-names: identity.linkerd.cluster.local
              cert-manager.io/certificate-name: linkerd-identity-issuer
              cert-manager.io/common-name: identity.linkerd.cluster.local
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group:
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: linkerd-trust-anchor
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
tls.crt:  749 bytes
tls.key:  227 bytes
ca.crt:   595 bytes

De you plan to provide a mutatingWebhookConfiguration to add the generated configmap on all pods in a target namespaces?

This can help us to deploy the bundle like the kube-root-ca.crtconfigmap.

I have multiple namespaces that don't trust each other and have their own per namespace root ca

I would like to be able to specify a bundle per namespace instead of 1 for the whole cluster

This would also help with the cleanup when said namespaces get deleted

I am attempting to install trust-manager into openshift with an existing cert-manager install that was installed via the operator hub.

bash-4.4 ~ $ helm upgrade -i -n openshift-operators trust-manager jetstack/trust-manager --set app.trust.namespace=openshift-operators --set app.webhook.tls.approverPolicy.certManagerNamespace=openshift-operators  --wait --dry-run --debug
history.go:56: [debug] getting history for release trust-manager
upgrade.go:142: [debug] preparing upgrade for trust-manager
Killed

Despite already having cert-manager, so I need to deploy an instance into a cert-manager namespace like the docs state, or should I be able to use the one I have?

The new useDefaultCAs option doesn't render nicely in kubectl describe:

Spec:
  Sources:
    Use Default C As:  true

This is because kubectl describe tries to be clever about separating words but fails to work in this case.

The relevant code from kubectl describe is here.

Options:

1. Keep name and change upstream handling of the name. Likely hard and slow to roll out.
2. Keep name and somehow change how it's rendered. I can't see any way to do this currently.
3. Change name
4. Live with it

I am looking for a way to create the Bundle with the domain in order to create the PEM

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: trust-bundle
  namespace: default
spec:
  sources:
  - domain: https://example.com           # <-- expecting the trust manager to pull the PEM by domain
  target:
    configMap:
      key: trust.pem

This is perhaps a flawed request from a security standpoint. However, it would increase the user-friendliness of the trust project potentially.

Just like how currently a certificate in cert-manager has a ca.crt key, it would be great to not have to manually fetch the root for a certain issuer and just have a Bundle object "trust" an issuer, so that it would get the root automatically since the issuer can already fetch it but also, more importantly, rotate the root automatically across the cluster when it changes.

Is this something you would explore perhaps?

Thanks!

👋 Hello there.

We're currently trying to deploy trust-manager v0.5.0, via the Helm chart, to a Kubernete 1.27.2 cluster and running in to a SIGSEGV from the trust-manager deployment.

It's worth noting that we have a few 1.24.* clusters running v0.5.0 of trust manager just fine.

If I understand the client-go versioning correctly, this project is consuming v0.27.2 so it should be 1.27.2 compatible. 🤔

Thanks for any/all pointers towards a solution here. 🙇‍♂️

sigsegv.txt 👇

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12c55d0]
goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
	/go/pkg/mod/k8s.io/[email protected]/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc000147e18, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/go/pkg/mod/k8s.io/[email protected]/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0001460f0, 0x15}, {0xc0003e05c0, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
	/go/pkg/mod/k8s.io/[email protected]/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x18da633?)
	/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc00013aaa0?)
	/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x1b6cd28, 0xc000133200})
	/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
	/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0000e6e90)
	/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x203000?)
	/go/pkg/mod/k8s.io/[email protected]/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x1b6cd28?, 0xc000133200?})
	/go/pkg/mod/k8s.io/[email protected]/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/apiutil/dynamicrestmapper.go:86 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/apiutil/dynamicrestmapper.go:119
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc00013cd80?, {0x0, 0x0, 0x1?})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/client/apiutil/dynamicrestmapper.go:99 +0x176
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc0000c0770?)
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc0003ee000, {0xc0004ad818, 0x1, 0x1b69148?})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc0000c0770, 0x0, 0x0, {{0x1b69148, 0xc000401e00}, 0x0}, 0x1, {0x0, 0x0}, ...})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/manager.go:351 +0xf9
github.com/cert-manager/trust-manager/cmd/trust-manager/app.NewCommand.func1(0xc00007e000?, {0x18db28d?, 0x9?, 0x9?})
	/workspace/cmd/trust-manager/app/app.go:67 +0x479
github.com/spf13/cobra.(*Command).execute(0xc00007e000, {0xc000124010, 0x9, 0x9})
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:916 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc00007e000)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
main.main()
	/workspace/cmd/trust-manager/main.go:29 +0x1e

Hello everyone! I would like to propose to you to add a namespaceOverride parameter in values.yaml like it was done for cert-manager.

In my case I have a single big chart, which used to install cert-manager, trust-manager and other stuff. In current configuration namespaced resources will be installed in release namespace, but I want to control that behavior.

It would be beneficial to add image source options to trust-manager in the same way that cert-manager has them.

For example, in cert-manager, each workload image has the following structure in the template and values file.
Template snippet:

containers:
        - name: {{ .Chart.Name }}-controller
          {{- with .Values.image }}
          image: "{{- if .registry -}}{{ .registry }}/{{- end -}}{{ .repository }}{{- if (.digest) -}} @{{ .digest }}{{- else -}}:{{ default $.Chart.AppVersion .tag }} {{- end -}}"
          {{- end }}

values.yaml snippet:

image:
  repository: quay.io/jetstack/cert-manager-controller
  # You can manage a registry with
  # registry: quay.io
  # repository: jetstack/cert-manager-controller

  # Override the image tag to deploy by setting this variable.
  # If no value is set, the chart's appVersion will be used.
  # tag: canary

  # Setting a digest will override any tag
  # digest: sha256:0e072dddd1f7f8fc8909a2ca6f65e76c5f0d2fcfb8be47935ae3457e8bbceb20

Repository and tag are the only options available on trust-manager.

Motivation

I deploy cert-manager and trust-manager together as sub-charts of a parent chart, and need to serve images from a private registry. Having this feature will allow a single registry to be configured once and declared as a yaml anchor, and referenced as an alias for cert-manager and trust-manager.

Since Bundle is cluster-scoped, ClusterBundle seems like it would be the proper name (which seems correct, given that spec.target.namespaceSelector suggests that the Bundle should be at a Cluster scope or need an "AcceptBundle" object in each namespace to indicate that the namespace wants to receive the certs).

Writing down what occurred to me at the end of the cert-manager biweekly meeting on 28/07

An issue with administering trust bundles is that it's very easy + tempting for users to put intermediates in the trust store as a way of "fixing" trust issues. Usually this occurs when an app is buggy and only sends the leaf certificate without its chain, and explicitly trusting the leaf's issuing intermediate CA seems to fix the problem for them. Crucially, when that intermediate is then rotated their app will break.

We could add a toggle, defaulting to false, such as allowIntermediateCertificates. If false, it would parse every certificate and ensure that it's self-signed. There might be other ways of achieving the same goal.

Having this in from the start could prevent so many misconfigurations by unsuspecting cluster admins.

(Complication: If we added a list of publicly trusted certs it'd probably need to default to allowing intermediates in that list by default - but we could definitely default to disallowing intermediates everywhere else)

We have the following use case:

Service A needs to talk to another service B within the same cluster, both are Java applications.
We are attempting to terminate SSL traffic at the pod level, which means loading the cryptomaterial at the Java runtime level.
We have used cert-manager to issue to server certificate and it has been loaded through the newly released Java Key Store secret output format in cert-manager 1.8+.

In order to establish trust across the services, we are making use of the trust project to use an already created Bundle object that contains the internal CA used for ingress.
However it turns out the trust bundle needs to also be loaded as a Java Keystore file, this is an assumption relayed from the customer and should be 100% confirmed first.

This is what this issue is trying to have trust Bundle have different output formats matching the cert-manager project with specifically the Java trust bundle store format as a priority.

It would be also nice if there was a way to get roots automatically into Bundle objects tracking a certain issuer, but this is a separate issue and not essential to this problem.

This issue is raised from discussion in #2, regarding whether or not to add support for adding publicly trusted roots to cert-manager/trust.

I'm raising the issue now as a point of discussion for this feature in the future, not because I'm pushing for it at the time I'm raising it - in other words, this is a cleanup.

My initial proposal:

I think Bundle would be tremendously improved by supporting the ability to include publicly trusted CAs, such as Mozilla's list. I worry that if we don't support a publicly trusted list, people will hack around it and get themselves into all kinds of trouble by statically including a bundle which will inevitably never get updated.

I don't know if such a publicly trusted list is required for a v1 release, but I think it would be a huge improvement to have it in place; it'd enable Bundle to essentially be the one-stop-shop for all trust needs for the vast majority of workloads.

While I absolutely agree this would be a great addition, I feel we run the risk of;

1. The baked in trust bundles becoming out of date meaning the tool has an expiry data if left unmaintained,
2. Open a flood gate of "also add this default trust store" feature creep.

I definitely understand why you'd be in two minds, but I'm not hugely concerned by either of these things.

1. The baked in trust bundles becoming out of date meaning the tool has an expiry [date] if left unmaintained,

I think there's a case to be made for allowing the tool to dynamically fetch updated trust stores at runtime, although there are definitely questions which would need to be answered to implement that (where to fetch from, how often, how to surface an updated trust bundle to cluster admins, etc).

Whether or not we implement dynamic updating at runtime, I don't think most cluster operators would be overly concerned by using a baked trust store, since a lot of them are just going to be trusting whatever's provided in their container images, be that alpine, centos, debian, whatever.

By "centralising" on cert-manager/trust, they potentially only have to update one trust store (via a Bundle) rather than having to keep track of the disparate trust stores in their running applications on different distros.

Plus if we stopped maintaining cert-manager/trust, I think they'd be fine; it's open source and they could easily create new versions with updated baked trust stores.

2. Open a flood gate of "also add this default trust store" feature creep.

In theory I'd be concerned about this, but I don't think in practice there are actually many lists of publicly trusted roots that people would want in the first place. Most Linux distros that I know of just use Mozilla's list, and the only other major list I'd expect to hear about is Chrome/Chromium's. If in the worst case people start requesting a bunch of stuff we don't want to add, I'm personally quite comfortable saying "no" 😉

UX Without Publicly Trusted Roots

My fear is that if we don't add support for a publicly trusted list of roots, the UX will suffer massively. The ideal UX for TLS clients connecting to servers IMO is:

• Create Bundle resource + mount resulting target ConfigMap into their containers
• Use TLS as normal both with public and private CAs

That is, they could mount the trust store and replace the system trust store for their container - so they'd not need to modify RootCAs in tls.Config, --ca-file in curl or whatever the equivalent is for their language / tool.

They could combine the mounted ConfigMap and the system trust store in the container, but that requires them to run something like this at container start-up time, which is a pain and is error prone.

Basically, my vision here is that cert-manager/trust could be the one-stop-shop for trust everywhere in a cluster, providing visibility and a central place to update trust stores.

Originally posted by @SgtCoDFish in #2 (comment)

We are on RKE with Kubernetes 1.21. When I install the helm chart I see the replicaset is stuck with 0 copies and this attached event:

% % k -n cert-manager describe rs cert-manager-trust-6796dcc8f5
Events:
  Type     Reason        Age                      From                   Message
  ----     ------        ----                     ----                   -------
  Warning  FailedCreate  4m32s (x548 over 5d20h)  replicaset-controller  Error creating: pods "cert-manager-trust-6796dcc8f5-" is forbidden: PodSecurityPolicy: unable to admit pod: [pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/cert-manager-trust]: Forbidden: seccomp may not be set pod.metadata.annotations[container.seccomp.security.alpha.kubernetes.io/cert-manager-trust]: Forbidden: seccomp may not be set]

and it does not appear that seccomp is a configurable field on a PSP. So I'm not sure how to allow the pod to define seccomp. Would the project be open to a flag for optionally hiding the securityContext option?

Currently only configMap is supported as target. It feels to me like that's the obvious choice for distributing a ca file. However, a lot of off-the-shelf applications out there only support mounting ca files via secrets

Would it be possible to add it for compatibility reasons?

I haven't found a way to get help, so perhaps the main issue for me is: is there a discord channel or something for questions? Sorry if this is not the correct way to do that.

I want to create a docker-registry secret for pulling images from local registry (trow). It works when I manually create a secret key, but I want to use trust-manager to handle that.

Method I use now:

kubectl create secret docker-registry trow-mydomain-credentials \
--namespace mynamespace \
--docker-server=trow.mydomain.home \
--docker-username='USERNAME' \
--docker-password='PASSWORD' \
[email protected]

See https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Installing the chart with apiVersion v1 through kustomize along with a bundle resource in the same go is resulting in dry run errors because the CRDs are not evaluated separately

I notice in the README it states and syncs them to a target in every namespace. - is there any capability to limit which namespaces a particular bundle is synced to? e.g. by label, or explicitly by name?

Hi,

Can we add a value to disable the install of the bundle CRD?

Happy to open a PR.

Thanks,
Max Sargent

(This is a follow up from discussions which took place in the cert-manager biweekly meeting on 2022-07-27)

Currently Bundle is essentially a Kubernetes implementation of the Unix cat command - it takes several text inputs and concatenates them into an output. This allows it to be agnostic to the type of thing it's concatenating, but it has a few shortcomings:

• It's not possible to concatenate binary data in a correct and meaningful way
• It's not possible for us to vary input/output types
• It's easy to misuse one type of output for a purpose it wasn't intended

For example, it's currently essentially impossible to create a TLS trust Bundle which includes as inputs a list of PEM certificates and separately a DER-encoded certificate, and produce an output in any meaningful format. Without context, we can't decode the base64-encoded binary data for the DER certificate or convert it to PEM, and we can't concatenate a PEM input certificate with a DER certificate without doing some conversion beforehand.

It would also be desirable to support outputting in PEM or JKS or other keystore types.

Proposal

What was discussed in the biweekly meeting was the following:

1. Bundle as it exists today becomes specialised for X.509 certificates (specifically for the purposes of creating TLS trust bundles, but we can support X.509 more widely)
2. We'd like to support other keystore formats, but might need a new CRD to be able to do so. Examples mentioned included Wireguard keys or keys for code signing verification
3. The default output type is assumed to be a concatenated list of PEM encoded certificates; that'll be the default. We don't necessarily need to support more formats at first.
4. The default input type is assumed to be a list of PEM encoded certificates. We should at least support DER encoded certs for input as well.
5. All input certificates are should be convertible to some intermediate representation, and that intermediate representation should be convertible to all output formats. We propose the intermediate format be a list of PEM encoded certificates.

Example

This would produce an output containing all the certs in "my-pem-certs" with the cert in "my-der-cert" appended to the end. The output would be in PEM format.

apiVersion: trust.cert-manager.io/v1alpha2
kind: Bundle
metadata:
  name: my-bundle
spec:
  sources:
  - secret: # defaults to PEM because the type isn't specified
      name: "my-pem-certs"
      key: "ca.pem"
  - configMap:
      name: "my-der-cert"
      key: "root-certs.crt"
      type: "der" # "der" assumes that the value is base64 encoded binary, which is the k8s convention
#  - inLine: |  # would error out because this isn't a valid cert
#      abc123 not a cert
  target:
    configMap:
      key: "output-certs.pem" # defaults to PEM because format not specified
    namespaceSelector:
      matchLabels:
        mylabel: "true"

Now that I have installed trust, configured a bundle and I have a configMap propagated around the various namespaces of my cluster, I now need to feasibly use said CA bundle with applications. I think there are some usability and security concerns to accommodate here:

• We probably need some docs on https://cert-manager.io/docs/projects/trust/ on how best to consume said bundles.
• Most containers don't run as root and therefore won't be able to use update-ca-certificates at runtime
• Application might not have TLS configuration options.
• Deployment resources need updated mounting the relevant configmap
• Potentially creating a dependency on that configMap in the application life cycle.

Am I right in thinking that the best approach to update CA's trusted at runtime of a non-root container, is to effectively use a privileged init container with a shared volume with the application and some entrypoint.sh type magic? Something like the suggestion here

Perhaps it's easier to explain where I think I would like to be. I manage clusters with many tenants. I would like at runtime to automatically inject my trusted CA bundle to applications. Application use only that bundle without having to be specifically referenced by the tenants applications and deployments.

End result is that you effectively control trust at runtime through the cluster without requiring tenant changes. There is then potential to roll trust by refreshing or creating new bundles. This sort of reflects my experience that TLS is usually a platform / security concern and that application teams tend not to care about the runtime environment so much. Say for example an environment relying on internal CAs.

I'm not sure how achievable the above is, but perhaps some documentation on the trust project on typical use cases might be a start at least? At least highlight the gaps or limitations to address.

It would be great if it would be possible to use CRDs (with a configurable path) as targets.

For more details please see: cert-manager/cert-manager#5073

We discussed this in the cert-manager meeting and the idea was to potentially make this possible via cert-manager/trust. (and that it's not a good fit for cainjector)

At the moment if bundle.spec.target gets modified, trust manager will apply two updates in separate reconciles- one where the old CA data is deleted from the target configmaps, another where the new CA data is written to the target configmaps. This will happen even when the CA data does not need to change, for example if namespace selector is removed.

In practice this means that there is a period of time where the CA data is not available and in some cases it could cause issues:

• if there is a latency for calls to kube apiserver either because there is a large number of targets that trust-manager needs to update or because the apiserver is busy, the period when the CA cert is not available will be longer
• if user applies a change to the Bundle that both changes the target and also references a source that cannot be used (is an invalid CA cert), the old CA data will be removed and the configmaps will remain empty till the user fixes the issue

Assuming that we think that CA data should be 'always' available, I think it would be preferable if target updates were atomic and we didn't have a period of time where the target configmaps are empty.

I ran an experiment with 1000 namespaces and observed that the after a target namespace selector change a target configmap had its CA cert deleted and remained empty for 5 minutes till the same CA data got added back:

irbe@trust-manager$ k logs deploy/trust-manager -ncert-manager | grep test-1000
I0119 13:24:53.280164       1 bundle.go:151] trust/bundle "msg"="deleted old target key" "bundle"="my-org.com" "namespace"="test-1000" "old_target"={"configMap":{"key":"ca.crt"},"namespaceSelector":{"matchLabels":{"foo":"bar"}}}
I0119 13:29:15.481179       1 sync.go:230] trust/bundle "msg"="synced bundle to namespace" "bundle"="my-org.com" "namespace"="test-1000"

Related (wrt whether we should prioritize CA data being always available): #99

I installed cert-manager-trust via Helm with the 0.2.0 chart (https://artifacthub.io/packages/helm/cert-manager/cert-manager-trust). I had no errors when installing it. When I try to create a bundle resource, I get the following message:

Internal error occurred: failed calling webhook "trust.cert-manager.io": Post "https://cert-manager-trust.cert-manager.svc:443/validate?timeout=10s": context deadline exceeded

I increased to 10s the timeout to discard that as an issue. When debugging, I saw the following logs in the trust pod:

I1018 09:07:55.682760       1 webhook.go:32] trust/webhook "msg"="registering webhook endpoints"
I1018 09:07:55.683436       1 internal.go:362] trust/manager "msg"="Starting server" "addr"={"IP":"::","Port":9402,"Zone":""} "kind"="metrics" "path"="/metrics"
I1018 09:07:55.683575       1 internal.go:362] trust/manager "msg"="Starting server" "addr"={"IP":"::","Port":6060,"Zone":""} "kind"="health probe"
I1018 09:07:55.683872       1 leaderelection.go:248] attempting to acquire leader lease cert-manager/cert-manager-trust-leader-election...
I1018 09:08:00.277660       1 leaderelection.go:258] successfully acquired lease cert-manager/cert-manager-trust-leader-election
I1018 09:08:00.283090       1 controller.go:178] trust/manager/controller/bundle "msg"="Starting EventSource" "reconciler group"="trust.cert-manager.io" "reconciler kind"="Bundle" "source"="kind source: *v1alpha1.Bundle"
I1018 09:08:00.283556       1 controller.go:178] trust/manager/controller/bundle "msg"="Starting EventSource" "reconciler group"="trust.cert-manager.io" "reconciler kind"="Bundle" "source"="kind source: *v1.PartialObjectMetadata"
I1018 09:08:00.284286       1 controller.go:178] trust/manager/controller/bundle "msg"="Starting EventSource" "reconciler group"="trust.cert-manager.io" "reconciler kind"="Bundle" "source"="kind source: *v1.Namespace"
I1018 09:08:00.284315       1 controller.go:178] trust/manager/controller/bundle "msg"="Starting EventSource" "reconciler group"="trust.cert-manager.io" "reconciler kind"="Bundle" "source"="kind source: *v1.PartialObjectMetadata"
I1018 09:08:00.284335       1 controller.go:178] trust/manager/controller/bundle "msg"="Starting EventSource" "reconciler group"="trust.cert-manager.io" "reconciler kind"="Bundle" "source"="kind source: *v1.PartialObjectMetadata"
I1018 09:08:00.284355       1 controller.go:186] trust/manager/controller/bundle "msg"="Starting Controller" "reconciler group"="trust.cert-manager.io" "reconciler kind"="Bundle"
I1018 09:08:00.283375       1 recorder.go:103] trust/manager/events "msg"="Normal"  "message"="cert-manager-trust-6c4bfdb98f-9b6zm_27895cf5-9a41-472b-b3b0-39252ad0a2af became leader" "object"={"kind":"ConfigMap","namespace":"cert-manager","name":"cert-manager-trust-leader-election","uid":"34490034-b485-42b3-8d5b-d5e0f05715ba","apiVersion":"v1","resourceVersion":"25487825"} "reason"="LeaderElection"
I1018 09:08:00.284555       1 recorder.go:103] trust/manager/events "msg"="Normal"  "message"="cert-manager-trust-6c4bfdb98f-9b6zm_27895cf5-9a41-472b-b3b0-39252ad0a2af became leader" "object"={"kind":"Lease","namespace":"cert-manager","name":"cert-manager-trust-leader-election","uid":"22f79707-6aca-497c-92ef-e964990795c6","apiVersion":"coordination.k8s.io/v1","resourceVersion":"25487826"} "reason"="LeaderElection"
I1018 09:08:00.385279       1 controller.go:220] trust/manager/controller/bundle "msg"="Starting workers" "reconciler group"="trust.cert-manager.io" "reconciler kind"="Bundle" "worker count"=1

I tried doing a port-forward to the service and I had issues with the certificate. I get a 404 page not found error. In the pod logs I can find:

2022/10/18 09:25:41 http: TLS handshake error from 127.0.0.1:53246: remote error: tls: unknown certificate

Then if I do the port-forward to the pod itself, no SSL is needed. And if I go to /readyz, it answers ok. So I am suspecting there is an issue with the service forcing to use SSL, and maybe not being able to connect because the certificate is invalid.

Please add CI step to expose CRDs as a release artifact. This is an example of the convenience I'm after from the cert-manager project: https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.crds.yaml

As a workaround, it's still possible to fetch the CRDs directly if you know the release tag, e.g., https://raw.githubusercontent.com/cert-manager/trust-manager/v0.5.0/deploy/crds/trust.cert-manager.io_bundles.yaml

We are using cert-manager to issue short-lived certificates to Kafka-clients running in our Kubernetes cluster. Currently we use ca.crt truststore.jks or truststore.p12 in the certificate Secret to establish trust to the Kafka cluster (mTLS). This is wrong, and only works because the client and server have certificates issued by the same root CA. 😄

To fix this, we were hoping to use this cert-manager component to distribute the trust. But since a great percentage of our clients are Java-based, we need to be able to provide the trust bundle in PKCS12/JKS format. Would it be possible to add an opt-in to provide the trust bundle in additional formats?

Hello,
I want to install this application to different namespace then app.trust.namespace but I got error about not sufficient permission to create leader election lock. My use case is to install trust application several times in cert-manager namespace but to point to different trust namespace.

The problem is caused by https://github.com/cert-manager/trust/blob/main/deploy/charts/trust/templates/role.yaml that creates permission in deployment namespace not in trust namespace causing application does not have sufficient permission to create leader election lock and to watch secrets in trust namespace

I would like to be able to add labels and annotations to the target.

There's no warning currently if you install trust-manager using the old chart repo or old container images.

People should get a warning on install that they need to install from a new chart repo + use new container images!

I propose to branch from the commit before we renamed trust -> trust-manager, and create a new commit for a one-off release of trust-manager (maybe v0.2.1). In that release, we'll add a NOTES.txt to the helm chart to add a note on upgrading to a new version, and add a "deprecated" flag to that helm chart if possible

We could also update the container images on that branch to print a warning on startup that they're deprecated, too.

We'll also need to add notes to the quay.io repo, and to the old artifacthub repo.

At the moment, each target configmap has an owner reference to the Bundle for which it was created.
This means that if a Bundle gets deleted, the configmaps will be deleted by Kubernetes garbage collector.

This can be useful for folks who sync them to a large number of namespaces and want to be able to easily delete Bundle and all related configmaps.

Judging from our experience with cert-manager, it is likely that in the future there will be scenarios where users end up uninstall and re-install trust manager including the Bundle CRD, either to perform a more complex upgrade or to perform some other cluster component configuration change like moving to a different deployment mechanism etc.

If configmaps with CA certs get deleted it means that

• new pods that mount these configmaps cannot start
• any service that reads the CA certs from configmaps directly can't perform its operations.
  This would mean that it is not safe to re-install trust-manager as it can cause downtime.

I think we should make it possible to re-install trust-manager without deleting the synced CA certs whilst still giving users an easy way to delete Bundles and associated configmaps.

We could:

• make owner references optional and configurable via flag
• make owner references optional and configurable on each Bundle CR
• remove owner references altogether, provide a cmctl command to clean up configmaps for Bundles (depends on splitting cmctl out of cert-manager codebase)

At the moment owner references are used heavily by trust controller (to filter configmaps to watch, list etc). If we are going to get rid of them or make them optional, we'd need to devise another way how to filter target configmaps. One way could be labelling. Server side apply could also help- we could look at managed fields to determine whether a configmap was added by trust manager #89 . Also, if we are going to do it, it'd be easier now than when there are more users and stability guarantees.

The next step for this issue would be to come up with a design.

We are using Cluster API(CAPI) for LCM of kubernetes. CAPI has the ability to deploy many target clusters from a management cluster. So we are creating a management cluster which can deploy multiple target clusters. Here, I would like to rotate a k8s cluster CA which involves many different steps and restarts (rolling upgrade) of pods and updates on other resources (config maps, secrets, service accounts) which is manual: k8s CA rotation

I am looking for available options to do automatic CA rotation at scale as manual operation on each cluster will be very costly and time consuming. So, it would be interesting to know and understand how the community is addressing this issue. Are there any plan to support this feature or how would you tackle this kind of situation? I would like to get your opinions and suggestions.

There are also use cases in which the CA of the target clusters might be different from that of the management cluster.

1. Deploy of management cluster and multiple target clusters with the same CA. Perform the cluster CA rotation on the target clusters and the management clusters without impact on traffic.
2. Deploy of management cluster and many target clusters with different CA. Perform the cluster CA rotation on the target clusters and the management clusters without impact on traffic.

/kind feature

When I try to talk about this project I struggle massively every time. "Trust" is such an important word in TLS that it can be incredibly confusing to talk about a "a trust managed trust bundle".

I usually end up calling this the "trust project" which sounds like the title of some investigative journalism initiative, or else "cert-manager/trust" when I write the name to avoid ambiguity, which is verbose.

I've proposed the name "truman" (trust manager) before, in part as a homage to one of the best films ever and in part because I think the name works. I'd happily accept another name. But I really think we've got a lot to gain from changing the name of this while it's still a relatively small project! Renaming is only going to get harder, so it might be a case of "the sooner the better".

Proposed names so far:

• truman (Ash)
• trust-manager (James M)
• trust (the status quo)
• confido (Steve)
• perfidy (Anon)
• credence (Anon)