Certificate issue in the agent container for docker pulling from public repository about kubernetes-elastic-agents HOT 6 CLOSED

@Maxsh Do you have any kind of a proxy in place, when it is used from within Kubernetes? You mentioned trying it in the node. Do you mind bringing up a pod and then trying it there?

I would set the task to something like: sleep 1800 and then kc exec into the pod to try this docker pull manually and see what happens and debug further.

from kubernetes-elastic-agents.

I was looking at issues such as docker-archive/toolbox#603 (comment) to see if this could be related. They were talking about proxies. You could also see if running update-ca-certificates helps, as mentioned in this other comment.

from kubernetes-elastic-agents.

The other aspect I was thinking about was whether the time inside the pod and container are correct. I was basing this on the last part of the message, where it says: ... or is not yet valid.

from kubernetes-elastic-agents.

May be https://docs.docker.com/machine/reference/regenerate-certs/ will help? On the k8s node on which the pod was scheduled. Best to check by doing a ssh into a k8s node and trying docker pull php:7.2-fpm-alpine3.8.

from kubernetes-elastic-agents.

Thanks all for your suggestions.
It seems the issue is resolved. It was depends on DNS resolving issue from the pod.

Here are some details:

1. It worked properly from docker container running through docker run
   $ docker run --privileged --rm -it gocd/gocd-agent-docker-dind:v19.7.0 sh
curl -v https://registry-1.docker.io/v2/_catalog

2. It doesn't work in the pod container:
   $ curl -v https://registry-1.docker.io/v2/_catalog
TLSv1.2 (OUT)
TLS alert, unknown CA (560):
SSL certificate problem: unable to get local issuer certificate
   and
   $ openssl s_client -CApath /etc/ssl/certs/ -connect registry-1.docker.io:443
CONNECTED(00000003)
depth=0 CN = localhost
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = localhost
verify error:num=21:unable to verify the first certificate
verify return:1

3. But, other domains works properly e.g. https://google.com:443 from the pod container

Then it's revealed, that DNS for registry-1.docker.io is not resolved from the agent container and from server container as well.
$ nslookup registry-1.docker.io
There are other pods in other k8s namespcases where this domain is resolved properly.
Moreover, in a new busybox pod in gocd namespace, resolving works.

So, the issue was resolved by changes in the Agent Profile / PodConfiguration
dnsPolicy: ClusterFirst
dnsConfig:
nameserver:
- 10.96.0.10
- <COMPANY_DNS_IP>
searches:
- cluster.local
- svc.cluster.local
- gocd.svc.cluster.local
options:
- name: ndots
value: '2'

Looks like the issue depends on the known ndots in k8s issue for Alpine distributive
jenkinsci/docker#710 (comment)
kubernetes/kubernetes#64924

But, I'm not sure precise, because other my pods also based on Alpine and there Docker registry host is resolved successfully. The issue is only for GoCD pods (server and agent)

from kubernetes-elastic-agents.

Hi @Maxsh

We tried to reproduce this issue on GKE and couldn't. We provisioned a static agent from the image "gocd/gocd-agent-docker-dind:v19.7.0".

We could verify that nslookup registry-1.docker.io is resolved.

We then ran curl -v https://registry-1.docker.io/v2/_catalog on the agent container. The docker registry certificate is verified.

Here's the output:

$curl -v https://registry-1.docker.io/v2/_catalog
*   Trying 3.210.179.11:443...
.
.
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.docker.io
*  start date: Jun  7 00:00:00 2019 GMT
*  expire date: Jul  7 12:00:00 2020 GMT
*  subjectAltName: host "registry-1.docker.io" matched cert's "*.docker.io"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.

Can you confirm to us where your K8s cluster is running so we can debug this further?

Sheroy and Arvind

from kubernetes-elastic-agents.