Comments (11)
brandond
commented on July 4, 2024
Can you check the containerd log file to see what exactly it's actually trying? The kubelet logs do not reflect all the actual endpoints that are attempted when trying the pull.
from k3s.
siddjellali
commented on July 4, 2024
Please find below :
# journalctl -u containerd
Jun 13 22:12:06 dane systemd[1]: Starting containerd container runtime...
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06+02:00" level=warning msg="containerd config version `1` has been deprecated and will be converted on each startup in containerd v2.0, use `containerd config migrate` after upgrading to containerd 2.0 to avoid conversion on startup"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.779983420+02:00" level=info msg="starting containerd" revision=d2d58213f83a351ca8f528a95fbd145f5654e957 version=1.6.33
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.806971366+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.811661649+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.14.0-362.24.1.el9_3>
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.811699248+02:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.811727474+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.811744053+02:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.811756820+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.811784558+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.811915103+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812102408+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.contain>
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812129381+02:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812150951+02:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812162007+02:00" level=info msg="metadata content store policy set" policy=shared
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812300655+02:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812328614+02:00" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812343427+02:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812378510+02:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812431131+02:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812482466+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812517165+02:00" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812532804+02:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812795644+02:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812822207+02:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812838877+02:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812855760+02:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812877434+02:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812893113+02:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812908440+02:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812923601+02:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812939217+02:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812962099+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812979473+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.812994925+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813009819+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813023834+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813039029+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813053456+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813068628+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813084303+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813100624+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813117134+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813130931+02:00" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813147985+02:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.tracing.processor.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813161390+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813175942+02:00" level=info msg="skip loading plugin \"io.containerd.internal.v1.tracing\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.internal.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813188779+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813258181+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813286437+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813395115+02:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] Privileged>
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813454431+02:00" level=info msg="Connect containerd service"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.813548758+02:00" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814182979+02:00" level=info msg="Start subscribing containerd event"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814303227+02:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814311354+02:00" level=info msg="Start recovering state"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814375950+02:00" level=info msg=serving... address=/run/containerd/containerd.sock
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814463729+02:00" level=info msg="Start event monitor"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814527225+02:00" level=info msg="Start snapshots syncer"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814555792+02:00" level=info msg="Start cni network conf syncer for default"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814576978+02:00" level=info msg="Start streaming server"
Jun 13 22:12:06 dane containerd[32602]: time="2024-06-13T22:12:06.814722808+02:00" level=info msg="containerd successfully booted in 0.035886s"
Jun 13 22:12:06 dane systemd[1]: Started containerd container runtime.
from k3s.
siddjellali
commented on July 4, 2024
You might also find this one interesting.
[root@dane pods]# tail -f /var/lib/rancher/k3s/agent/containerd/containerd.log
time="2024-06-13T22:28:38.407225150+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:longhorn-ui-966cf4bb4-fkk69,Uid:eebdc890-fac8-4b99-b2ea-c5d8e3b8fbbd,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error"
time="2024-06-13T22:28:38.407288713+02:00" level=info msg="stop pulling image docker.io/rancher/mirrored-pause:3.6: active requests=0, bytes read=0"
time="2024-06-13T22:28:40.378677781+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:cilium-ppw2b,Uid:3608a2ba-395a-4328-bb54-ab458faebeb7,Namespace:kube-system,Attempt:0,}"
time="2024-06-13T22:28:42.361881615+02:00" level=info msg="trying next host" error="failed to do request: Head \"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 34.226.69.105:443: i/o timeout" host=registry-1.docker.io
time="2024-06-13T22:28:42.363255193+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:traefik-j7sbz,Uid:92c2c2db-a035-4a99-aed0-bacfae01c27c,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error"
time="2024-06-13T22:28:42.363315655+02:00" level=info msg="stop pulling image docker.io/rancher/mirrored-pause:3.6: active requests=0, bytes read=0"
time="2024-06-13T22:28:45.362642962+02:00" level=info msg="trying next host" error="failed to do request: Head \"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 34.226.69.105:443: i/o timeout" host=registry-1.docker.io
time="2024-06-13T22:28:45.364140685+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:longhorn-ui-966cf4bb4-vmr5w,Uid:88d51dcf-04ac-43a9-b24e-fa999a47d088,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error"
time="2024-06-13T22:28:45.364238148+02:00" level=info msg="stop pulling image docker.io/rancher/mirrored-pause:3.6: active requests=0, bytes read=0"
time="2024-06-13T22:28:46.351879381+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-576bfc4dc7-ctrj8,Uid:d72dc388-2480-410c-b193-399b31321db7,Namespace:kube-system,Attempt:0,}"
time="2024-06-13T22:28:51.360740186+02:00" level=info msg="trying next host" error="failed to do request: Head \"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 34.226.69.105:443: i/o timeout" host=registry-1.docker.io
time="2024-06-13T22:28:51.362068104+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:longhorn-manager-ml6z9,Uid:3e4a460f-c35a-44ce-b17c-b58146cad5b1,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error"
time="2024-06-13T22:28:51.362135009+02:00" level=info msg="stop pulling image docker.io/rancher/mirrored-pause:3.6: active requests=0, bytes read=0"
time="2024-06-13T22:28:51.378998139+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:longhorn-driver-deployer-dd57bf77b-js5zh,Uid:610ffe0e-d6c5-4d3f-b9bb-fe3d6a6974b1,Namespace:kube-system,Attempt:0,}"
time="2024-06-13T22:28:52.365412952+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:longhorn-ui-966cf4bb4-fkk69,Uid:eebdc890-fac8-4b99-b2ea-c5d8e3b8fbbd,Namespace:kube-system,Attempt:0,}"
from k3s.
brandond
commented on July 4, 2024
The first log shows only the startup, no pulls. The second log shows only pulling from the upstream, which is the opposite of what you reported. Can you attach the full log, showing a pull that is doing as you described?
from k3s.
siddjellali
commented on July 4, 2024
Traefik image are stored localy but i cannot access it.
I don't have access to internet on this server.
#ctr images ls
REF TYPE DIGEST SIZE PLATFORMS LABELS
docker.io/library/traefik:v3.0.2 application/vnd.docker.distribution.manifest.v2+json sha256:6e5dce54a05259388ae7f48d88a6080adb7e149f46b91327cb4e0b741260d0e1 164.6 MiB linux/amd64 -
# crictl pull docker.io/library/traefik:v3.0.2
E0613 22:59:14.794517 35177 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/library/traefik:v3.0.2\": failed to resolve reference \"docker.io/library/traefik:v3.0.2\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/library/traefik/manifests/v3.0.2?ns=docker.io: 500 Internal Server Error" image="docker.io/library/traefik:v3.0.2"
FATA[0030] pulling image: failed to pull and unpack image "docker.io/library/traefik:v3.0.2": failed to resolve reference "docker.io/library/traefik:v3.0.2": unexpected status from HEAD request to https://127.0.0.1:6443/v2/library/traefik/manifests/v3.0.2?ns=docker.io: 500 Internal Server Error
# tail -f /var/lib/rancher/k3s/agent/containerd/containerd.log | grep traefik
time="2024-06-13T22:58:44.784037699+02:00" level=info msg="PullImage \"docker.io/library/traefik:v3.0.2\""
time="2024-06-13T22:58:51.400635406+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:traefik-j7sbz,Uid:92c2c2db-a035-4a99-aed0-bacfae01c27c,Namespace:kube-system,Attempt:0,}"
time="2024-06-13T22:59:14.792084668+02:00" level=info msg="trying next host" error="failed to do request: Head \"https://registry-1.docker.io/v2/library/traefik/manifests/v3.0.2\": dial tcp 54.196.99.49:443: i/o timeout" host=registry-1.docker.io
time="2024-06-13T22:59:14.793745397+02:00" level=error msg="PullImage \"docker.io/library/traefik:v3.0.2\" failed" error="failed to pull and unpack image \"docker.io/library/traefik:v3.0.2\": failed to resolve reference \"docker.io/library/traefik:v3.0.2\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/library/traefik/manifests/v3.0.2?ns=docker.io: 500 Internal Server Error"
time="2024-06-13T22:59:14.793832826+02:00" level=info msg="stop pulling image docker.io/library/traefik:v3.0.2: active requests=0, bytes read=0"
time="2024-06-13T22:59:21.411640254+02:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:traefik-j7sbz,Uid:92c2c2db-a035-4a99-aed0-bacfae01c27c,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error"
time="2024-06-13T22:59:33.375585471+02:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:traefik-j7sbz,Uid:92c2c2db-a035-4a99-aed0-bacfae01c27c,Namespace:kube-system,Attempt:0,}"
from k3s.
brandond
commented on July 4, 2024
I'm confused by what your problem is then. You're airgapped, and don't want it to fall back to a default endpoint because it's not reachable, but you opened an issue complaining that it doesn't fall back to the default endpoint? And then provided logs showing that it IS falling back to the default endpoint.
What do you want to happen, and what is actually happening?
from k3s.
brandond
commented on July 4, 2024
#ctr images ls
How did you import this image? If you're importing it manually, did you follow the documented instructions to import it into the correct namespace? If you don't import it into the correct containerd namespace, and don't see it in ctr image ls, it won't be available to pull from other nodes.
https://docs.k3s.io/installation/private-registry#adding-images-to-the-private-registry
from k3s.
siddjellali
commented on July 4, 2024
I might be a bit confused myself, so thank you for giving me some time to clarify. I am in an air-gapped environment where I upload my images to the registry using the command ctr pull image . When I start my environment, the first image that Kubernetes tries to load is docker.io/rancher/mirrored-pause:3.6, which is available in the image registry:
# ctr image ls | grep docker.io/rancher/mirrored-pause
docker.io/rancher/mirrored-pause:3.6 application/vnd.docker.distribution.manifest.v2+json sha256:79b611631c0d19e9a975fb0a8511e5153789b4c26610d1842e9f735c57cc8b13 669.8 KiB linux/amd64 -
However, it cannot find the image.
To address this, I have enabled the Embedded Registry Mirror but without success.
I don't think I need to retag the images since I don't have an endpoint, correct?
The image is stored locally, but k3s is still trying to access the internet to download the image docker.io/rancher/mirrored-pause:3.6. This doesn't make sense, right?
# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cilium-operator-6c48dbf7b9-4ncr8 0/1 ContainerCreating 0 87m
kube-system cilium-ppw2b 0/1 Init:0/6 0 87m
# kubectl describe pods -n kube-system cilium-operator-6c48dbf7b9-4ncr8
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 2m51s (x120 over 87m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error
from k3s.
brandond
commented on July 4, 2024
I am in an air-gapped environment where I upload my images to the registry using the command ctr pull image.
That pulls images from a registry, into the containerd image store. Where exactly are you pulling it from if you are airgapped and don't have an upstream registry, or a local mirror? How did it get there?
When I start my environment, the first image that Kubernetes tries to load is docker.io/rancher/mirrored-pause:3.6, which is available in the image registry:
You're not showing the contents of your registry here, you're showing the contents of the containerd image store.
The image is stored locally, but k3s is still trying to access the internet to download the image
Is the image in the correct namespace, as described in the documentation I linked earlier? If its in the wrong namespace, it's not visible to the kubelet.
from k3s.
siddjellali
commented on July 4, 2024
You are right ! i store image with ctr image import <tarball_file>
It look like all image are store in default namespace...
I move every image in namespace where pods are running and i got the same error.
ctr -n kube-system image import <tarball_file>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 81s default-scheduler Successfully assigned kube-system/coredns-576bfc4dc7-z69qj to dane
Warning FailedCreatePodSandBox 10s (x2 over 51s) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to resolve reference "docker.io/rancher/mirrored-pause:3.6": unexpected status from HEAD request to https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io: 500 Internal Server Error
what is ==> .6?ns=docker.io ns as namespace maybe ?.
from k3s.
brandond
commented on July 4, 2024
Please stop trying random things and go read the docs page I linked above... you're still using the wrong namespace.
You need to use the actual namespace shown in the docs, not whatever namespace the pods are in. Containerd's namespaces have nothing to do with Kubernetes namespaces.
from k3s.
Related Issues (20)
- Node cannot get it´s IP - host IP unknown; known addresses: [] HOT 2
- etcd issues when removing and re-adding nodes HOT 1
- Validate SLES 15 SP6
- Validate Amazon Linux 2
- Validate RHEL/Rocky/Oracle 8.10
- Validate RHEL/Rocky/Oracle 9.4
- k3s server crashes abruptly when approximately 300 requests are sent in a span of 60 seconds (client QPS is 5 and bursty limit is 10) HOT 1
- panic: runtime error: invalid memory address or nil pointer dereference HOT 7
- Configuration problem with static CPU manager HOT 1
- Migrate Docker tests from Drone to GHA
- [Release 1.29] Bump flannel v0.25.4 and fix issue with IPv6Masq
- [Release 1.27] Bump flannel v0.25.4 and fix issue with IPv6Masq
- [Release 1.28] Bump flannel v0.25.4 and fix issue with IPv6Masq
- Install systemd unit to run cleanup on shutdown or reboot
- Ingress is going on all the nodes HOT 4
- k3s-agent not starting on ec2 error "failed to get CA certs" HOT 2
- Ingress is going on all the nodes HOT 1
- It looks like the Spegel is not working.
- How to increase NODE podCIDR than `10.42.0.0/24` HOT 1
- Not able to connect to kubelet when using bind-address flag HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k3s.