How to force kubernetes cluster to use local registry?

Question

My kubernetes cluster (LAB) is cut off today from the internet. Apparently because of that none of my pods can startup (no problem with pods in kube-system though):

  Normal   Pulling         40m (x4 over 42m)      kubelet  Pulling image "nginx:latest"
  Warning  Failed          40m (x4 over 42m)      kubelet  Error: ErrImagePull
  Warning  Failed          40m                    kubelet  Failed to pull image "nginx:latest": failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/latest": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: read udp 127.0.0.1:46391->127.0.0.53:53: i/o timeout
  Warning  BackOff         39m (x7 over 41m)      kubelet  Back-off restarting failed container nginx in pod nginx-deployment-5b9455bc67-j9sh9_default(46e9f74f-8d35-467b-b294-b8c469f95fa6)
  Warning  Failed          37m                    kubelet  Failed to pull image "nginx:latest": failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/latest": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: read udp 127.0.0.1:45430->127.0.0.53:53: i/o timeout
  Warning  Failed          7m52s (x3 over 18m)    kubelet  (combined from similar events): Failed to pull image "nginx:latest": failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/latest": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: read udp 127.0.0.1:49261->127.0.0.53:53: i/o timeout
  Normal   BackOff         2m58s (x150 over 42m)  kubelet  Back-off pulling image "nginx:latest"

This is a surprise to me as I thought that any pod that I start on my cluster will first download it's container(s) image to local registry and then the image will be used from here. Apparently im wrong.

Question: How can I force kubernetse cluster to download/pull the images to the LOCAL registry?

Moreover, although I am cut off from internet, i a still able to export image from the (local) repository. HOW COME???

sudo ctr -n k8s.io image export --local testimage docker.io/library/nginx@sha256:18090843a20ba39719ca4f389d509063ea4fcce9d16c7168f62404b6d630bc3e

[UPDATE/SOLUTION] If I get it right, all the images I can see with sudo ctr -n k8s.io image list are actually in local registry.

What is preventing pods to startup is that in their manifest it is set: imagePullPolicy: Always This means: every time u are starting up look for latest image in the remote/native registry

For test, I have created new POD setting imagePullPolicy: IfNotPresent and specifying image name of the pod/container that worked when the internet connectivity was here. This created pod successfully.

This means: imagePullPolicy: IfNotPresent - if u cant reach remote/native registry while starting up, use the image that was alraedy downloaded/pulled earlier (at the first deployment) to the local repository.

I have found that kube-system deployments manifest are set this what and probably this is why they work even though there is no internet.

How to force kubernetes cluster to use local registry?

0 Answers0