Unable to pull images in GKE

[leon8442]

Hello Team,

I have installed kube-image-keeper on my GKE private cluster via helm following your instructions here: https: //github.com/enix/kube-image-keeper/tree/main but as soon as installed I am unable to pull any images from external repositories.

This one of the errors:
"ImagePullBackOff Cannot pull image 'localhost:7439/registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v17.11.0' from the registry."

Same if I try just to deploy a simple hell world app with this command:
kubectl create deployment hello-server
--image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0

Error:
ImagePullBackOff Cannot pull image 'localhost:7439/us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0' from the registry.

Any help is much appreciated.

Thanks!

Leo

[plaffitt]

Hello,

Does the error mention anything about the reason why it cannot pull the image ? (authentication failed, image not found, other...)

You can also check if the image has been put in cache kubectl get cachedimages, if it doesn't, try to describe the image to see latest events related to this image. You can also watch events in the kuik namespace when a new CachedImage gets created to see if there are any details about why it doesn't get cached. Alternatively you could see logs about caching failures in controllers.

If the image is correctly cached, it may be an issue with the proxy component. Inspecting the logs of the proxy may help to determine the cause.

Also, kuik uses the imagePullSecrets found in the corresponding pods to cache and proxy images. You can check that it is aware of what secret it should uses by looking at the Repository yaml kubectl get repositories -o yaml.

[leon8442]

Hi,

This is the detailed error:
Back-off pulling image "localhost:7439/us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0": ErrImagePull: failed to pull and unpack image "localhost:7439/us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0": failed to resolve reference "localhost:7439/us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0": failed to do request: Head "http://localhost:7439/v2/us-docker.pkg.dev/google-samples/containers/gke/hello-app/manifests/1.0": dial tcp [::1]:7439: connect: connection refused: ImagePullBackOff

Image has been storage in cache correctly:
leo@Leos-MacBook-Pro Documents % kubectl get cachedimages
NAME STATUS CACHED RETAIN EXPIRES AT PODS COUNT AGE
us-docker.pkg.dev-google-samples-containers-gke-hello-app-1.0 Ready true 1 2m39s

I don't see any pod running for kube-image-keeper-proxy, I see the workload correctly deployed and in healthy state but no pod scheduled, it just says 0/1.

It is a new private cluster in GKE. I just deployed it, installed the cert manager, installed kube-image-keeper and tried to run a container, this can be easily reproduced for troubleshooting.

Hope this helps,

Leo

[leon8442]

@plaffitt I have done some other tests and it seems GKE nodes cannot resolve localhost, I'm not sure if I have done anything wrong or if this is the default behaviour.
Have you have tried to run kube-image-keeper in GKE?

Regards,

Leo