KinD fails to run single-architecture amd64 manifests on Apple Silicon: "no match for platform in manifest"

[csuich2]

We currently run KinD clusters on our Apple Silicon Macs with a mix of amd64 images and multi-platform (amd64 and arm64) manifests.

Docker 29 switched the default image store to containerd. This causes docker push ... to push multi-platform manifests by default, even when the local build only produced a single platform image. Running docker push --platform linux/amd64, for example, can force a single amd64 image to be pushed rather than a manifest.

Our testing with this change revealed that while running a non-manifest amd64 image works in KinD on an Apple Silicon Mac, running a manifest with a single amd64 image in it does not. Pods attempting to run such images fail to start with this error: no match for platform in manifest: not found.

We're trying to investigate options moving forward, including:

• Updating all docker push commands for non-manifest builds throughout our entire company to add --platform linux/amd64
  • This wouldn't help for any 3rd party single architecture manifests
• Forcing Docker Engine to use the non-default storage engine on our build system after upgrading to Docker 29
  • It is unclear how long this configuration will be supported and the non-default storage engine will be supported

Questions:

• Is this expected behavior? Should KinD support single-architecture manifest lists?
• Is there a containerd configuration within KinD nodes that could handle
• Are there recommended workarounds besides forcing docker push --platform ...?

I also opened this discussion in Moby about the default behavior changing: moby/moby#51532

[BenTheElder]

We currently run KinD clusters on our Apple Silicon Macs with a mix of amd64 images and multi-platform (amd64 and arm64) manifests.

Generally speaking, you should use arm64 on apple silicon, or multi-platform that includes arm64.

see #2718 (kind itself has to run on the host architecture, due to limitations of how docker implements cross-architecture image support)

Our testing with this change revealed that while running a non-manifest amd64 image works in KinD on an Apple Silicon Mac, running a manifest with a single amd64 image in it does not. Pods attempting to run such images fail to start with this error: no match for platform in manifest: not found.

This is expected behavior, the container runtime will look at the manifest for a matching image, this behavior is in containerd, not kind.

Are there recommended workarounds besides forcing docker push --platform ...?

You could explicitly reference the amd64 image regardless of if you push a manifest list or not, if you inspect the manifest list tag then there will be a digest for the amd64 image.

e.g. an image from: https://github.com/kubernetes-sigs/kind/releases/tag/v0.30.0

$ crane manifest kindest/node:v1.34.0@sha256:7416a61b42b1662ca6ca89f02028ac133a309a2a30ba309614e8ec94d976dc5a
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 743,
         "digest": "sha256:ede5bb6984830375d264e41b7169d03643f586d3e35316bf5229eaeb12bb18e0",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 743,
         "digest": "sha256:3229b411042d457a44ea8807e90520e4b7da8700a8617ed20f0c4cb799bebe2b",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

If you use kindest/node@sha256:ede5bb6984830375d264e41b7169d03643f586d3e35316bf5229eaeb12bb18e0 that's always amd64 (which will not work as the kind node, but you can apply this technique to an application image).

Note also #3795 needs fixing, would love for some help with that :-)

I would personally recommend building images for arm64+amd64 or during development just for the host achitecture, if only for performance reasons AND avoiding emulation bugs ...