Toolkit version parser broken for casual kind demo user

[echoH00]

Installed with A (kind) demo

DRA driver components are running and in a Ready state

root@ubuntu24:~# kubectl get pods -n nvidia-dra-driver-gpu
NAME READY STATUS RESTARTS AGE
nvidia-dra-driver-gpu-controller-5d996d7c4f-bhjzw 1/1 Running 0 16m
nvidia-dra-driver-gpu-kubelet-plugin-grcdn 2/2 Running 0 16m

run workload pod CrashLoopBackOff

root@ubuntu24:~/k8s-dra-driver-gpu# kubectl apply -f ./demo/specs/quickstart/gpu-test2.yaml
gpu-test2 pod 0/2 CrashLoopBackOff 14 (2m29s ago) 13m

Events:

Warning Failed 12m (x4 over 12m) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running createContainer hook #5: exit status 3, stdout: , stderr: No help topic for 'disable-device-node-modification': unknown

how to fix it

[jgehrcke]

Thanks.

I believe the decisive detail is

No help topic for 'disable-device-node-modification': unknown

But I don't yet fully understand what's happening here. @elezar maybe you can educate me. Until then -- @echoH00 -- which version of the NVIDIA container toolkit are you running?

[echoH00]

Thanks.

I believe the decisive detail is

No help topic for 'disable-device-node-modification': unknown

But I don't yet fully understand what's happening here. @elezar maybe you can educate me. Until then -- @echoH00 -- which version of the NVIDIA container toolkit are you running?

Thanks，v1.17.8

[elezar]

The issue is that the CDI spec that we're generating in this case includes a disable-device-node-modification hook which has only been implemented in v1.18.0 of the NVIDIA Container Toolkit. @echoH00 which version of the DRA driver are you using and how is it being installed? I was under the impression that later versions of the DRA driver ensure that a compatible version of the NVIDIA Container Toolkit is available on the host, and reference this executable explicitly when generating CDI specs.

update: in terms of what's happening. The generated CDI spec(s) include a reference to the nvidia-cdi-hook disable-disable-device-node-modification hook. When running this we see the error above:

 $ nvidia-cdi-hook some-non-existent-hook
No help topic for 'some-non-existent-hook'

(This is the default behaviour for our CLI library urfave for "unknown" / "unexpected" commands)

[echoH00]

@elezar Thanks I used main Branch

Installed driver follow these steps

git clone https://github.com/NVIDIA/k8s-dra-driver-gpu.git
cd k8s-dra-driver-gpu
export KIND_CLUSTER_NAME="kind-dra-1"
./demo/clusters/kind/build-dra-driver-gpu.sh
./demo/clusters/kind/create-cluster.sh
./demo/clusters/kind/install-dra-driver-gpu.sh

when run build-dra-driver-gpu.sh got Error

109 | # Pull the nvidia-cdi-hook binary out of the relevant toolkit container 110 | # (arch: TARGETPLATFORM, set via --platform). 111 | >>> FROM ${TOOLKIT_CONTAINER_IMAGE} AS toolkit 112 | 113 | # Construct production image (arch: TARGETPLATFORM, set via the --platform CLI -------------------- ERROR: failed to build: failed to solve: base name (${TOOLKIT_CONTAINER_IMAGE}) should not be blank make: *** [deployments/container/Makefile:59: build] Error 1

After export TOOLKIT_CONTAINER_IMAGE="nvcr.io/nvidia/k8s/container-toolkit:v1.17.8-ubuntu20.04"
then got image nvcr.io/nvidia/k8s-dra-driver-gpu:v25.8.0-dev

I installed the nvidia-container-toolkit manually on the kind worker node, which could be problematic.
Since I has only a single card and follow the ReadME to do tests with kind,is there a more detailed description

[klueska]

The DRA driver doesn't verify a specific version of the toolkit installed on the host.

It downloads the nvidia-cdi-hook binary from the toolkit container that matches the version of the toolkit vendored into the DRA driver. At runtime it then installs this onto the host at a well known location and references that in the CDI specs that it generates. This way we are sure that the CDI spec we generate is using a compatible nvidia-cdi-hook binary.

This definitely used to work, but I haven't actually tried it myself since this PR was merged: #494

Maybe a regression was introduced.

[jgehrcke]

I took liberty to change the title, now that I believe I understand what happened here.

Maybe a regression was introduced.

With #494, we introduced a new dependency for some local dev activities. Maybe unknowingly, certainly without documenting it.

base name (${TOOLKIT_CONTAINER_IMAGE}) should not be blank

That means: go.mod-parsing (for extracting the toolkit version) failed in your local env.

That, in turn, was resulting in a failing container image build (thanks for sharing the details).

The workaround that you tried was a smart choice, generally:

After export TOOLKIT_CONTAINER_IMAGE="nvcr.io/nvidia/k8s/container-toolkit:v1.17.8-ubuntu20.04"

However, with that you did bake in a bit of an old nvctk binary into the DRA driver's container image. You could retry that with e.g. v1.18.0-rc.3 (what we currently reference in go.mod).

I installed the nvidia-container-toolkit manually on the kind worker node, which could be problematic.

This after all wasn't even used, probably.

For us, we should think about walking back from requiring go list to properly function (we introduced that dependency here). Instead, we may want to parse go.mod with more portable tooling.

--

Edit: motivated by cases like this, I typically like to define these goals:

1. identify the most prominent user/developer workflows that people want to run on their laptops right after cloning the repo
2. properly define their dependencies (like make, docker, ...)
3. document the dependencies
4. test the workflows in CI (given just those limited dependencies)

That's the only way to stay happy.

[echoH00]

However, with that you did bake in a bit of an old nvctk binary into the DRA driver's container image. You could retry that with e.g. v1.18.0-rc.3 (what we currently reference in go.mod).

@jgehrcke Thanks
With toolkit v1.18.0-rc.3 I successfully ran the pod