Wrong CDI spec creates error "Unresolvable CDI device"

[manuelbuil]

I am deploying the nvidia gpu operator with RKE2 Kubernetes in a two node clusters, one node having an nvidia GPU. The version of the nvidia-container-toolkit is v1.17.8. I am enabling cdi with cdi.enabled: true. I believe it installs correctly since all pods are running.

Then, I create a pod with runtimeClassName: nvidia-cdi which requests one nvidia.com/gpu. It fails to start it because:

  Warning  Failed     10s (x2 over 10s)  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-e78e68c0-4e6a-737e-f9cc-4223fe50aead

If I go to the node, I can see it in the output of nvidia-smi:

$> nvidia-smi -L
GPU 0: Tesla T4 (UUID: GPU-e78e68c0-4e6a-737e-f9cc-4223fe50aead)

However, when checking /var/run/cdi/management.nvidia.com-gpu.yaml, it is not there. The only device I see is one with name: all. If I manually run: sudo /usr/local/nvidia/toolkit/nvidia-ctk cdi generate, I can see three devices:

• name: "0"
• name: GPU-e78e68c0-4e6a-737e-f9cc-4223fe50aead
• name: all
  I believe that's the correct yaml. Therefore, I suspect the nvidia-container-toolkit is not correctly generating the CDI spec.

If I check its logs, I actually see a line stating that it is generating the CDI spec and there is no error:

$> kubectl logs nvidia-container-toolkit-daemonset-6jrl2 -n gpu-operator | grep CDI -n2
Defaulted container "nvidia-container-toolkit-ctr" out of: nvidia-container-toolkit-ctr, driver-validation (init)
110-time="2025-08-11T13:36:58Z" level=info msg="Skipping: /host/dev/nvidia-uvm already exists"
111-time="2025-08-11T13:36:58Z" level=info msg="Skipping: /host/dev/nvidia-uvm-tools already exists"
112:time="2025-08-11T13:36:58Z" level=info msg="Generating CDI spec for management containers"
113-time="2025-08-11T13:36:58Z" level=info msg="Using /host/usr/lib64/libnvidia-ml.so.570.172.08"
114-time="2025-08-11T13:36:58Z" level=info msg="Selecting /host/dev/nvidia-modeset as /dev/nvidia-modeset"

I'm a bit at a loss. Can anyone help me debug this further? Do I perhaps need an extra flag in the operator? Is there perhaps something missing in my containerd config.toml file? Thanks for your time!

[oreillymj]

Seems like a known issue with CDI and RKE2 - https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.3.2/release-notes.html

Configuring the Operator to enable CDI is not supported on Rancher Kubernetes Engine 2 (RKE2).

[manuelbuil]

Seems like a known issue with CDI and RKE2 - https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.3.2/release-notes.html

Configuring the Operator to enable CDI is not supported on Rancher Kubernetes Engine 2 (RKE2).

Thanks, I missed that line. I wonder what are the reasons for that problem. I'm part of the RKE2 team and it'd be great to collaborate on fixing this

[manuelbuil]

After some investigation, I found out what's going on and could make it work. There are a couple of wrong configurations.

1 - If I look at the logs of nvidia-container-toolkit-daemonset, I can see it generating the CDI spec

time="2025-09-08T09:08:21Z" level=info msg="Generating CDI spec for management containers"

however, if I look at "/var/run/cdi/management.nvidia.com-gpu.yaml", there is no GPU device, just one called "ALL".

2 - The "/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml" includes the following line:

    [nvidia-container-runtime.modes.cdi]
      annotation-prefixes = ["nvidia.cdi.k8s.io/"]
      default-kind = "management.nvidia.com/gpu"
      spec-dirs = ["/etc/cdi", "/var/run/cdi"]

And in the "/var/run/cdi/management.nvidia.com-gpu.yaml" file I can see:

kind: management.nvidia.com/gpu

however, the nvidia resource in the nodes is announced as:

nvidia.com/gpu

Therefore, there is a misalignment there.

Fix:
1 - Run "nvidia-ctk cdi generate > /var/run/cdi/management.nvidia.com-gpu.yaml", which successfully creates the correct yaml file with the GPU
2 - Change /var/run/cdi/management.nvidia.com-gpu.yaml and "/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml", so that they use "nvidia.com/gpu"

After that, I can see the pod correctly attaching the GPU.

I'd appreciate if someone knew how to install the "nvidia gpu operator" in a way that all this is automated

[elezar]

@manuelbuil could you please confirm the configuration that is applied to the k8s-device-plugin when deployed on your system. We had a recent bug that meant that the correct device list strategy was not applied meaning that this was incompatible with the specified container toolkit configuration (when using CDI mode).

[manuelbuil]

@elezar thanks for replying. I am deploying the GPU operator with the default values, and just enabling cdi. These are the envvars that the operator uses on the nvidia-device-plugin:

    Environment:
      PASS_DEVICE_SPECS:           true
      FAIL_ON_INIT_ERROR:          true
      DEVICE_LIST_STRATEGY:        envvar,cdi-annotations
      DEVICE_ID_STRATEGY:          uuid
      NVIDIA_VISIBLE_DEVICES:      all
      NVIDIA_DRIVER_CAPABILITIES:  all
      MPS_ROOT:                    /run/nvidia/mps
      MIG_STRATEGY:                single
      NVIDIA_MIG_MONITOR_DEVICES:  all
      CDI_ENABLED:                 true
      CDI_ANNOTATION_PREFIX:       nvidia.cdi.k8s.io/
      NVIDIA_CDI_HOOK_PATH:        /usr/local/nvidia/toolkit/nvidia-cdi-hook

[youngjeong46]

Fix:
1 - Run "nvidia-ctk cdi generate > /var/run/cdi/management.nvidia.com-gpu.yaml", which successfully creates the correct yaml file with the GPU
2 - Change /var/run/cdi/management.nvidia.com-gpu.yaml and "/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml", so that they use "nvidia.com/gpu"

@manuelbuil - do I need to restart the RKE service or containerd after this? Doesn't seem to change anything for me after this.

[manuelbuil]

Fix:
1 - Run "nvidia-ctk cdi generate > /var/run/cdi/management.nvidia.com-gpu.yaml", which successfully creates the correct yaml file with the GPU
2 - Change /var/run/cdi/management.nvidia.com-gpu.yaml and "/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml", so that they use "nvidia.com/gpu"

@manuelbuil - do I need to restart the RKE service or containerd after this? Doesn't seem to change anything for me after this.

It should work automatically, is it not working?

[manuelbuil]

When using v25.10.0, as cdi is default, I am seeing again this issue, but now with the default deployment

The container-toolkit incorrectly generates /var/run/cdi/management.nvidia.com-gpu.yaml and the result does not include my GPU.

I tried to run nvidia-ctk cdi generate inside the nvidia-container-toolkit pod (since it is there by default) and, after creating a couple of links to libraries to make it execute, I can see that it generates a correct cdi yaml.

So after that, I started comparing the code of how nvidia-ctk and nvidia-toolkit generate the cdi yaml. Note that nvidia-toolkit is just a soft link to nvidia-ctk-installer. I found two notable differences:

1 - nvcdi mode:

nvidia-ctk uses mode: auto which ends up using nvml

nvidia-toolkit / nvidia-ctk-installer uses mode: management

2 - nvcdi deviceNamer

nvidia-ctk passes the option deviceNamers

nvidia-toolkit / nvidia-ctk-installer does not pass that option

If I change the code of nvidia-toolkit / nvidia-ctk-installer and use nvml mode and pass an empty deviceNamer, it correctly generates the yaml and I can see the GPU there:

    - name: GPU-3185fb51-f984-bdd1-824f-de25e6deb592
      containerEdits:
        deviceNodes:
            - path: /dev/nvidia0
              hostPath: /dev/nvidia0
            - path: /dev/dri/card0
              hostPath: /dev/dri/card0
            - path: /dev/dri/renderD128
              hostPath: /dev/dri/renderD128
        hooks:
            - hookName: createContainer
              path: /usr/local/nvidia/toolkit/nvidia-cdi-hook
              args:
                - nvidia-cdi-hook
                - create-symlinks
                - --link
                - ../card0::/host/dev/dri/by-path/pci-0000:00:1e.0-card
                - --link
                - ../renderD128::/host/dev/dri/by-path/pci-0000:00:1e.0-render
              env:
                - NVIDIA_CTK_DEBUG=false

When the /var/run/cdi/management.nvidia.com-gpu.yaml is correctly generated, the GPUs get correctly assigned to the pods.

Any idea what we might be doing differently to require such changes for the cdi yaml to be correctly generated?

[cdesiniotis]

Hi @manuelbuil, it is expected that /var/run/cdi/management.nvidia.com-gpu.yaml only contains a single device named management.nvidia.com/gpu=all. It is NOT expected for this file to have named entries for each GPU on your node.

Am I understanding it correct that all of the pods in the gpu-operator namespace are in the running state, but you are encountering issues when running a pod with the nvida-cdi runtime class? Note, with GPU Operator v25.10.0, we use native-CDI support in containerd by default, so you should not need to specify this runtime class in your pod spec. What happens when you remove the runtimeClassName setting altogether?

[w7team]

We encountered the same issue with this model of graphics card, Tesla V100-SXM3-32GB: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-59ab1128-3130-b95a-9870-4bd748926e97: unknown

[manuelbuil]

Hi @manuelbuil, it is expected that /var/run/cdi/management.nvidia.com-gpu.yaml only contains a single device named management.nvidia.com/gpu=all. It is NOT expected for this file to have named entries for each GPU on your node.

Am I understanding it correct that all of the pods in the gpu-operator namespace are in the running state, but you are encountering issues when running a pod with the nvida-cdi runtime class? Note, with GPU Operator v25.10.0, we use native-CDI support in containerd by default, so you should not need to specify this runtime class in your pod spec. What happens when you remove the runtimeClassName setting altogether?

Thanks for the help! When removing the runtimeClassName things worked again :)