Upgrade to 25.10.0 - Pod with GPU will not start

[xhejtman]

Describe the bug

After upgrading to 25.10.0 from 25.3.2, I get error:
CrashLoopBackOff (back-off 40s restarting failed container=hello-kubernetes pod=hello-kubernetes-69575f56b-9dzz4_test-ns(5e20c659-44d1-4c22-9d8f-560e3411fc58)) | Last state: Terminated with 128: StartError (failed to create containerd task: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: unresolvable CDI devices management.nvidia.com/gpu=GPU-feefc289-ca5f-9917-2bf1-9477651da944), started: Thu, Jan 1 1970 1:00:00 am, finished: Sun, Nov 9 2025 12:34:53 am

Probably because CDI is not default and if I set runtimeClassName: nvidia-cdi it fails in the above way.

If I omit runtimeClassName, it starts and I can see nvidia devices in /dev, but there are no injected userspace files like nvidia-smi and libraries.

Do I need to change something in the operator configuration?

I use for toolkit:

 - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS
   value: "true"
 - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
   value: "false"

and for the devicePlugin:

env:
    - name: PASS_DEVICE_SPECS
      value: "true"
    - name: FAIL_ON_INIT_ERROR
      value: "true"
    - name: DEVICE_LIST_STRATEGY
      value: volume-mounts
    - name: DEVICE_ID_STRATEGY
      value: uuid
    - name: NVIDIA_VISIBLE_DEVICES
      value: all
    - name: NVIDIA_DRIVER_CAPABILITIES
      value: all

Environment (please provide the following information):

• GPU Operator Version: v25.10.0
• OS: Ubuntu 24.04
• Kernel Version: 6.8.0-generic
• Container Runtime Version: containerd 2.1.4
• Kubernetes Distro and Version: RKE 1.33.5

[regdude]

I faced a similar issue, but I doubt that your issue is the same one.
Due to unrelated reasons I had to reinstall my RKE2 cluster and running the same Helm chart it did not want to recognize "runtimeClassName: nvidia", kept getting an error when starting a Pod:

task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: nvidia-container-cli.real: device error: /var/run/nvidia-container-devices: unknown device

After many attempts to change the Helm chart and some other changes, I ended up with this:

    - name: Run nvidia-ctk to configure containerd runtime
      command: >
        nvidia-ctk runtime configure
        --runtime=containerd
        --config=/var/lib/rancher/rke2/agent/etc/containerd/config.toml

    - name: Restart RKE2 server to apply containerd changes
      systemd:
        name: rke2-server
        state: restarted
        daemon_reload: yes

    - name: Install NVIDIA GPU Operator via Helm
      kubernetes.core.helm:
        name: gpu-operator
        chart_ref: nvidia/gpu-operator
        #chart_version: v25.3.2
        release_namespace: gpu-operator
        create_namespace: true
        wait: yes
        values:
          cdi:
            enabled: false
          driver:
            enabled: false
          toolkit:
            enabled: false
            env:
              - name: CONTAINERD_CONFIG
                value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml
              - name: CONTAINERD_SOCKET
                value: /run/k3s/containerd/containerd.sock
              - name: CONTAINERD_RUNTIME_CLASS
                value: nvidia
              - name: CONTAINERD_SET_AS_DEFAULT
                value: "true"
              #- name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
               # value: "false"
              #- name: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS
              #  value: "true"
          #devicePlugin:
          #  env:
          #  - name: DEVICE_LIST_STRATEGY
          #    value: volume-mounts

I might have did this too before running the Helm chart:

rm -rf /etc/cdi/nvidia.yaml

Maybe something got broken because "cdi.enabled" was not set to true by default and RKE2 does not support CDI yet.
Also I did had to comment out a few env flags that the RKE2 documentation said to add, meanwhile SUSE documentation did not mention the same flags.
Maybe this is unrelated and I had messed up my own config at some point.

[xhejtman]

I do not know, if the toolkit is sill used in the CDI mode and if the toolkit should inject libraries and nvidia tools into container. I am able to configure the operator and nodes so that a Pod starts with GPU attached, but without libraries and tools.

I also noticed, that this is already happening with 25.3.4 version. Version 25.3.2 is ok.

[xhejtman]

The problem here is indeed with:

  - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS
    value: "true"
  - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
    value: "false"

If I remove these envs, everything works as expected. Esp. the device plugin has:

- name: DEVICE_LIST_STRATEGY
  value: cdi-annotations,cdi-cri

however, NVIDIA_VISIBLE_DEVICES=all is considered when a pod sets runtimeClassName: nvidia, privileged container is not needed which is unfortunate.

But is seems, that if I keep:

- name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
  value: "false"

for nvidia-toolkit, it works as expected, only privileged containers can claim GPU via NVIDIA_VISIBLE_DEVICES (when set the nvidia runtimeclass).

maybe worth of adding to docs.

[cdesiniotis]

Hi @xhejtman can you clarify how your pods are requesting access to GPUs? Based on your comments, it appears as if you are bypassing the device-plugin and using the NVIDIA_VISIBLE_DEVICES envvar? If true, you would need to set runtimeClassNamae=nvidia in your pod spec -- as starting in GPU Operator 25.10.0 we no longer configure the nvidia runtime as the default runtime.

I am removing the bug label.

[xhejtman]

For context, up to version 25.3.2 (inclusive), probably many users set:

ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS=true
ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED=false

These settings enforce that a pod must request a GPU through the nvidia.com/gpu resource when it is not privileged; relying on the NVIDIA_VISIBLE_DEVICES environment variable alone is insufficient.
In addition, the default runtime was typically set to nvidia.

• If you keep all this and upgrade to 25.10.0, then starting Pod that requests nvidia.com/gpu ends with OCI error as above.
• If you remove default nvidia runtime settings from containerd config, Pod with nvidia.com/gpu starts, but there are no userspace apps/libraries (nvidia-smi, etc.).
• If you remove ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS=true, then everything works as expected (but ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED=false should be kept).

Also, slight change in how to request gpu only via env var:

• needs to set runtime to nvidia
• pod must be privileged
• pod must export NVIDIA_VISIBLE_DEVICES

so, all this is not a bug, it just deserved a bit better release notes and changes in behavior.

[cdesiniotis]

Thanks for the details @xhejtman, you're feedback is valuable. Agreed that our documentation on this topic (preventing unprivileged access to GPUs via the NVIDIA_VISIBLE_DEVICES envvar) can be improved. Now that 25.10.0 defaults to using native-CDI support in runtimes, the needed configuration may be slightly different. We will look into this and get back to you on the recommended configuration.