docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed.

[praxi-roshan]

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed

Machine Specs & Environment Setup
OS : Ubuntu 24.04 LTS
NVIDIA Driver : 550.90.07
CUDA Version : 12.4
GPU Model : A6000
Docker Version : 24.0.7, build afdd53b

Steps of Setting up the Nvidia Container Toolkit
I followed the steps mentioned in this documentation: Installing the Toolkit

Installing with Apt

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit

Configuring Docker

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

Issue 2:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown.

Caused by: sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Notes:
Tried to run the above command after starting the nvidia persistenced mode by sudo nvidia-smi -pm ENABLED command, and that didn't resolved it, also tried sudo nvidia-persistenced to start it as a daemon srevice

Then finally tried unload and reload of the NVIDIA drivers:

sudo systemctl stop gdm

sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia

sudo modprobe nvidia
sudo modprobe nvidia_modeset
sudo modprobe nvidia_drm
sudo modprobe nvidia_uvm

sudo systemctl start gdm

sudo systemctl restart docker

After reloading tried executing the same docker command and now i'm getting a different error mentioned below.

Issue 2:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: could not apply required modification to OCI specification: error modifying OCI spec: failed to inject CDI devices: failed to inject devices: failed to stat CDI host device "/dev/dri/card1": no such file or directory: unknown.

Caused by:
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Notes:
This issue originated after unloading and reloading of the NVIDIA drivers in the attempt to fix the Issue 1 (which is still not resolved).

Special Notes
nvidia-persistenced not starting when attempted to start as in the following command sudo nvidia-persistenced throwing an error message:

nvidia-persistenced failed to initialize. Check syslog for more details.

After investigating the syslogs found these logs:

sudo cat /var/log/syslog | grep nvidia-persistenced
2024-09-03T20:57:06.322985+00:00 toro docker.nvidia-container-toolkit[1787337]: time="2024-09-03T20:57:06Z" level=info msg="Selecting /run/nvidia-persistenced/socket as /run/nvidia-persistenced/socket"
2024-09-03T20:57:06.324024+00:00 toro docker.nvidia-container-toolkit[1787337]: time="2024-09-03T20:57:06Z" level=info msg="Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-persistenced as /var/lib/snapd/hostfs/usr/bin/nvidia-persistenced"
2024-09-03T20:57:06.657932+00:00 toro docker.nvidia-container-toolkit[1787496]: time="2024-09-03T20:57:06Z" level=info msg="Selecting /run/nvidia-persistenced/socket as /run/nvidia-persistenced/socket"
2024-09-03T20:57:06.658053+00:00 toro docker.nvidia-container-toolkit[1787496]: time="2024-09-03T20:57:06Z" level=info msg="Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-persistenced as /var/lib/snapd/hostfs/usr/bin/nvidia-persistenced"
2024-09-06T11:39:16.670934+00:00 toro nvidia-persistenced: device 0000:01:00.0 - persistence mode disabled.
2024-09-06T11:39:16.671113+00:00 toro nvidia-persistenced: device 0000:01:00.0 - NUMA memory offlined.
2024-09-06T11:39:24.050313+00:00 toro nvidia-persistenced: device 0000:01:00.0 - persistence mode enabled.
2024-09-06T11:39:24.050493+00:00 toro nvidia-persistenced: device 0000:01:00.0 - NUMA memory onlined.
2024-09-06T11:59:14.012734+00:00 toro nvidia-persistenced: Failed to lock PID file: Resource temporarily unavailable
2024-09-06T11:59:14.012804+00:00 toro nvidia-persistenced: Shutdown (1902356)
2024-09-06T11:59:19.584467+00:00 toro nvidia-persistenced: Failed to lock PID file: Resource temporarily unavailable
2024-09-06T11:59:19.584576+00:00 toro nvidia-persistenced: Shutdown (1902361)
2024-09-06T11:59:59.116947+00:00 toro nvidia-persistenced: Failed to lock PID file: Resource temporarily unavailable
2024-09-06T11:59:59.117104+00:00 toro nvidia-persistenced: Shutdown (1902386)
2024-09-06T12:00:43.320943+00:00 toro nvidia-persistenced: device 0000:01:00.0 - persistence mode disabled.
2024-09-06T12:00:43.321232+00:00 toro nvidia-persistenced: device 0000:01:00.0 - NUMA memory offlined.
2024-09-06T12:00:46.881181+00:00 toro nvidia-persistenced: Failed to lock PID file: Resource temporarily unavailable
2024-09-06T12:00:46.881291+00:00 toro nvidia-persistenced: Shutdown (1902413)
2024-09-06T12:01:13.921359+00:00 toro nvidia-persistenced: Failed to lock PID file: Resource temporarily unavailable
2024-09-06T12:01:13.921592+00:00 toro nvidia-persistenced: Shutdown (1902426)

I'm not sure if this is the root cause for the issues I'm experiencing, hope these information will helps to troubleshoot and provide a solution.

[elezar]

Thanks for reporting this @praxi-roshan

First some questions:

• What is the output of nvidia-smi run on your system?
• Which version of the nvidia-container-toolkit are you using? (nvidia-container-toolkit --version)
• How is docker installed? (e.g. Is it installed using snap?)

[praxi-roshan]

Hi @elezar,

We installed the docker based on the instructions mentioned here: Install using the apt repository,

$nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H100 PCIe Off | 00000000:01:00.0 Off | 0 |
| N/A 26C P0 48W / 350W | 4MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

$ nvidia-container-toolkit --version
NVIDIA Container Runtime Hook version 1.16.1
commit: a470818

[xieleo5]

Got a similar error here:

$ docker run --gpus all hello-world
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: change root failed: no such file or directory: unknown.

nvidia-smi output:

$ nvidia-smi
Wed Oct  9 00:36:13 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA RTX A5000               On  | 00000000:01:00.0 Off |                  Off |
| 30%   25C    P8              24W / 230W |      7MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A5000               On  | 00000000:23:00.0 Off |                  Off |
| 30%   24C    P8              24W / 230W |      7MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA RTX A5000               On  | 00000000:41:00.0 Off |                  Off |
| 30%   25C    P8              12W / 230W |   1239MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA RTX A5000               On  | 00000000:61:00.0 Off |                  Off |
| 30%   25C    P8              17W / 230W |      6MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   4  NVIDIA RTX A5000               On  | 00000000:81:00.0 Off |                  Off |
| 35%   64C    P2             216W / 230W |  18092MiB / 24564MiB |     93%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   5  NVIDIA RTX A5000               On  | 00000000:A1:00.0 Off |                  Off |
| 34%   64C    P2             211W / 230W |  17995MiB / 24564MiB |     81%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   6  NVIDIA RTX A5000               On  | 00000000:C1:00.0 Off |                  Off |
| 30%   61C    P2             214W / 230W |  17973MiB / 24564MiB |     97%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   7  NVIDIA RTX A5000               On  | 00000000:E1:00.0 Off |                  Off |
| 33%   62C    P2             214W / 230W |  21371MiB / 24564MiB |     98%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

[F1zzyD]

I've got the same issue running on headless Ubuntu. Crazy

[asadchev]

Try removing snapd package,
sudo apt remove snapd

[Eir-nya]

I'm having the same issue on my headless server box. nvidia-container-toolkit was working fine for the past few months and then as soon as I did an apt upgrade, I've been running into this. And nvidia-persistenced gives an error message too.

nvidia-smi output:

Sat Dec 21 20:27:37 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1070        On  |   00000000:01:00.0  On |                  N/A |
|  0%   41C    P8              6W /  151W |       9MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

nvidia-container-toolkit --version

NVIDIA Container Runtime Hook version 1.17.3
commit: cb82e29c75d387992bf59eb6eadf5d96cb6d4747

Edit: Downgrading nvidia-toolkit-container-base and nvidia-toolkit-container to version 1.15.0-1 fixed the issue for me

[YantingChi]

I am faced with the same problem and I solve this problem by reinstalling docker without using snap.

When I try to run a container with docker run -it --entrypoint "/bin/bash" --rm --runtime=nvidia --gpus "device=0" ubuntu, and try to find if nvidia-smi appears anywhere in the container, it shows that nvidia-smi is actually mounted into the container's snap folder, which definitely is not in the container's $PATH

Then I just remove the docker installed by snap and run the /usr/bin/docker to start the same container, nvidia-smi works now.

[pjgoodall]

Same Ubuntu 24.04.2 LTS
Built and ran container using GPU. Would not run again. Could not compose again. Doing exercises from DDD book.
So its likely that compose file is correct.
Docker and docker desktop installed from docker repository:

❯ nvidia-smi
Fri Mar 21 16:41:40 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1050 Ti     Off |   00000000:01:00.0  On |                  N/A |
| 44%   39C    P8             N/A /   75W |     531MiB /   4096MiB |     15%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      6264      G   /usr/lib/xorg/Xorg                            242MiB |
|    0   N/A  N/A      6561      G   /usr/bin/gnome-shell                           74MiB |
|    0   N/A  N/A      7706      G   ...erProcess --variations-seed-version        137MiB |
|    0   N/A  N/A     10184      G   ...seed-version=20250320-180804.471000         68MiB |
|    0   N/A  N/A     10869      G   /usr/bin/x-terminal-emulator                    2MiB |
+-----------------------------------------------------------------------------------------+

❯ nvidia-container-toolkit --version
NVIDIA Container Runtime Hook version 1.17.5
commit: f785e908a7f72149f8912617058644fd84e38cde

[tkircher]

I'm having the same issue, Ubuntu 24.04, toolkit version 1.17.6. Had it working fine a year ago on this machine, after upgrading a few times I went to run it again, and neither nvidia-smi, or any CUDA libraries, exist in any of the docker images. Tried purging everything and reinstalling, newest NVidia driver, tried several different nvidia docker images with different versions, and it just plain doesn't work.

[hdnh2006]

I solved it adding the flag --runtime nvidia.

Example:

docker run --runtime nvidia --gpus all -it --ipc=host --network="host"  yourcontainer /bin/bash

[FahYen]

if youre running run_dev.sh on the jetson, you have to add the flag "--gpus all -it" to "docker run"

[taikinakano0722]

I've resolved this issue. Here are the steps that worked for me:

1. Check the group ownership of the GPU device files.
   Run the following command to identify the group that has access to the /dev/nvidia* files.
   ls -l /dev/nvidia*
   In my case, the group was vglusers. I will refer to this as the 'group name' below.

2. Edit the NVIDIA Container Runtime configuration.
   Open the config file with root privileges:
   sudo vi /etc/nvidia-container-runtime/config.toml
   Find the line #user = "...", uncomment it (remove the #), and set the value to root:[group name]. For example:
   user = "root:vglusers"

3. Restart the Docker daemon.
   Apply the changes by restarting Docker:
   sudo systemctl restart docker

After these steps, the container started successfully. I hope this helps others.