unable to create new device filters program: load program: invalid argument: last insn is not an exit or jmp #1193
Description
Getting
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: failed to add device rules: unable to generate new device filter program from existing programs: unable to create new device filters program: load program: invalid argument: last insn is not an exit or jmp
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0: unknown.on Linux 6.6 with or without sudo (so kernel.unprivileged_bpf_disabled = 1 shouldn't matter).
Relevant pieces are up to date:
$ pacman -Qi docker nvidia-container-toolkit
Name : docker
Version : 1:28.3.2-1
Description : Pack, ship and run any application as a lightweight container
Architecture : x86_64
URL : https://www.docker.com/
Licenses : Apache-2.0
Groups : None
Provides : None
Depends On : glibc iproute2 device-mapper sqlite systemd-libs libseccomp libtool runc containerd
Optional Deps : btrfs-progs: btrfs backend support [installed]
pigz: parallel gzip compressor support [installed]
docker-buildx: extended build capabilities [installed]
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 96.61 MiB
Packager : Lukas Fleischer <[email protected]>
Build Date : Sat 12 Jul 2025 08:00:56 AM EDT
Install Date : Sun 13 Jul 2025 12:16:37 PM EDT
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
Name : nvidia-container-toolkit
Version : 1.17.8-1
Description : NVIDIA container toolkit
Architecture : x86_64
URL : https://github.com/NVIDIA/nvidia-container-toolkit
Licenses : Apache-2.0
Groups : None
Provides : None
Depends On : glibc libnvidia-container=1.17.8
Optional Deps : None
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 36.15 MiB
Packager : Jakub Klinkovský <[email protected]>
Build Date : Sat 31 May 2025 04:40:07 PM EDT
Install Date : Mon 30 Jun 2025 01:50:43 PM EDT
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
daemon.json has
"runtimes": {
"nvidia": {
"args": [],
"path": "nvidia-container-runtime"
}
},and host CUDA is at 12.8 (doing mistral.rs things and relevant candle PR is still in the queue to enable 12.9).
Podman appears to work with the Nvidia CDI so it's something specific to the Docker modality.