nvidia-ctk requires run after kernel/akmod update

[ahmed-alhindawi]

Hello,

When working with podman on fedora and using nvidia GPU using the CDI backend, everytime the kernel is updated or the driver is updated, nvidia-ctk cdi generate ... to be required to be re-run for the CDI container to find the appropriate libraries.

Is this a fedora issue or would it be possible to have a hook to run this command automatically once the kernel module is rebuilt?

[elezar]

@ahmed-alhindawi this is a known issue. We are working on improving the UX around this.

[ahmed-alhindawi]

Thank you

[billtown]

I have to run it after every reboot on a fresh fedora 41. I'm uncertain of this whole flow tbh. Or what I'm doing wrong. I'm missing /dev/nvidia-uvm until I "generate" below.

I just run both of these even though the files persist.
nvidia-ctk config default > /etc/nvidia-container-runtime/config.toml
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

[iamfil]

I created a systemd unit to do this on each boot for my Fedora CoreOS host. I know it's overkill, but it keeps me from forgetting to run it.

/etc/systemd/system/nvidia-toolkit-cdi.service

[Unit]
Description=Run nvidia-ctk cdi

[Service]
Type=simple
ExecStart=nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml

[Install]
WantedBy=multi-user.target

[westurner]

I must run this with Fedora Silverblue 42 (rpm-ostree) to update the CDI definitions YAML in nvidia.yaml. I haven't needed to modify config.toml.

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Aren't there hooks in akmod for this; isn't that what akmod is for?

Standard RPMs have e.g. %install and %post for commands to run.

• docs: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-yum-or-dnf
  • https://podman-desktop.io/docs/podman/gpu#version-mismatch describes this problem and the solution
• RPMfusion packages:
  • Akmods docs: https://rpmfusion.org/Packaging/KernelModules/Akmods
  • https://rpmfusion.org/Howto/NVIDIA#OSTree_.28Silverblue.2FKinoite.2Fetc.29
    • docs: add a section on CDI/CTK
    • This is an example of how to run something every time the kernel modules are updated IIUC:
      • https://github.com/CheariX/silverblue-akmods-keys
        • https://github.com/CheariX/silverblue-akmods-keys/blob/d8da70db94a78759d0ceca1829eb825c0efd002e/akmods-keys.spec#L30-L37
      • (https://github.com/ublue-os/akmods is an alternative to locally-signing)
• create a variable in the .spec for the path to the nvidia.yaml CDI definitions
  if the path varies on other distros?
  • Fedora w/ RPMFusion packages:
    /etc/cdi/nvidia.yml defines device and volume mounts for containers that request --gpus
• call nvidia-ctk cdi generate in %install or %post ?
  • Is this the .spec file for this?
    • https://github.com/rpmfusion/nvidia-kmod/blob/master/nvidia-kmod.spec
    • https://github.com/rpmfusion/nvidia-kmod/blob/42dbd49de41784a1bf9965d87412c8efff24a52d/nvidia-kmod.spec#L89 %install

[billtown]

EDIT: apologies. I'm discussing every boot :/ I've not had issues with akmod or updates that I'm aware.

actually, you can just call nvidia-smi once. It'll load it all. something isn't initiated until then. At least that's what I eventually discovered. So I too made a systemd run once thing for nvidia-smi. on my workstation I don't care, but on my compute I installed a systemd process.

/etc/systemd/system/nvidia-init.service
`
[Unit]
Description=Initialize NVIDIA GPU (run nvidia-smi)
After=systemd-modules-load.service
Before=podman.service
Wants=podman.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi

[Install]
WantedBy=multi-user.target
`

[westurner]

Running nvidia-smi on boot to multi-user runlevel updates /etc/cdi/nvidia.yaml and/or /var/run/cdi/nvidia.yaml?

Will try to remember to test this on the next kmod package update

[billtown]

No, it just initializes things. Needed on every boot for me. I have a problem with every boot not updates....

…

On Sun, Jun 22, 2025, 7:16 PM Wes Turner ***@***.***> wrote: *westurner* left a comment (NVIDIA/nvidia-container-toolkit#856) <#856 (comment)> Running nvidia-smi on boot to multi-user runlevel updates /etc/cdi/nvidia.yaml and/or /var/run/cdi/nvidia.yaml? Will try to remember to test this on the next kmod package update — Reply to this email directly, view it on GitHub <#856 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB7FRLKW3QZKXHIGL3TBPHT3E42OVAVCNFSM6AAAAABU57NHD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSOJUGUYTKNZQGA> . You are receiving this because you commented.Message ID: ***@***.***>

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[westurner]

Is this still an issue?

…

On Tue, Nov 4, 2025, 11:34 PM github-actions[bot] ***@***.***> wrote: *github-actions[bot]* left a comment (NVIDIA/nvidia-container-toolkit#856) <#856 (comment)> This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label. — Reply to this email directly, view it on GitHub <#856 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMNS6KNGY7ECOCWYOF42333F447AVCNFSM6AAAAABU57NHD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIOBZGI3TINRXGY> . You are receiving this because you commented.Message ID: ***@***.***>

[ahmed-alhindawi]

yup, i just run:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Whenever the kernel or the driver are updated....It's annoying but at least it works.

[elezar]

(removing the lifecycle/stale label).

The v1.18.0 release of the NVIDIA Container Toolkit includes an nvidia-cdi-refresh.service that should ensure that CDI specs are regenerated on driver installations / kernel upgrades. If this is not working as expected for your scenario, please provide more information.

Note that we are already aware of some issues such as #1395 around this which will be addressed in an upcoming patch release.