FabricManager doesn't get started correctly (and no error handling)

[cfstras]

Describe the bug

It seems that the NVIDIA fabricmanager doesn't get started correctly, merely due to an invalid option being passed?

Here's a snippet from the logs of nvidia-gpu-operator/nvidia-driver-daemonset-5.15.0-151-generic-ubuntu22.04-dcl4t:

Starting NVIDIA fabric manager daemon for NVLink5+...
Unknown option: /usr/share/nvidia/nvswitch/fabricmanager.cfg
NVIDIA Fabric Manager Start Script
Warning: If you are not a fabric manager developer, please avoid running this script directly.
Please either start or stop using the nvidia-fabricmanager systemd service,
or refer to the fabric manager user guide for how to run the fabric manager program.

Usage: /usr/bin/nvidia-fabricmanager-start.sh [script options] --args [arguments to run fabric manager]
Script options:
  -h, --help              Show this help message
  -d, --debug             Enable debug output for this script
  -m, --mode              Mode to run this script (start|stop|interactive)
  --fm-config-file        Path to fabric manager configuration file
  --fm-pid-file           Path to fabric manager PID file
  --nvlsm-config-file     Path to NVLink Subnet Manager configuration file
  --nvlsm-pid-file        Path to NVLink Subnet Manager PID file
  --args                  Trailing arguments to run fabric manager
Mounting NVIDIA driver rootfs...
Done, now waiting for signal

To Reproduce
Deploy the GPU Operator on k3s on an NVLink system.

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: nvidia-gpu-operator
  namespace: default
spec:
  chart: gpu-operator
  repo: https://helm.ngc.nvidia.com/nvidia
  version: v25.3.2
  targetNamespace: nvidia-gpu-operator
  createNamespace: true
  valuesContent: |-
    cdi:
      enabled: true
      default: false
    driver:
      enabled: true
      usePrecompiled: true
      version: 570

    mig:
      strategy: single
    toolkit:
      enabled: false

Expected behavior

• Fabricmanager starts up and configures the system.
• Or, there's at least an error (bash -euo pipefail etc)

Instead, I just get CUDA errors from my containers.

I've verified this was the issue by running nvidia-fabricmanager outside k3s:

apt install nvidia-container-toolkit nvidia-container-runtime nvidia-fabricmanager
systemctl enable --now nvidia-fabricmanager

After that, my workloads started to run normally.

Environment (please provide the following information):

• GPU Operator Version: v25.3.2
• OS: Ubuntu 22.04 LTS
• Kernel Version: 5.15.0-151-generic
• Container Runtime Version:

  # k3s ctr version
  Client:
    Version:  v2.0.5-k3s2
    Revision:
    Go version: go1.24.4
  
  Server:
    Version:  v2.0.5-k3s2
    Revision:
    UUID: 8acbe27a-7e75-4ca6-98d2-e669fb7499b2
  

• Kubernetes Distro and Version: k3s

  # k3s --version
  k3s version v1.33.3+k3s1 (236cbf25)
  go version go1.24.4
  

Information to attach (optional if deemed irrelevant)

• kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE
• kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE
• If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME
• If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers
• Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi
• containerd logs journalctl -u containerd > containerd.log

scratch_26.txt

[tariq1890]

Thanks for raising this issue.

Fabricmanager starts up and configures the system.

Unfortunately, this is a known issue with the 570.158.01 driver container on B200s, please use the 570.172.08 driver container instead.

Or, there's at least an error (bash -euo pipefail etc)

Yes, we will look into adding this improvement in the future driver container releases

[cfstras]

@tariq1890 thanks for the quick response, I'll try that version :)

Is there a listing of the driver versions somewhere? Just wondering why I have to set 570.172.08 explicitly, why it's not automatically picked up by specifying 570.

As we're getting into this I keep wanting to look at the source to troubleshoot things like this more accurately -- but I can't seem to find the code for the container dockerfiles, or most of the components deployed by the operator? Is there a list of links somewhere?
Much appreciated 😊

[tariq1890]

Is there a listing of the driver versions somewhere? Just wondering why I have to set 570.172.08 explicitly, why it's not automatically picked up by specifying 570.

I see that you're using the precompiled drivers. In the case of precompiled driver containers, we don't have any way of determining the exact the driver version included in the precompiled driver package we download from Canonical's package repositories. I have opened this PR to fix the issue you're seeing. Once that PR is merged, precompiled driver containers in the future will have that fix. You can consume that fix through a kernel upgrade thereafter

[tariq1890]

The sources for your Ubuntu 22.04 precompiled driver container can be found here

[cfstras]

Thanks for your explanations and the quick fix @tariq1890!

I couldn't pull the precompiled 570.172.08 driver (setting the version causes Back-off pulling image "nvcr.io/nvidia/driver:570.172.08-5.15.0-151-generic-ubuntu22.04"
And indeed I can see 570-5.15.0-151-generic-ubuntu22.04 at https://catalog.ngc.nvidia.com/orgs/nvidia/containers/driver/tags and 570.172.08-ubuntu22.04 but nothing with 570.172.08-5.*.

I've switched to non-precompiled drivers now and it seems to be working!

driver:
    version: 570.172.08
    usePrecompiled: false

[shivakunv]

Path is incorrect for precompiled driver image:
set the driver version to 570 for precompiled image.
http://nvcr.io/nvidia/driver:570-5.15.0-152-generic-ubuntu22.04 ( latest precompiled image)

[just1not2]

Would it be possible to expose this driver version as an apt package? When querying (on arm ubuntu22.04 here) the available versions we only get 570.158.01 and 570.133.20:

$ apt policy nvidia-driver-570-server
nvidia-driver-570-server:
  Installed: (none)
  Candidate: 570.158.01-0ubuntu0.22.04.1
  Version table:
     570.158.01-0ubuntu0.22.04.1 500
        500 http://us-east-1.ec2.ports.ubuntu.com/ubuntu-ports jammy-updates/restricted arm64 Packages
     570.133.20-0ubuntu0.22.04.1 500
        500 http://ports.ubuntu.com/ubuntu-ports jammy-security/restricted arm64 Packages

[cdesiniotis]

Closing this bug as a fix was implemented in our gpu-driver-container: NVIDIA/gpu-driver-container#412