Does Talos v1.11 have drivers for Nvidia ConnectX-7 Adapter Cards?

[tan-wei-xin-alez]

Hello, I see that Talos has a mellanox-ofed-pkg, is that the driver for the NVIDIA ConnectX-7 Adapter Cards seen below?

➜  ~ kubectl run test-pod -it --rm --restart=Never --image nvcr.io/nvidia/cuda:12.5.0-base-ubuntu22.04 --overrides '{"spec": {"runtimeClassName": "nvidia"}}' -- bash
...
If you don't see a command prompt, try pressing enter.
root@test-pod:/# apt-get update && apt-get install -y pciutils
...
root@perception-train-test-alez:/# lspci | grep mellanox -i
03:00.0 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]
03:00.1 Infiniband controller: Mellanox Technologies MT2910 Family [ConnectX-7]
11:00.0 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]
11:00.1 Ethernet controller: Mellanox Technologies MT2910 Family [ConnectX-7]

I got the above from NVIDIA ConnectX-7 Adapter Cards User Manual > Linux Driver Installation but I do not know if that means the Talos v1.11 installed on my node already has the drivers or if I need some other system extension to have the drivers or if I need to figure out how to follow the instructions at NVIDIA Networking > BlueField DPUs / SuperNICs & DOCA > DOCA Documentation v3.1.0 Core Update > BF-Bundle Installation and Upgrade to get them

[rothgar]

Do you have anything plugged into the connectx-7 card? Do they show up with talosctl get links? That would probably be the first place to check if the drivers are installed and the hardware is working.

If there are special kernel modules that need to be loaded you'll need to add those via machine config patches.

[tan-wei-xin-alez]

Do you have anything plugged into the connectx-7 card? Do they show up with talosctl get links? That would probably be the first place to check if the drivers are installed and the hardware is working.

ah ok, I'll go and plug something in, I'm getting the following with talosctl get links

NODE        NAMESPACE   TYPE         ID              VERSION   TYPE         KIND     HW ADDR                                                       OPER STATE   LINK STATE
10.5.0.81   network     LinkStatus   bond0           1         ether        bond     7a:14:56:7f:74:b9                                             down         false
10.5.0.81   network     LinkStatus   cni0            3         ether        bridge   8e:56:e8:06:14:11                                             up           true
10.5.0.81   network     LinkStatus   dummy0          1         ether        dummy    76:8f:9f:72:30:d5                                             down         false
10.5.0.81   network     LinkStatus   enp17s0f0np0    2         ether                 58:a2:e1:cd:d6:dc                                             down         false
10.5.0.81   network     LinkStatus   enp17s0f1np1    2         ether                 58:a2:e1:cd:d6:dd                                             down         false
10.5.0.81   network     LinkStatus   enp225s0f0np0   3         ether                 a0:ad:9f:25:37:21                                             up           true
10.5.0.81   network     LinkStatus   enp225s0f1np1   3         ether                 a0:ad:9f:25:37:22                                             up           true
10.5.0.81   network     LinkStatus   flannel.1       2         ether        vxlan    96:d0:70:50:08:2f                                             unknown      true
10.5.0.81   network     LinkStatus   ibs1f0          1         infiniband   ipoib    00:00:00:90:fe:80:00:00:00:00:00:00:58:a2:e1:03:00:e1:bc:cc   down         false
10.5.0.81   network     LinkStatus   ibs1f1          1         infiniband   ipoib    00:00:01:99:fe:80:00:00:00:00:00:00:58:a2:e1:03:00:e1:bc:cd   down         false
10.5.0.81   network     LinkStatus   ip6tnl0         1         tunnel6      ip6tnl   00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00               down         false
10.5.0.81   network     LinkStatus   lo              2         loopback              00:00:00:00:00:00                                             unknown      true
10.5.0.81   network     LinkStatus   sit0            1         sit          sit      00:00:00:00                                                   down         false
10.5.0.81   network     LinkStatus   teql0           1         void                                                                                down         false
10.5.0.81   network     LinkStatus   tunl0           1         ipip         ipip     00:00:00:00                                                   down         false
10.5.0.81   network     LinkStatus   veth5837436b    2         ether        veth     fe:66:74:8a:33:4e                                             up           true
10.5.0.81   network     LinkStatus   vetha1c8bfac    3         ether        veth     86:a4:49:b6:b2:5b                                             up           true

so I think it's pretty obvious that the ones with Type: infiniband belong to the cards

then I double checked the PCI vendor ID of the others and it seems as expected (i.e. enp17* ones are down while enpc225* ones are up) since the current connection is via the Intel NICs

➜  ~ for IF in enp17s0f0np0 enp17s0f1np1 enp225s0f0np0 enp225s0f1np1; do
  echo "== $IF =="
  talosctl -n 10.5.0.81 read /sys/class/net/$IF/device/uevent | grep PCI_SLOT_NAME
done
== enp17s0f0np0 ==
PCI_SLOT_NAME=0000:11:00.0
== enp17s0f1np1 ==
PCI_SLOT_NAME=0000:11:00.1
== enp225s0f0np0 ==
PCI_SLOT_NAME=0000:e1:00.0
== enp225s0f1np1 ==
PCI_SLOT_NAME=0000:e1:00.1
➜  ~ for bdf in 0000:11:00.0 0000:11:00.1 0000:e1:00.0 0000:e1:00.1; do
  echo "== $bdf =="
  talosctl -n 10.5.0.81 read /sys/bus/pci/devices/$bdf/uevent | grep DRIVER
done
== 0000:11:00.0 ==
DRIVER=mlx5_core
== 0000:11:00.1 ==
DRIVER=mlx5_core
== 0000:e1:00.0 ==
DRIVER=i40e
== 0000:e1:00.1 ==
DRIVER=i40e

If there are special kernel modules that need to be loaded you'll need to add those via machine config patches.

If I have to do this, would it be done the same way as Enabling the NVIDIA OSS modules? I didn't find any other documentation on what the other kernel modules are called besides the nvidia ones i.e. nvidia, nvidia_uvm, nvidia_drm and nvidia_modeset

Is it one of the mellanox ones in talos/hack/modules-amd64.txt? Like just the last substring containing *.ko after the slash?

[tan-wei-xin-alez]

Oh wait, the driver is already installed since I'm getting a result for grep DRIVER 🤦

and as expected, the same driver is being used for the infiniband interfaces (now I just need to test that the drivers actually work by plugging something in)

➜  ~ for IF in ibs1f0 ibs1f1; do
  echo "== $IF =="
  talosctl -n 10.5.0.81 read /sys/class/net/$IF/device/uevent | grep -E 'PCI_SLOT_NAME|DRIVER|PCI_ID'
done
== ibs1f0 ==
DRIVER=mlx5_core
PCI_ID=15B3:1021
PCI_SLOT_NAME=0000:03:00.0
== ibs1f1 ==
DRIVER=mlx5_core
PCI_ID=15B3:1021
PCI_SLOT_NAME=0000:03:00.1