VF VFIO device support #146

winsopc · 2025-09-17T19:31:18Z

Add both PF and VF VFIO mode support, which is needed by kubevirt support.

winsopc · 2025-09-18T16:28:03Z

@adrianchiris @SchSeba could you please review the changes?

SchSeba · 2025-09-28T11:13:22Z

Hi @adrianchiris will NVIDIA be able to check this one I don't have an env to test infiniband.

almaslennikov

@adrianchiris PTAL at this PR

cmd/ib-sriov-cni/main.go

pkg/utils/utils_test.go

cmd/ib-sriov-cni/main.go

pkg/sriov/sriov.go

almaslennikov

LGTM. @adrianchiris PTAL

adrianchiris · 2025-10-23T15:47:25Z

Will review on sunday. sorry for the delay.

winsopc · 2025-11-05T16:08:12Z

@adrianchiris PTAL

adrianchiris · 2025-11-09T15:20:36Z

cmd/ib-sriov-cni/main.go

+		return nil, nil, err
+	}
+
+	// If vfioPciMode is true (either set manually or auto-detected), skip SR-IOV setup


we do want the check related to netConf.RdmaIso

i would just skip the call for config.LoadDeviceInfo() if netConf.VfioPciMode is true

ack. I will update the PR.

pkg/utils/utils.go

cmd/ib-sriov-cni/main.go

adrianchiris · 2025-11-09T15:55:39Z

cmd/ib-sriov-cni/main.go

+	// Only handle GUID setting for VF devices, keep previous code path for PF devices
+	if isVF && netConf.GUID != "" {
+		// Load device info needed for GUID setting (VFIO VF version - no network interface)
+		err = config.LoadDeviceInfoVfioVF(netConf)


any reason why this should not be rolled into configLoadDeviceInfo which will handle both regular and vfio cases ?

yes, I should put them together, if it's VF only, many codes can be reused.

adrianchiris · 2025-11-09T15:58:58Z

cmd/ib-sriov-cni/main.go

-	if err != nil {
-		return err
+		sm := sriov.NewSriovManager()
+		if err := sm.ApplyVFConfig(netConf); err != nil {


this handles also applying linkState why do you call it conditionally if its a VF and GUID was provided ?

it's related to the PF related code path. I agree, this part need be optimized.
maybe I need submit a VF VFIO only change fist for this PR

pkg/sriov/sriov.go

adrianchiris · 2025-11-09T17:17:19Z

pkg/sriov/sriov.go

+	// For VFIO VF devices, we don't have a network interface, so skip VF link operations
+	if conf.VfioPciMode && conf.HostIFNames == "" {
+		// VFIO VF: Just set the GUID directly via PF, no VF link to query
+		return s.setVfGUID(conf, pfLink, conf.GUID)


this currently breaks the non-vfio use-case.

for both cases you need to set the node and port GUID and only for the non-vfio case you need to rebind the VF.

pkg/sriov/sriov.go

adrianchiris · 2025-11-09T17:24:43Z

pkg/sriov/sriov.go

+	// Validate that we have container interface name to work with
+	if conf.ContIFNames == "" {
+		// Silently skip cleanup if container interface name is missing
+		// This can happen with corrupted or incomplete cached configs


if cache config failed to load we exit early.

i really find it hard to follow this PR. IMO this PR should focus around VFIO support.

Agree. I will address this in next update.

Add VFIO mode support for VF devices, which is needed by kubevirt. Signed-off-by: Zhen(Winson) Wang <[email protected]>

winsopc · 2025-11-12T17:57:55Z

@adrianchiris thanks for your review comments. I update my PR with VF VFIO support only. PTAL when you have time.
If upstream agree PF support is needed, I can add PF support in a new PR.

kozmod · 2025-11-25T08:04:39Z

Hi, guys)
What about the progress of PR (@winsopc, @adrianchiris)?
our team need to this kind of functionality -> now we are creating VM using KubeVirt with Infinband interfaces (VF):

$ lspci -nnk | grep -A 3 -i mell
81:00.0 Infiniband controller [0207]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
	Subsystem: Hewlett Packard Enterprise MT27800 Family [ConnectX-5] [1590:0255]
	Kernel driver in use: mlx5_core # ⚠️
	Kernel modules: mlx5_core
81:00.1 Infiniband controller [0207]: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [15b3:1018]
	Subsystem: Hewlett Packard Enterprise MT27800 Family [ConnectX-5 Virtual Function] [1590:0255]
	Kernel driver in use: vfio-pci # ⚠️
	Kernel modules: mlx5_core
81:00.2 Infiniband controller [0207]: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [15b3:1018]
	Subsystem: Hewlett Packard Enterprise MT27800 Family [ConnectX-5 Virtual Function] [1590:0255]
	Kernel driver in use: vfio-pci  # ⚠️
	Kernel modules: mlx5_core

I have built and installed ib-sriov bin file from this branch (didn't look at the changes attentively) and tried to create VM (with UFM integration + Nvidia-networl-operator + Multus) -> GUID (Pkey Members) in the UFM creates normally...
... but I have found one problem: GUID didn't delete from UFM after deleting VM.

winsopc · 2025-11-26T04:09:18Z

Hi @kozmod , Thanks for validating my PR. This PR is pending @adrianchiris 's review for my latest changes.
For the stale guid issue on UFM, are you using ib-kubernetes ?
My understanding, ib-sriov-cni cannot control the UFM side, it's ib-kubernetes with ufm plugin.
https://github.com/Mellanox/ib-kubernetes/tree/master
@almaslennikov @adrianchiris can you guys comments on the guid UFM support?

kozmod · 2025-11-27T09:58:43Z

@adrianchiris , could you continue validate this PR, please?

winsopc mentioned this pull request Sep 18, 2025

Is VFIO supported? #53

Open

almaslennikov reviewed Oct 17, 2025

View reviewed changes

cmd/ib-sriov-cni/main.go Outdated Show resolved Hide resolved

pkg/utils/utils_test.go Outdated Show resolved Hide resolved

cmd/ib-sriov-cni/main.go Outdated Show resolved Hide resolved

pkg/sriov/sriov.go Outdated Show resolved Hide resolved

pkg/sriov/sriov.go Outdated Show resolved Hide resolved

winsopc force-pushed the vfio-support-upstream branch 2 times, most recently from cc34bd7 to 1210b64 Compare October 18, 2025 00:23

almaslennikov approved these changes Oct 23, 2025

View reviewed changes

adrianchiris reviewed Nov 9, 2025

View reviewed changes

feat: Add VF VFIO device support

6a69941

Add VFIO mode support for VF devices, which is needed by kubevirt. Signed-off-by: Zhen(Winson) Wang <[email protected]>

winsopc force-pushed the vfio-support-upstream branch from 1210b64 to 6a69941 Compare November 12, 2025 17:50

winsopc changed the title ~~VFIO device support~~ VF VFIO device support Nov 14, 2025

winsopc requested a review from adrianchiris November 19, 2025 18:54

VF VFIO device support #146

Are you sure you want to change the base?

VF VFIO device support #146

Uh oh!

Conversation

winsopc commented Sep 17, 2025

Uh oh!

winsopc commented Sep 18, 2025

Uh oh!

SchSeba commented Sep 28, 2025

Uh oh!

almaslennikov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

almaslennikov left a comment

Choose a reason for hiding this comment

Uh oh!

adrianchiris commented Oct 23, 2025

Uh oh!

winsopc commented Nov 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

winsopc commented Nov 12, 2025

Uh oh!

kozmod commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

winsopc commented Nov 26, 2025

Uh oh!

kozmod commented Nov 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kozmod commented Nov 25, 2025 •

edited

Loading