Qn - node-specific config support for VM Virtual Functions

[mshannongit]

@adrianchiris @nlbrown2 @SchSeba

Hi - apologies for raising this question as an issue - the NPWG slack invite link (from http://intel-corp.herokuapp.com/) is broken - and hence I cannot post this question on the slack channel

What issue would you like to bring attention to?

For our particular cloud-based k8s application, we have VM-based compute worker nodes that have may differing numbers of attached virtual network interfaces, and these VNICs potentially attached to different subnets/VLANs.

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin?tab=readme-ov-file#configurations

If we had compute instances with VNICs always at same PCI address for a particular network, we could do something like ..

e.g. subnetA always at 00:05.0 and 00:07.0
e.g. subnetB always at 00:06.0 and 00:08.0

Node0: VNIC1=subnetA, VNIC2=subnetB, VNIC3=subnetA, VNIC4=subnetB
Node1: VNIC1=subnetA, VNIC2=subnetB
Node2: VNIC1=subnetA, VNIC2=subnetB, VNIC3=subnetA, VNIC4=subnetB

# Node0/2

$ /usr/sbin/lspci -nn  | grep "Virtual Function"
...
00:05.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]
00:06.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]
00:07.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]
00:08.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]

# Node1
$ /usr/sbin/lspci -nn  | grep "Virtual Function"
...
00:05.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]
00:06.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "mlnx_sriov_subnetA",
                "resourcePrefix": "mellanox.com",
                "selectors": {
                    "vendors": ["15b3"],
                    "devices": ["101e"],
                    "drivers": ["mlx5_core"],
                    "pciAddresses": ["0000:00:05.0","0000:00:07.0"]
                }
            },
            {
                "resourceName": "mlnx_sriov_subnetB",
                "resourcePrefix": "mellanox.com",
                "selectors": {
                    "vendors": ["15b3"],
                    "devices": ["101e"],
                    "drivers": ["mlx5_core"],
                    "pciAddresses": ["0000:00:06.0","0000:00:08.0"]
                }
            }
        ]
    }

Unfortunately however, we cannot rely on VNICs for a particular subnet always using the same PCI addresses.

e.g.
Node0: VNIC1=subnetA, VNIC2=subnetA, VNIC3=subnetB
Node1: VNIC1=subnetB, VNIC2=subnetA, VNIC3=subnetC
Node2: VNIC1=subnetA
Node4: VNIC1=subnetC, VNIC2=subnetA

# e.g. for Node0 and Node1

$ /usr/sbin/lspci -nn  | grep "Virtual Function"
...
00:05.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]
00:06.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]
00:07.0 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e]

Is there an alternative approach to the configmap using the SriovNetworkNodePolicy CRD or equivalent to provide node specific configuration to the SR-IOV device plugin

Such that in case of above - specifying mlnx_sriov_subnetA is associated with Node0 at 00:05.0 and 00:06.0,
and mlnx_sriov_subnetA is associated with Node1 at 00:06.0

... and mlnx_sriov_subnetB on Node0 at 00:07.0, and on Node1 at 00:05.0

[adrianchiris]

Hi, you can define sriovnetworknodepolicy with node selector, either targeting a specific node or a specific group of nodes with the same vnic configuration.

An alternative approach is related to the vm infrastructure, where you bring up the vm with predictable pci config (pci address) where a vnic associate with a specific network is always enumerated with the same address

@SchSeba got additional ideas ?

[mshannongit]

Thanks @adrianchiris

The predictable address piece is tricky when using IaC systems like Terraform - as systems like Terraform will try to create resources in parallel (even a single resource with count > 1). That means when running calls like attachVNIC (subnet1), attachVNIC (subnet2) on an instance, the first call to execute and complete gets the lower PCI address - which could be either call.

The ability thus to give explicit node-specific configuration to the SRIOV-DP becomes rather important.

[SchSeba]

Hi @mshannongit, in general we need to find a way to fix the slack channel. I will try to handle that with the intel folks.

to your question we have a solution for this that is implemented by kubevirt also.

you can use the acpi filder. let me add some info
https://libvirt.org/formatdomain.html#network-interfaces

sriov-network-device-plugin/README.md

Line 335 in 174cd16

| "acpiIndexes" | N | Target device's acpi index as string | `string` list Default: `null` | "acpiIndexes": ["101"] |

let me know if that helps you

[mshannongit]

Thanks @SchSeba - Another approach one of our engineers is looking in to is parameterizing the configmap key that is referenced from the daemonset definition. Such that, on each worker node, rather than looking for and mounting config.json from the configmap, we mount based on the Node Name.

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin/blob/master/deployments/sriovdp-daemonset.yaml

 volumes:
        - name: config-volume
          configMap:
            name: sriovdp-config
            items:
            - key: "{{ .Node.Name }}"
              path: config.json