Skip to content

Add NUMA node ID attribute for NUMA aware allocation between CPU, GPU and NICs #690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add NUMA node ID attribute for NUMA aware allocation between CPU, GPU and NICs #690

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions cmd/gpu-kubelet-plugin/deviceinfo.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,14 @@ import (
"k8s.io/utils/ptr"
)

const (
// SharedDeviceAttributePrefix is the prefix used for shared attributes among NVIDIA devices {NIC, GPU}.
SharedDeviceAttributePrefix = "resource.nvidia.com/"
// DeviceAttributeNumaNodeID is a device attribute name which describe the NUMA node ID of the device.
// This attribute can be used to identify devices that share the same NUMA node.
SharedDeviceAttributeNumaNodeID resourceapi.QualifiedName = SharedDeviceAttributePrefix + "numaNodeID"
)

type GpuInfo struct {
UUID string `json:"uuid"`
minor int
Expand All @@ -42,6 +50,7 @@ type GpuInfo struct {
pcieBusID string
pcieRootAttr *deviceattribute.DeviceAttribute
migProfiles []*MigProfileInfo
numaNodeID *int
}

type MigDeviceInfo struct {
Expand All @@ -55,6 +64,7 @@ type MigDeviceInfo struct {
ciInfo *nvml.ComputeInstanceInfo
pcieBusID string
pcieRootAttr *deviceattribute.DeviceAttribute
numaNodeID *int
}

type MigProfileInfo struct {
Expand Down Expand Up @@ -119,6 +129,9 @@ func (d *GpuInfo) GetDevice() resourceapi.Device {
if d.pcieRootAttr != nil {
device.Attributes[d.pcieRootAttr.Name] = d.pcieRootAttr.Value
}
if d.numaNodeID != nil {
device.Attributes[SharedDeviceAttributeNumaNodeID] = resourceapi.DeviceAttribute{IntValue: ptr.To(int64(*d.numaNodeID))}
}
return device
}

Expand Down Expand Up @@ -181,5 +194,8 @@ func (d *MigDeviceInfo) GetDevice() resourceapi.Device {
if d.pcieRootAttr != nil {
device.Attributes[d.pcieRootAttr.Name] = d.pcieRootAttr.Value
}
if d.numaNodeID != nil {
device.Attributes[SharedDeviceAttributeNumaNodeID] = resourceapi.DeviceAttribute{IntValue: ptr.To(int64(*d.numaNodeID))}
}
return device
}
9 changes: 9 additions & 0 deletions cmd/gpu-kubelet-plugin/nvlib.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,13 @@ func (l deviceLib) getGpuInfo(index int, device nvdev.Device) (*GpuInfo, error)
klog.Warningf("error getting PCIe root for device %d, continuing without attribute: %v", index, err)
}

var numaNodeID *int
if id, ret := device.GetNumaNodeId(); ret == nvml.SUCCESS {
numaNodeID = &id
} else {
klog.Warningf("error getting NUMA node ID for device %d, continuing without attribute: %v", index, ret)
Copy link
Collaborator

@jgehrcke jgehrcke Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting! Would this be entirely exceptional, or is this an expected scenario on some systems?

This leads me to thinking:

  • On non-NUMA systems this should probably not be a warning. On those systems, I suppose this code path is expected, and maybe it's worth emitting an info-level log message.

  • On NUMA systems this should maybe be an error. At least I'd like to hear our thoughts about this.

  • How can we reliably distinguish a NUMA from a non-NUMA system (as a reference point)?

Copy link
Collaborator

@jgehrcke jgehrcke Oct 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ref docs for nvmlDeviceGetNumaNodeId() indicate that we should distinguish three types of errors:

NVML_SUCCESS if the NUMA node is retrieved successfully
NVML_ERROR_NOT_SUPPORTED if request is not supported on the current platform
NVML_ERROR_INVALID_ARGUMENT if device node is invalid

I don't want to answer my own questions from above, but it looks like we probably want to give NVML_ERROR_NOT_SUPPORTED its own code path (not emitting a warning/error log), and then NVML_ERROR_INVALID_ARGUMENT is handled separately (and maybe that one should be fatal?).

Copy link
Contributor Author

@shivamerla shivamerla Oct 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jgehrcke good point, the not supported error can be logged as info instead of warning. After discussing with Kevin, we actually want to use the nvmlDeviceGetMemoryAffinity() . This will give the bitmask of NUMA nodes as the GPU can belong to multiple within the given socket. This means the current matchAttibute is not sufficient for this. Instead i will propose a new parameter to allow attributes to be matched from a list instead of a single element.

In that case it will be new set of error codes that we need to handle and i will update based on that.

Copy link
Collaborator

@jgehrcke jgehrcke Oct 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack, thanks!

}

var migProfiles []*MigProfileInfo
for i := 0; i < nvml.GPU_INSTANCE_PROFILE_COUNT; i++ {
giProfileInfo, ret := device.GetGpuInstanceProfileInfo(i)
Expand Down Expand Up @@ -275,6 +282,7 @@ func (l deviceLib) getGpuInfo(index int, device nvdev.Device) (*GpuInfo, error)
pcieBusID: pcieBusID,
pcieRootAttr: pcieRootAttr,
migProfiles: migProfiles,
numaNodeID: numaNodeID,
}

return gpuInfo, nil
Expand Down Expand Up @@ -368,6 +376,7 @@ func (l deviceLib) getMigDevices(gpuInfo *GpuInfo) (map[string]*MigDeviceInfo, e
ciInfo: &ciInfo,
pcieBusID: gpuInfo.pcieBusID,
pcieRootAttr: gpuInfo.pcieRootAttr,
numaNodeID: gpuInfo.numaNodeID,
}
return nil
})
Expand Down