GPU sharing: support Dynamic MIG (using DRA partitionable devices)

[jgehrcke]

Dynamically provision MIG devices based on user requests (needs to be further spec'd out).

[jgehrcke]

Resources

MIG

• https://www.escape-technology.de/images/news/2025/NVIDIA_MIG/MIG_User_Guide.pdf
• https://docs.nvidia.com/datacenter/tesla/mig-user-guide/

DRA / Partitionable Devices

• [KEP-4815] Update KEP for 1.35 kubernetes/enhancements#5515
• https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4815-dra-partitionable-devices

Dyn MIG poc based on 'classic DRA'

From Kevin's earlier work:

• createMigDevice()
• enumerateAllPossibleDevices()
• getMigProfileInfos()

nv-partitionable-resources (poc, slightly out of date)

From Kevin's earlier work:

• wg-device-management/nv-partitionable-resources
• ToResourceSliceSpec()
• GetperGPUAllocatableDevices()

[jgehrcke]

More resources.

• nvlib device.VisitMigProfiles()
• nvlib device.VisitMigDevices()

[jgehrcke]

NVML ref docs: https://docs.nvidia.com/deploy/nvml-api/group__nvmlMultiInstanceGPU.html

A few interesting quotes of mixed relevance:

nvmlDeviceSetMigMode() -- This API may unbind or reset the device to activate the requested mode. Thus, the attributes associated with the device, such as minor number, might change. The caller of this API is expected to query such attributes again.

Compute instance IDs are unique per GPU instance and remain valid until the compute instance is destroyed.

A placement represents the location of a GPU instance within a device. This API only returns all the possible placements for the given profile regardless of whether MIG is enabled or not. A created GPU instance occupies memory slices described by its placement. Creation of new GPU instance will fail if there is overlap with the already occupied memory slices.

nvmlGpuInstanceGetComputeInstanceProfileInfo() -- this API can be used to enumerate all MIG profiles supported by NVML in a forward compatible way by invoking it on profile values starting from 0, until the API returns NVML_ERROR_INVALID_ARGUMENT.

[jgehrcke]

Trying to answer some of my questions, about the interaction between allocation of a full device vs. allocation of partitions thereof.

First, an experiment in which I ask an AI chat bot a question about the KEP -- if the answers are wrong that's an interesting outcome. Soon, I will have my answer anyway. Is there really a higher-level notion distinguishing partitions from whole devices, or does everything rely on counters?

"Allocating the full device marks all partitions as unavailable for separate allocation, but does not decrement any partition counter. The system tracks the whole device and its partitions as separate, non-overlapping resources." -- is that in any way true?

"The system enforces mutual exclusivity: as soon as any partition of a device is allocated the device becomes unavailable for exclusive allocation as a whole device. Works both ways: allocating any partition blocks the whole device, and allocating the whole device blocks all partitions from being allocated individually."

---

Interesting quote from KEP 4815, to keep in the back of my mind:

it is recommended that ResourceSlices include devices in smallest-to-largest order. Otherwise, the largest one will always be allocated.

---

Allowed units for counter values? Dimensionless integers, I hope. But multipliers seem to be allowed; example:

sharedCounters:
- name: gpu-0-counter-set
  counters:
...
    memory:
      value: 40Gi

[jgehrcke]

There's the "walk left-to-right" principle when trying to understand valid combinations of MIG devices. I find this specified logically and rather precisely in the MIG user guide, section 6:

a valid combination can be built by starting with an instance profile on the left and combining it with other instance profiles as you move to the right

Applying this principle to A100 shows an interesting asymmetry:

After creating a MIG device with just one compute slice, a MIG device with four compute slices cannot be created anymore, despite there being seven overall.

Is that true? That's what I deduct from this schema:

Naive usage of slice counters would not account for this asymmetry. But of course we can make allocation of a single compute slice decrement more than one counter. For example:

• available single-slices from seven to six
• available four-slices from one to zero

Probably @klueska has already thought about this.