Should simultaneous allocation of a GPU and its MIG partitions be allowed?

[nojnhuh]

Once a Pod has already been allocated a MIG partition for a GPU, should another Pod be able to claim the whole parent GPU?

In my cluster I have two nodes each with two A100 GPUs. One is MIG-disabled and the other node has both GPUs divided into 4 partitions each. Workloads that consume all of the MIG partitions on the MIG node and all of the GPUs on both nodes are able to schedule, where I'd expect allocating either all of the MIGs or all of the GPUs on the MIG node to fail since they're the same underlying hardware.

% kubectl get pods,resourceclaims -o wide
NAME                            READY   STATUS    RESTARTS   AGE    IP             NODE                              NOMINATED NODE   READINESS GATES
pod/all-gpus-5484b696c5-57qzr   1/1     Running   0          9m6s   10.244.3.103   aks-migpool-68842551-vmss000001   <none>           <none>
pod/all-gpus-5484b696c5-m9gdf   1/1     Running   0          9m6s   10.244.2.34    aks-gpupool-15127565-vmss000002   <none>           <none>
pod/all-migs-c6849886c-sw92r    1/1     Running   0          9m7s   10.244.3.47    aks-migpool-68842551-vmss000001   <none>           <none>

NAME                                                                 STATE                AGE
resourceclaim.resource.k8s.io/all-gpus-5484b696c5-57qzr-gpus-26tnm   allocated,reserved   9m6s
resourceclaim.resource.k8s.io/all-gpus-5484b696c5-m9gdf-gpus-gtvpt   allocated,reserved   9m6s
resourceclaim.resource.k8s.io/all-migs-c6849886c-sw92r-migs-7f5c8    allocated,reserved   9m7s

% for p in $(kubectl get po -o name); do echo $p; kubectl logs $p; done
pod/all-gpus-5484b696c5-57qzr
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-ddf64803-f421-0e5b-883e-cd752de16db7)
GPU 1: NVIDIA A100 80GB PCIe (UUID: GPU-61572322-0336-adb6-f996-0ae0cf40abcc)
pod/all-gpus-5484b696c5-m9gdf
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-199490ab-88da-c1e2-0f96-631264c2f512)
GPU 1: NVIDIA A100 80GB PCIe (UUID: GPU-13719f71-412c-d11e-b236-a262f530a4fc)
pod/all-migs-c6849886c-sw92r
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-ddf64803-f421-0e5b-883e-cd752de16db7)
  MIG 3g.40gb     Device  0: (UUID: MIG-e774da9b-cceb-5e61-91c3-3b1f58636990)
  MIG 2g.20gb     Device  1: (UUID: MIG-b8e88dd7-8b46-5717-91e6-71f276dd31dc)
  MIG 1g.10gb     Device  2: (UUID: MIG-d74574ee-26ab-5d71-ad7a-dc6c515ac642)
  MIG 1g.10gb     Device  3: (UUID: MIG-6a0297e8-9708-515f-99e2-f111fc87221d)
GPU 1: NVIDIA A100 80GB PCIe (UUID: GPU-61572322-0336-adb6-f996-0ae0cf40abcc)
  MIG 3g.40gb     Device  0: (UUID: MIG-a642816e-3a0d-529c-9263-00b38d842101)
  MIG 2g.20gb     Device  1: (UUID: MIG-be872654-574b-558c-bcb1-fb5d5a9891ee)
  MIG 1g.10gb     Device  2: (UUID: MIG-4e3c5199-f362-5ed3-82e8-58b19759fc12)
  MIG 1g.10gb     Device  3: (UUID: MIG-96aa58ce-6075-57da-87a4-bf1c3b5ade79)

workloads.yaml

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: all-migs
spec:
  spec:
    devices:
      requests:
      - name: migs
        exactly:
          deviceClassName: mig.nvidia.com
          allocationMode: All
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: all-gpus
spec:
  spec:
    devices:
      requests:
      - name: gpu
        exactly:
          deviceClassName: gpu.nvidia.com
          allocationMode: All
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: all-migs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: all-migs
  template:
    metadata:
      labels:
        app: all-migs
    spec:
      containers:
      - name: ctr
        image: ubuntu:22.04
        command: ["bash", "-c"]
        args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
        resources:
          claims:
          - name: migs
      resourceClaims:
      - name: migs
        resourceClaimTemplateName: all-migs
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: all-gpus
spec:
  replicas: 2
  selector:
    matchLabels:
      app: all-gpus
  template:
    metadata:
      labels:
        app: all-gpus
    spec:
      containers:
      - name: ctr
        image: ubuntu:22.04
        command: ["bash", "-c"]
        args: ["nvidia-smi -L; trap 'exit 0' TERM; sleep 9999 & wait"]
        resources:
          claims:
          - name: gpus
      resourceClaims:
      - name: gpus
        resourceClaimTemplateName: all-gpus

resourceslices.yaml

apiVersion: v1
items:
- apiVersion: resource.k8s.io/v1
  kind: ResourceSlice
  metadata:
    creationTimestamp: "2025-11-05T17:31:13Z"
    generateName: aks-gpupool-15127565-vmss000002-gpu.nvidia.com-
    generation: 1
    name: aks-gpupool-15127565-vmss000002-gpu.nvidia.com-sdzwp
    ownerReferences:
    - apiVersion: v1
      controller: true
      kind: Node
      name: aks-gpupool-15127565-vmss000002
      uid: 8816457f-9747-456c-86fc-6dd6fbaee6bb
    resourceVersion: "512490"
    uid: 32e7cc60-f7b0-42db-a030-d78a7004d0cb
  spec:
    devices:
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        pcieBusID:
          string: "0001:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        type:
          string: gpu
        uuid:
          string: GPU-199490ab-88da-c1e2-0f96-631264c2f512
      capacity:
        memory:
          value: 80Gi
      name: gpu-0
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        pcieBusID:
          string: "0002:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        type:
          string: gpu
        uuid:
          string: GPU-13719f71-412c-d11e-b236-a262f530a4fc
      capacity:
        memory:
          value: 80Gi
      name: gpu-1
    driver: gpu.nvidia.com
    nodeName: aks-gpupool-15127565-vmss000002
    pool:
      generation: 1
      name: aks-gpupool-15127565-vmss000002
      resourceSliceCount: 1
- apiVersion: resource.k8s.io/v1
  kind: ResourceSlice
  metadata:
    creationTimestamp: "2025-11-05T17:31:23Z"
    generateName: aks-migpool-68842551-vmss000001-gpu.nvidia.com-
    generation: 1
    name: aks-migpool-68842551-vmss000001-gpu.nvidia.com-22vnt
    ownerReferences:
    - apiVersion: v1
      controller: true
      kind: Node
      name: aks-migpool-68842551-vmss000001
      uid: 728d24f6-eb2a-4ffb-9364-1de88566f990
    resourceVersion: "512562"
    uid: f859ed88-842a-46a0-8bd0-085ee5d3353d
  spec:
    devices:
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-ddf64803-f421-0e5b-883e-cd752de16db7
        pcieBusID:
          string: "0001:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 3g.40gb
        type:
          string: mig
        uuid:
          string: MIG-e774da9b-cceb-5e61-91c3-3b1f58636990
      capacity:
        copyEngines:
          value: "3"
        decoders:
          value: "2"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 40192Mi
        memorySlice4:
          value: "1"
        memorySlice5:
          value: "1"
        memorySlice6:
          value: "1"
        memorySlice7:
          value: "1"
        multiprocessors:
          value: "42"
        ofaEngines:
          value: "0"
      name: gpu-0-mig-9-4-4
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-ddf64803-f421-0e5b-883e-cd752de16db7
        pcieBusID:
          string: "0001:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 2g.20gb
        type:
          string: mig
        uuid:
          string: MIG-b8e88dd7-8b46-5717-91e6-71f276dd31dc
      capacity:
        copyEngines:
          value: "2"
        decoders:
          value: "1"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 19968Mi
        memorySlice0:
          value: "1"
        memorySlice1:
          value: "1"
        multiprocessors:
          value: "28"
        ofaEngines:
          value: "0"
      name: gpu-0-mig-14-0-2
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        pcieBusID:
          string: "0002:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        type:
          string: gpu
        uuid:
          string: GPU-61572322-0336-adb6-f996-0ae0cf40abcc
      capacity:
        memory:
          value: 80Gi
      name: gpu-1
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-61572322-0336-adb6-f996-0ae0cf40abcc
        pcieBusID:
          string: "0002:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 2g.20gb
        type:
          string: mig
        uuid:
          string: MIG-be872654-574b-558c-bcb1-fb5d5a9891ee
      capacity:
        copyEngines:
          value: "2"
        decoders:
          value: "1"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 19968Mi
        memorySlice0:
          value: "1"
        memorySlice1:
          value: "1"
        multiprocessors:
          value: "28"
        ofaEngines:
          value: "0"
      name: gpu-1-mig-14-0-2
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        pcieBusID:
          string: "0001:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        type:
          string: gpu
        uuid:
          string: GPU-ddf64803-f421-0e5b-883e-cd752de16db7
      capacity:
        memory:
          value: 80Gi
      name: gpu-0
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-ddf64803-f421-0e5b-883e-cd752de16db7
        pcieBusID:
          string: "0001:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 1g.10gb
        type:
          string: mig
        uuid:
          string: MIG-d74574ee-26ab-5d71-ad7a-dc6c515ac642
      capacity:
        copyEngines:
          value: "1"
        decoders:
          value: "0"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 9728Mi
        memorySlice2:
          value: "1"
        multiprocessors:
          value: "14"
        ofaEngines:
          value: "0"
      name: gpu-0-mig-19-2-1
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-ddf64803-f421-0e5b-883e-cd752de16db7
        pcieBusID:
          string: "0001:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 1g.10gb
        type:
          string: mig
        uuid:
          string: MIG-6a0297e8-9708-515f-99e2-f111fc87221d
      capacity:
        copyEngines:
          value: "1"
        decoders:
          value: "0"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 9728Mi
        memorySlice3:
          value: "1"
        multiprocessors:
          value: "14"
        ofaEngines:
          value: "0"
      name: gpu-0-mig-19-3-1
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-61572322-0336-adb6-f996-0ae0cf40abcc
        pcieBusID:
          string: "0002:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 1g.10gb
        type:
          string: mig
        uuid:
          string: MIG-4e3c5199-f362-5ed3-82e8-58b19759fc12
      capacity:
        copyEngines:
          value: "1"
        decoders:
          value: "0"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 9728Mi
        memorySlice2:
          value: "1"
        multiprocessors:
          value: "14"
        ofaEngines:
          value: "0"
      name: gpu-1-mig-19-2-1
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-61572322-0336-adb6-f996-0ae0cf40abcc
        pcieBusID:
          string: "0002:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 1g.10gb
        type:
          string: mig
        uuid:
          string: MIG-96aa58ce-6075-57da-87a4-bf1c3b5ade79
      capacity:
        copyEngines:
          value: "1"
        decoders:
          value: "0"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 9728Mi
        memorySlice3:
          value: "1"
        multiprocessors:
          value: "14"
        ofaEngines:
          value: "0"
      name: gpu-1-mig-19-3-1
    - attributes:
        architecture:
          string: Ampere
        brand:
          string: Nvidia
        cudaComputeCapability:
          version: 8.0.0
        cudaDriverVersion:
          version: 13.0.0
        driverVersion:
          version: 580.95.5
        parentUUID:
          string: GPU-61572322-0336-adb6-f996-0ae0cf40abcc
        pcieBusID:
          string: "0002:00:00.0"
        productName:
          string: NVIDIA A100 80GB PCIe
        profile:
          string: 3g.40gb
        type:
          string: mig
        uuid:
          string: MIG-a642816e-3a0d-529c-9263-00b38d842101
      capacity:
        copyEngines:
          value: "3"
        decoders:
          value: "2"
        encoders:
          value: "0"
        jpegEngines:
          value: "0"
        memory:
          value: 40192Mi
        memorySlice4:
          value: "1"
        memorySlice5:
          value: "1"
        memorySlice6:
          value: "1"
        memorySlice7:
          value: "1"
        multiprocessors:
          value: "42"
        ofaEngines:
          value: "0"
      name: gpu-1-mig-9-4-4
    driver: gpu.nvidia.com
    nodeName: aks-migpool-68842551-vmss000001
    pool:
      generation: 1
      name: aks-migpool-68842551-vmss000001
      resourceSliceCount: 1
kind: List
metadata:
  resourceVersion: ""

I encountered this scenario on AKS with Kubernetes v1.34.0 using only GA DRA features. The cluster has GPU Operator v25.10.0 (with the device plugin disabled) and v25.8.0 of the NVIDIA DRA driver (with GPUs enabled).

[guptaNswati]

This should not happen. Definitely a bug.

[guptaNswati]

using your workload.yaml, I was able to repro it in a local kind cluster where both full GPU and mig is allocated.

1 driver.go:162] Returning newly prepared devices for claim '9f4f9e67-ebcb-4b27-80df-2ac86653de80': [{[migs] kind-dra-1-worker gpu-1-mig-19-4-1 [k8s.gpu.nvidia.com/device=gpu-1-mig-19-4-1]} {[migs] kind-dra-1-worker gpu-0-mig-19-1-1 [k8s.gpu.nvidia.com/device=gpu-0-mig-19-1-1]} {[migs] kind-dra-1-worker gpu-1-mig-19-6-1 [k8s.gpu.nvidia.com/device=gpu-1-mig-19-6-1]} {[migs] kind-dra-1-worker gpu-1-mig-19-1-1 [k8s.gpu.nvidia.com/device=gpu-1-mig-19-1-1]} {[migs] kind-dra-1-worker gpu-1-mig-19-3-1 [k8s.gpu.nvidia.com/device=gpu-1-mig-19-3-1]} {[migs] kind-dra-1-worker gpu-0-mig-19-5-1 [k8s.gpu.nvidia.com/device=gpu-0-mig-19-5-1]} {[migs] kind-dra-1-worker gpu-1-mig-19-0-1 [k8s.gpu.nvidia.com/device=gpu-1-mig-19-0-1]} {[migs] kind-dra-1-worker gpu-0-mig-19-3-1 [k8s.gpu.nvidia.com/device=gpu-0-mig-19-3-1]} {[migs] kind-dra-1-worker gpu-0-mig-19-6-1 [k8s.gpu.nvidia.com/device=gpu-0-mig-19-6-1]} {[migs] kind-dra-1-worker gpu-1-mig-19-2-1 [k8s.gpu.nvidia.com/device=gpu-1-mig-19-2-1]} {[migs] kind-dra-1-worker gpu-1-mig-19-5-1 [k8s.gpu.nvidia.com/device=gpu-1-mig-19-5-1]} {[migs] kind-dra-1-worker gpu-0-mig-19-0-1 [k8s.gpu.nvidia.com/device=gpu-0-mig-19-0-1]} {[migs] kind-dra-1-worker gpu-0-mig-19-2-1 [k8s.gpu.nvidia.com/device=gpu-0-mig-19-2-1]} {[migs] kind-dra-1-worker gpu-0-mig-19-4-1 [k8s.gpu.nvidia.com/device=gpu-0-mig-19-4-1]}]
I1114 21:56:06.996160       1 driver.go:162] Returning newly prepared devices for claim 'ef63bdee-d466-4d65-b0cb-b888c47541a8': [{[gpu] kind-dra-1-worker gpu-1 [k8s.gpu.nvidia.com/device=gpu-1]} {[gpu] kind-dra-1-worker gpu-0 [k8s.gpu.nvidia.com/device=gpu-0]}]

I think the problem is the ResourceSlice we publish https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/main/cmd/gpu-kubelet-plugin/driver.go#L174

we don't distinguish between the type of device and has separate entry for both full GPU and MIG device and we don't do any checks during prepare also.

  - attributes:
      architecture:
        string: Hopper
      brand:
        string: Nvidia
      cudaComputeCapability:
        version: 9.0.0
      cudaDriverVersion:
        version: 12.8.0
      driverVersion:
        version: 570.86.15
      pcieBusID:
        string: "0019:01:00.0"
      productName:
        string: NVIDIA GH200 96GB HBM3
      resource.kubernetes.io/pcieRoot:
        string: pci0019:00
      type:
        string: gpu
      uuid:
        string: GPU-9e6df7cb-64d4-5e53-2b1d-cee9e58aeb94
    capacity:
      memory:
        value: 97871Mi
    name: gpu-1
  
  - attributes:
      architecture:
        string: Hopper
      brand:
        string: Nvidia
      cudaComputeCapability:
        version: 9.0.0
      cudaDriverVersion:
        version: 12.8.0
      driverVersion:
        version: 570.86.15
      pcieBusID:
        string: "0009:01:00.0"
      productName:
        string: NVIDIA GH200 96GB HBM3
      resource.kubernetes.io/pcieRoot:
        string: pci0009:00
      type:
        string: gpu
      uuid:
        string: GPU-a4f34abc-7715-3560-dcea-7238b9611a45
    capacity:
      memory:
        value: 97871Mi
    name: gpu-0
    
    - attributes:
      architecture:
        string: Hopper
      brand:
        string: Nvidia
      cudaComputeCapability:
        version: 9.0.0
      cudaDriverVersion:
        version: 12.8.0
      driverVersion:
        version: 570.86.15
      parentUUID:
        string: GPU-9e6df7cb-64d4-5e53-2b1d-cee9e58aeb94
      pcieBusID:
        string: "0019:01:00.0"
      productName:
        string: NVIDIA GH200 96GB HBM3
      profile:
        string: 1g.12gb
      resource.kubernetes.io/pcieRoot:
        string: pci0019:00
      type:
        string: mig
      uuid:
        string: MIG-4e9fed73-e2a4-512f-ad8e-e7bb75aa920e
    capacity:
      copyEngines:
        value: "1"
      decoders:
        value: "1"
      encoders:
        value: "0"
      jpegEngines:
        value: "1"
      memory:
        value: 11Gi
      memorySlice5:
        value: "1"
      multiprocessors:
        value: "16"
      ofaEngines:
        value: "0"
    name: gpu-1-mig-19-5-1

[klueska]

With static MIG we should not be advertising the full GPU if any MIG devices are being advertised.

If both are being advertised then that's a bug.

Before fixing, it would be good to understand when / how the bug was introduced because I'm certain it didn't always exist.

[nojnhuh]

Thanks! I created #719 to track this.

[jgehrcke]

Before fixing, it would be good to understand when / how the bug was introduced because I'm certain it didn't always exist.

I agree. I had a look, and I think I found the relevant change, see #719 (comment). Very large change without the support of tests -- hard to keep free of regressions.