Is it possible to enable MIG only on specific nodes when using the GPU Operator?

[larcane97]

hi, im a beginner with gpu operator and i have a basic question.

i have multiple gpu nodes(2 nodes with A100).
i want to enable mig only on one node, and keep the other as a normal gpu node(mis disabled)

i already know that it's not possible to have heterogeneous gpus within a single node, and that all nodes should have the same type of GPU.

however, i'm wordering is it possible to enable mig on only some of the nodes in the cluster(only partial nodes)?
if that's possible, i plan to assign GPUs to pods using node labels to control which node the pod is assigned

thanks in advance :)

[kittywaresz]

Hi @larcane97!

If I understand your question correctly, you can set the nvidia.com/mig.config label to all-disabled on each node where you don't want MIG to be enabled. This value disables MIG on all GPU devices on a node.

Please note that this value is set on each MIG-capable node by default right after installing the gpu-operator via the Helm chart, so you don't need to set it explicitly. You only need to change the nvidia.com/mig.config label to other values on nodes where you want MIG to be enabled.

[visheshtanksale]

Hello @larcane97, By default, nodes are labeled with nvidia.com/mig.config: all-disabled and you must specify the MIG configuration to apply.
You can find the details here
Closing this. Please reopen the issue if you need further clarification