Vgpu and GPU Passthrough on same cluster using GPU Operator with KubeVirt

[ss-armada]

Is it possible to configure the GPU Operator on a single Kubernetes cluster where:
• Node A is configured to run virtual machines with GPU passthrough
• Node B is configured to run virtual machines with vGPU

I’ve been able to verify that each configuration works individually on separate clusters or test environments. However, I’d like to know if it’s officially supported—or at least known to work reliably—to run both modes concurrently across different nodes in the same cluster.

Are there any limitations, recommended configurations, or known issues when mixing passthrough and vGPU-enabled nodes under a single GPU Operator deployment?

[cdesiniotis]

Yes, this configuration is supported. Refer to https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kubevirt.html

[ss-armada]

Thanks! Just to confirm — if the GPU Operator is already deployed on a cluster without vGPU enabled, and one of our GPU nodes (currently running with the default workload and no special labels) is still unused, can we:
1. Add the appropriate vgpu node label to that unused node, and
2. Perform a helm upgrade on the GPU Operator,

…and expect vGPU to be enabled only on that node, without affecting any of the existing passthrough-configured nodes?

I just want to ensure this won’t interfere with the existing nodes already using passthrough.

[cdesiniotis]

Yes, this should work. Based on your message, I am assuming the following: GPU Operator is installed with sandboxWorkloads.enabled=true and sandboxWorkloads.defaultWorkload=vm-passthrough. None of the nodes are labeled with nvidia.com/gpu.workload.config, so all of them get configured for GPU passthrough.

If my assumption is correct, then the following procedure should achieve what you are aiming to do:

1. Perform a helm upgrade and enable vGPU in the chart values -- this primarily involves enabling the vgpuManager component and specifying what image to use.
2. Label your unused GPU node with nvidia.com/gpu.workload.config=vm-vgpu. The vGPU stack should get deployed only on this node.

[ss-armada]

I just checked, One of the nodes is labeled nvidia.com/gpu.workload.config=vm-passthrough, The other is labeled nvidia.com/mig.strategy=single

[visheshtanksale]

@ss-armada Do you need any further help on this one? Closing this. Please reopen if you need further clarification