Multi-GPU allocation with precise control in shared environment

[FourierMourier]

Hi!

So far I played with time-slicing and I need a way to guarantee that when specifying gpu.shared: 2 (or any other number > 1), a pod gets access to two different physical GPUs, not the same GPU twice. While testing shows this seems to work through some implicit affinity, there's no explicit way to ensure this behavior through deployment manifests.

Additionally, for GPUs with different capacities, I'd like to be able to set different numbers of allowed instances per GPU.

I've tried using the devices field the sharing.timeSlicing config, but logs show the only available value is "all".

Is there a way to explicitly control GPU allocation to ensure a pod gets access to multiple distinct GPUs when needed? Perhaps there's another approach besides time-slicing that might address this use case?

I'm trying to achieve a setup where:

1. All GPUs are shared
2. Some pods need access specifically to multiple different GPUs (not the same GPU multiple times)
3. Other pods need access to just one GPU per pod

For context, I've verified that when using gpu.shared: 2, pods do seem to access different GPUs, but I'd like a more explicit way to guarantee and control this behavior.

Currently I have a gpu-operator deployed via helm of version version=v24.9.1 on the node(s) with 2+ gpu(s), while k8s is v1.30.6+rke2r1

[chipzoller]

Is there a way to explicitly control GPU allocation to ensure a pod gets access to multiple distinct GPUs when needed? Perhaps there's another approach besides time-slicing that might address this use case?

No, this is not supported and no intent to support this in the static device plugin as has been stated elsewhere. Additionally, as stated in the device plug-in repository here:

Unlike with "normal" GPU requests, requesting more than one shared GPU does not imply that you will get guaranteed access to a proportional amount of compute power. It only implies that you will get access to a GPU that is shared by other clients (each of which has the freedom to run as many processes on the underlying GPU as they want).

[FourierMourier]

I see, so the option left here is mps, right? Or there's literally no way to do so and won't be?

[chipzoller]

MPS wouldn't be able to do this either. DRA will be the go-forward way to accomplish this use case but I don't think it has been solved for yet.

[FourierMourier]

thanks! so you're referring to the k8s-dra-driver-gpu as DRA, right?

[chipzoller]

Yes

[changhyuni]

@FourierMourier should use HAMi

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[cdesiniotis]

Closing this issue as the question was answered in #1400 (comment) and #1400 (comment). Thanks @chipzoller!