ComputeDomain: support more than one domain per node, with subsets of GPUs

[jgehrcke]

Currently, there can be at most one ComputeDomain (CD) per node. This holds true regardless of the size of the workload in that domain. If the workload does not consume all node-associated GPUs then the remaining GPU resources (provided by that node) cannot be part of any other CD. Hence, the ability to spawn more than one CD per node will help with resource utilization, as indicated in this figure:

The CD indicated above in yellow is something that cannot be spawned today after the green and blue ones are already in place -- building out support for that is the goal tracked here.

Supporting more than one CD per node requires significant implementation changes: currently, there is a one-to-one mapping between CD and IMEX daemons, and only one IMEX daemon can be run per node (a technical constraint that at least for now we take for granted). That is, supporting more than one CD per node in all likelihood requires us to build a one-to-many relationship between IMEX daemon and ComputeDomain. That, on the other hand, might have impact on security isolation:

• one-to-one mapping (current): each CD uses its own IMEX daemon ensemble (one IMEX domain, one IMEX channel).
• one-to-many mapping (future): different CDs would be run in the same IMEX daemon ensemble, share an IMEX domain, and be isolated "only" by using different IMEX channels.

[ZiMengSheng]

绿色的和蓝色的目前也是不允许的，多个 ResourceClaim 无法在一个节点上分配出来，因为我们在 ResourceSlice 里只上报了一个 Channel 设备。

[wsxiaozhang]

In my understanding of imex service, the imex domain/channel can only "isolate" the MNNVL communication on node (compute tray) level, but not device (GPU on one node) level. In that case, how multiple CDs on one node can isolate GPU1 in podA to GPU2 in podB showing in your sample.
I think even using different channels in one CD, there seems to be no way to specify which GPU ids to be isolated on one node. I guess that depends on which GPUs are exposed by the CUDA application running in pod.

In face, I can't find any detailed doc of the exact isolation level of imex service itself. So please correct me if my understanding is wrong. thanks.

[jgehrcke]

@wsxiaozhang

In that case, how multiple CDs on one node can isolate GPU1 in podA to GPU2 in podB showing in your sample

there seems to be no way to specify which GPU ids to be isolated on one node

There is a way!

• There are many IMEX channels within an IMEX domain.
• The DRA driver controls which IMEX channel(s) to specifically inject into each container.

To build on your example, podA (on GPU 1+2) and podB (on GPU 2+3) can be orchestrated so that they are in the same IMEX domain, but do not share any IMEX channel and hence cannot read/write each other's memory via MNNVL primitives.

In other words: the DRA driver can guarantee for pods to use different channels, even if on the same domain. That can be used to implement secure isolation between workloads.

Notably, in the IMEX security model, this is a first-class workload isolation technique (even between workloads sharing a node) [1].

While this type of isolation can be considered strong, we appreciate any defense-in-depth concerns and think about building an opt-out when introducing channel-backed ComputeDomains. In past discussions, we were envisioning adding a new parameter that can be set on a per-CD basis: something like an isolationMode that can be set to "domain". That would guarantee for workload consuming this ComputeDomain to be backed by an exclusive IMEX domain (i.e., to never share a node with workload from a different ComputeDomain).

--

[1] The IMEX docs (section "multi-node channels") discuss the usage of channels as a valid isolation barrier (both, between nodes and within a node):