Define new ComputeDomain API in `resource.nvidia.com/v1beta2`

[klueska]

We've learned a lot iterating on the resource.nvidia.com/v1beta1 API.

With the introduction of several new features (including using DNS names as the default solution for linking IMEX daemons), as well as several upcoming features (including supporting worker pods from different ComputeDomains on the same node), it's time to start thinking about a new API structure that better matches these features.

[klueska]

Related:

• ComputeDomain: do not require a priori node count (numNodes) for creation #364

[jgehrcke]

Quick notes; ideas we want to consider (some from previous discussions):

• a parameter for controlling channel vs. domain-based isolation (something like isolationMode), cf. ComputeDomain: support more than one domain per node, with subsets of GPUs #353 (comment)
• a parameter to enable gating: holding back workload until the IMEX domain fully formed (w/o any a priori knowledge of numNodes -- i.e., facing the challenge of knowing what 'fully formed' means)
• a parameter that allows for opting in or opting out of a ComputeDomain's ability to span more than one NVLink partition
• let's maybe review IMEX daemon config parameters again and see if anything should be exposed on a per-CD basis

Edit, another thought that we had today:

• Introduce a CD mode in which it every node is required to be MNNVL-capable -- maybe make the non-MNNVL fallback be an explicit opt-in, instead of an opt-out -- the automatic fallback may always be at risk of being unintentional (also see Review fallback to non-MNNVL mode (enhance logging, ...) #626 and Release workload on a non-MNNVL node in a CD #625).

[jgehrcke]

A question for us to explore, in relationship to this topic: what can we do to in fact remove the IMEXDaemonsWithDNSNames=false code path as soon as possible (and with that, the feature flag)?

What would be the most extreme action to take, and could we justify taking that? Maybe we could get away with that because the project is so new.

Will we want to support the current API in parallel with the new API? If we were to do that, one can argue that using the current (and then old) API version together with numNodes>0 should also be backed by the current IMEXDaemonsWithDNSNames=false code path. That would mean we'd have to carry that on (and test, maintain, ...) for a while. Which would be quite a bit of effort for questionable impact.