Manage lifecycle of compute-domain-daemon's service account outside of helm

[klueska]

At present, if a DRA driver gets helm uninstalled, the service account created for any running daemonsets spawned by the compute domain controller gets removed. This means that the daemonset will fail to access any API service objects during the time that the DRA driver is uninstalled. In general, this only really affect upgrades / downgrades of the DRA driver, but makes things especially tricky when switching namespaces across an upgrade in the context of the GPU Operator. To fix this, we need to move the lifecylce management of this service account into the controller so that it gets created / deleted alongside the daemonset itself.

[jgehrcke]

Thanks for the great description.

daemonset will fail to access any API service objects during the time that the DRA driver is uninstalled

I read this as a race condition during a multi-step uninstall procedure, or even a deterministically wrong order of removal. Interesting, and important for us to start thinking about this class of problems.

Does Helm even try to provide primitives to declare dependencies and hence order-of-teardown? Regardless of that, I think that indeed we want to move as much of this logic as we can into our components because that might also yield self-healing properties in view of events not triggered by the Helm CLI.

In any case, I believe we want to make teardown of the driver (complex system with inter-dependencies) reliable and truly best-effort (instead of fail-fast).

We clearly must account for inter-dependencies, and for race conditions.

Connecting some dots; this relates to (which I want to read into more deeply):

• Add a MultiNamespaceDaemonsetManager for ComputeDomains #452 (comment)
• Create one service account even if additional namespaces are configured #585

[jgehrcke]

during the time that the DRA driver is uninstalled

After sync call: I have misunderstood: this is about the time after the DRA driver was uninstalled, with the workload still running.

Edit: from sync meet: the CD daemon cannot read the CD object anymore then.

[jgehrcke]

Strongly related topic: how should driver helm uninstall affect running workload? Workload should stay running. That requires for example all subsystems for CD fault tolerance and elasticity. (Just deletion does not need to work).