CD plugin: channel prepare: fail if allocated as of checkpoint

[jgehrcke]

A patch proposal that addresses #641.

When a resource on a node gets released by a pod and then -- in quick succession -- gets consumed by a new pod (through a new resource claim) then we must make sure that the old resource claim actually gets Unprepare() ed first, before the new resource claim gets Prepare()d.

If said Prepare() comes in early, it needs to be rejected because the device is still allocated from the node's point of view. If Unprepare()ing the old resource claim is done last, workload fails. See #641.

This patch implements one of many ways to make sure that the early Prepare() gets rejected. It raises an interesting question about entries in the checkpoint JSON in the PrepareStarted state.

[jgehrcke]

from review: rename to AssertImexChannelNotAllocated() -- call further below, closer to other asserts.

[jgehrcke]

Is that true during upgrades though?

Yes, of course -- that's why we introduced the file-based locking (to never have interleaving Prepare/Unprepare calls, even when having multiple processes).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jgehrcke]

Tested current state of the patch in a failover test. The new check in action, from a kubelet plugin log:

[pod/nvidia-dra-driver-gpu-kubelet-plugin-lmpgt/compute-domains] 2025-10-06T19:00:00.354419370Z E1006 19:00:00.354370 1 workqueue.go:138] Failed to reconcile work item: error preparing devices for claim 8c6f8987-f8b8-4534-a163-769716e8e4f6: prepare devices failed: error applying config: allocation failed: channel 0 already allocated by claim 294bf5c1-c9b6-44ff-8171-afa14939193d (according to checkpoint)

[jgehrcke]

It raises an interesting question about entries in the checkpoint JSON in the PrepareStarted state.

Kevin and I discussed this today at length. The discussion naturally converged to what we believe is a fundamentally required state reconciliation; once again based on a periodic type of cleanup: periodically, we have to perform a (beautifully named) self-initiated Unprepare() of previously partially performed Prepare()s.

Routine, at the high level:

Perform periodically:

• read checkpoint
• iterate through RCs in PrepareStarted
• for each: RC still known in API server? If not: initiate an Unprepare, and remove from checkpoint file

Value is two-fold:

• Critical: unprepare any partially performed claim preparation (this might take care of e.g. removing a CD node label)
• Likely never a real-world problem, but still a decent thing to do: stop unbounded growth of the checkpoint JSON file over time.

Edit: now tracking this here: #643

Edit 2: thanks for another review, @klueska :)