Fix Prepare() idempotency by preventing reverse state transition and atomically detecting already-prepared claims

[shivamerla]

• Avoid overwriting PrepareCompleted with PrepareStarted on subsequent Prepare() calls.
• Move claim prepared state checks into a single updateCheckpoint call for atomicity.
• Ensure checkpoint state does not transition from completed to started state again.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[shivamerla]

Run dir: /tmp/k8s-dra-driver-gpu-tests-out-smerla/bats-tests-1763521347-XXXXX.DYUaakUsGu
+ cd /cwd
+ echo 'Running k8s cluster cleanup (invasive)...'
Running k8s cluster cleanup (invasive)...
+ set +x
--- STARTING TEST SUITE ---
+ TMPDIR=/tmp/k8s-dra-driver-gpu-tests-out-smerla/bats-tests-1763521347-XXXXX.DYUaakUsGu
+ bats --print-output-on-failure --no-tempdir-cleanup --timing --abort tests/bats/test_gpu_basic.bats tests/bats/test_gpu_stress.bats
test_gpu_basic.bats
 ✓ 1 pod(s), 1 full GPU [19569]
 ✓ 2 pod(s), 1 full GPU each [21535]
 ✓ 2 pod(s), 1 full GPU (shared, 1 RC) [17509]
 ✓ 1 pod(s), 2 cntrs, 1 full GPU (shared, 1 RCT) [17893]
test_gpu_stress.bats
 ✓ Stress: shared ResourceClaim across 15 pods x 5 loops [275201]

5 tests, 0 failures in 355 seconds

[jgehrcke]

Thanks for thinking this through! Any brain power on that front is much appreciated.

Higher-level notes:

• I haven't looked at the details yet, at all! Will do.
• I will want to compare this to commits made as part of the dynamic MIG device allocation PoC -- I found at least one issue with the current (gpu plugin) code that I addressed; and I just didn't get to opening a PR yet since coming back from Kubecon.
• In the long run (or short term?) I think we should do ourselves a favor ando find a way to have more common code between the CD and GPU sides of things -- bug fixes should benefit both plugins and of course that's easier with either 1) fully shared code or 2) duplicated, but not overly diverging code.

I hope we're OK with slightly slow movement here. 🐌

[shivamerla]

• In the long run (or short term?) I think we should do ourselves a favor ando find a way to have more common code between the CD and GPU sides of things -- bug fixes should benefit both plugins and of course that's easier with either 1) fully shared code or 2) duplicated, but not overly diverging code.

Totally agree on this. There is some duplication here between both plugins.