Add DRA support for GPU pod eviction during driver upgrades

[karthikvetrivel]

Summary

Extends the driver-upgrade controller to detect and evict GPU workloads using Dynamic Resource Allocation (DRA) in addition to traditional nvidia.com/gpu resources. This ensures GPU driver upgrades work correctly as Kubernetes transitions from device plugins to the DRA model (GA in K8s 1.34+).

Testing

Tested in a kubeadm cluster (K8s 1.34) with NVIDIA DRA driver installed:

1. Created test workloads:

   • DRA GPU pod with allocated ResourceClaim (driver: gpu.nvidia.com)
   • Traditional GPU pod (nvidia.com/gpu: 1)

2. Verified ResourceClaim allocation:

   $ kubectl get resourceclaim -o yaml | grep -A 5 allocation
   status:
     allocation:
       devices:
         results:
         - driver: gpu.nvidia.com
           device: gpu-0

3. Triggered driver upgrade eviction:

   $ k8s-driver-manager uninstall_driver --node-name=<node>
   
   GPU pod - default/dra-allocated-pod
   GPU pod - default/traditional-allocated-pod
   evicting pod default/traditional-allocated-pod
   evicting pod default/dra-allocated-pod

4. Verified both pods evicted successfully:

   $ kubectl get pods
   No resources found

As I have it right now, I only evict DRA GPU pods if their ResourceClaim has status.allocation != nil (actively using a GPU), but I'm wondering if we should evict ANY pod with a ResourceClaim requesting nvidia.com GPUs (regardless of allocation status) to prevent race conditions where a pending claim gets allocated during the upgrade - thoughts?

[shivamerla]

LGTM

[rahulait]

I'm wondering if we should evict ANY pod with a ResourceClaim requesting nvidia.com GPUs (regardless of allocation status) to prevent race conditions where a pending claim gets allocated during the upgrade - thoughts?

Don't we cordon the node before starting the upgrade? If the node is cordoned, then there won't be new allocations to that node.

[karthikvetrivel]

I'm wondering if we should evict ANY pod with a ResourceClaim requesting nvidia.com GPUs (regardless of allocation status) to prevent race conditions where a pending claim gets allocated during the upgrade - thoughts?

Don't we cordon the node before starting the upgrade? If the node is cordoned, then there won't be new allocations to that node.

I think you're right here. Good point, thanks for bringing it up.

[rajathagasthya]

Should we consider returning an error here? Not getting ResourceClaims after retries seems like a legitimate reason for failing the driver upgrade.

[karthikvetrivel]

If the claim doesn't exist, continuing is correct — no claim means no GPU allocation. We could distinguish NotFound (continue) from transient errors (fail), but that adds complexity. Given the retry + 10s timeout, the risk of missing a GPU pod due to transient errors is low.

[rajathagasthya]

Agreed when the claim is not found. But when there other errors (i.e. apierrors.IsNotFound(err) == false) that show up even after retries, IMO the right thing to do is to bubble up and fail the uninstall_driver command because it means we couldn't guarantee a clean eviction.

[karthikvetrivel]

I think that's fair! Will look into adding this.

[karthikvetrivel]

Updated.