Clarification on Automatic Component Cleanup When Node Labels Change (e.g., `container` ↔ `vm-passthrough`)

[kingeasternsun]

Description

According to the GPU Operator with KubeVirt documentation, we can assign labels like container, vm-passthrough, or vm-vgpu to nodes, and the GPU Operator deploys specific operands based on these labels.

Question:
If a node is initially labeled with container and the GPU Operator deploys corresponding components, will it automatically clean up the container-related components and deploy vm-passthrough-specific operands when an SRE modifies the node’s label to vm-passthrough? Conversely, if a node’s label is changed from vm-passthrough back to container, will the Operator also handle this transition automatically?

Context:
We need to confirm whether the GPU Operator’s controller actively reconciles node label changes to ensure only the correct operands remain on nodes, or if manual intervention (e.g., deleting old DaemonSets/Pods) is required after label updates.

Why This Matters

Understanding this behavior is critical for:

• Ensuring clean transitions between GPU workload modes (e.g., containerized vs. VM-passthrough).
• Avoiding resource conflicts or performance degradation due to orphaned components.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[cdesiniotis]

@kingeasternsun yes, the gpu-operator watches for changes to this node label and will ensure the right set of pods are running in the gpu-operator namespace to enable that particular use case. So for example, let's say the node is currently labeled with nvidia.com/gpu.workload.config=container. If you update the label to nvidia.com/gpu.workload.config=vm-passthrough the gpu-operator will tear down all of the pods running on that node and start up new ones that enable the vm-passthrough use case. Updating the node label should be taken with precaution, e.g. ensure the node is cordoned and no active GPU workloads are running on it.