AUTO_UPGRADE_POLICY_ENABLED set to true, but eviction and drain are "disabled by the upgrade policy"

[futurewasfree]

See log below, specifically lines with Current value of AUTO_UPGRADE_POLICY_ENABLED=true and Auto eviction of GPU pods .. , Auto drain ...:

Getting current value of the 'nvidia.com/gpu.deploy.operator-validator' node label                                                                                                                             
Current value of 'nvidia.com/gpu.deploy.operator-validator=true'                                                                                                                                               
Getting current value of the 'nvidia.com/gpu.deploy.container-toolkit' node label                                                                                                                              
Current value of 'nvidia.com/gpu.deploy.container-toolkit=true'                                                                                                                                                
Getting current value of the 'nvidia.com/gpu.deploy.device-plugin' node label                                                                                                                                  
Current value of 'nvidia.com/gpu.deploy.device-plugin=true'                                                                                                                                                    
Getting current value of the 'nvidia.com/gpu.deploy.gpu-feature-discovery' node label                                                                                                                          
Current value of 'nvidia.com/gpu.deploy.gpu-feature-discovery=true'                                                                                                                                            
Getting current value of the 'nvidia.com/gpu.deploy.dcgm-exporter' node label                                                                                                                                  
Current value of 'nvidia.com/gpu.deploy.dcgm-exporter=true'                                                                                                                                                    
Getting current value of the 'nvidia.com/gpu.deploy.dcgm' node label                                                                                                                                           
Current value of 'nvidia.com/gpu.deploy.dcgm=true'                                                                                                                                                             
Getting current value of the 'nvidia.com/gpu.deploy.mig-manager' node label                                                                                                                                    
Current value of 'nvidia.com/gpu.deploy.mig-manager='                                                                                                                                                          
Getting current value of the 'nvidia.com/gpu.deploy.nvsm' node label                                                                                                                                           
Current value of 'nvidia.com/gpu.deploy.nvsm='                                                                                                                                                                 
Getting current value of the 'nvidia.com/gpu.deploy.sandbox-validator' node label                                                                                                                              
Current value of 'nvidia.com/gpu.deploy.sandbox-validator='                                                                                                                                                    
Getting current value of the 'nvidia.com/gpu.deploy.sandbox-device-plugin' node label                                                                                                                          
Current value of 'nvidia.com/gpu.deploy.sandbox-device-plugin='                                                                                                                                                
Getting current value of the 'nvidia.com/gpu.deploy.vgpu-device-manager' node label                                                                                                                            
Current value of 'nvidia.com/gpu.deploy.vgpu-device-manager='                                                                                                                                                  
Current value of AUTO_UPGRADE_POLICY_ENABLED=true'                                                                                                                                                             
Shutting down all GPU clients on the current node by disabling their component-specific nodeSelector labels                                                                                                    
node/aks-gputest-50947407-vmss000001 labeled                                                                                                                                                                   
Waiting for the operator-validator to shutdown                                                                                                                                                                 
pod/nvidia-operator-validator-kqw2v condition met                                                                                                                                                              
Waiting for the container-toolkit to shutdown                                                                                                                                                                  
Waiting for the device-plugin to shutdown                                                                                                                                                                      
Waiting for gpu-feature-discovery to shutdown                                                                                                                                                                  
Waiting for dcgm-exporter to shutdown                                                                                                                                                                          
Waiting for dcgm to shutdown                                                                                                                                                                                   
Auto eviction of GPU pods on node aks-gputest-50947407-vmss000001 is disabled by the upgrade policy                                                                                                            
Unloading NVIDIA driver kernel modules...                                                                                                                                                                      
nvidia_modeset       1306624  0                                                                                                                                                                                
nvidia_uvm           1527808  4                                                                                                                                                                                
nvidia              56717312  143 nvidia_uvm,nvidia_modeset                                                                                                                                                    
drm                   622592  3 drm_kms_helper,nvidia,hyperv_drm                                                                                                                                               
i2c_core               90112  3 drm_kms_helper,nvidia,drm                                                                                                                                                      
Could not unload NVIDIA driver kernel modules, driver is in use                                                                                                                                                
Auto drain of the node aks-gputest-50947407-vmss000001 is disabled by the upgrade policy                                                                                                                       
Failed to uninstall nvidia driver components                                                                                                                                                                   
Auto eviction of GPU pods on node aks-gputest-50947407-vmss000001 is disabled by the upgrade policy                                                                                                            
Auto drain of the node aks-gputest-50947407-vmss000001 is disabled by the upgrade policy                                                                                                                       
Rescheduling all GPU clients on the current node by enabling their component-specific nodeSelector labels                                                                                                      
node/aks-gputest-50947407-vmss000001 labeled             

just by reading the logic in https://github.com/NVIDIA/k8s-driver-manager/blob/master/driver-manager
(_is_driver_auto_upgrade_policy_enabled in particlar) it should not happen, I believe

1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04): Ubuntu 22.04
• Kernel Version: 5.15.0-1066
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): Containerd
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): AKS
• GPU Operator Version: 24.3.0

2. Issue or feature description

Auto eviction and auto drain checks are further evaluated and not stopped by _is_driver_auto_upgrade_policy_enabled

3. Steps to reproduce the issue

Default install with Upgrade Controller enabled.
Kill the nvidia-driver-daemonset pod and trigger the driver reinstall on a node with GPU enabled pods already running.

4. Information to attach (optional if deemed irrelevant)

• kubernetes pods status: kubectl get pods -n OPERATOR_NAMESPACE
• kubernetes daemonset status: kubectl get ds -n OPERATOR_NAMESPACE
• If a pod/ds is in an error state or pending state kubectl describe pod -n OPERATOR_NAMESPACE POD_NAME
• If a pod/ds is in an error state or pending state kubectl logs -n OPERATOR_NAMESPACE POD_NAME --all-containers
• Output from running nvidia-smi from the driver container: kubectl exec DRIVER_POD_NAME -n OPERATOR_NAMESPACE -c nvidia-driver-ctr -- nvidia-smi
• containerd logs journalctl -u containerd > containerd.log

Collecting full debug bundle (optional):

curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/main/hack/must-gather.sh
chmod +x must-gather.sh
./must-gather.sh

NOTE: please refer to the must-gather script for debug data collected.

This bundle can be submitted to us via email: operator_feedback@nvidia.com

[cdesiniotis]

Kill the nvidia-driver-daemonset pod and trigger the driver reinstall on a node with GPU enabled pods already running.

@futurwasfree is there a particular reason why you are directly killing the nvidia-driver-daemonset pod? If you need to perform a driver upgrade, you should be editing driver.version in clusterpolicy, and the driver upgrade-controller would take care of facilitating the upgrade, as documented here: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html

[futurewasfree]

@cdesiniotis
this way I'm mimicking the actual node restart, e.g. after upgrade or for other reason.
behaviour is the same as I've described in initial post.

[SeanEsper]

Hi @futurwasfree ,
I get the same errors even though auto drain and auto eviction are allowed. Do you know what could be the reason?

[SeanEsper]

I get the following errors:
Auto eviction of GPU pods on node ... is disabled by the upgrade policy
Auto drain of the node ... is disabled by the upgrade policy

[futurewasfree]

This sounds exactly like my issue. Some configuration checks in k8s-driver-manager are broken in my opinion.

Btw, currently I'm using slightly different configuration, but it also have problems and require manual attention from time to time (on node restart): #902

[SeanEsper]

Do you now how / where I can configure the K8s-Driver-Manager settings?

These settings:
manager:
env:
- name: ENABLE_GPU_POD_EVICTION
value: "false"
- name: ENABLE_AUTO_DRAIN
value: "true"
- name: DRAIN_USE_FORCE
value: "false"
- name: DRAIN_POD_SELECTOR_LABEL
value: ""
- name: DRAIN_TIMEOUT_SECONDS
value: "0s"
- name: DRAIN_DELETE_EMPTYDIR_DATA
value: "false"

And where can do this --set driver.upgradePolicy.autoUpgrade=false instruction?

Thanks for your efforts!

[futurewasfree]

Basically there are two ways of doing upgrade: either with Upgrade Controller or k8s-driver-manager (I'm using the latter one at the moment), you could read more about them on this page: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html
Re: env settings like ENABLE_GPU_POD_EVICTION they're on driver level. So it's your NVIDIADriver entry, either defined on Cluster Policy CRD or NVIDIA Driver CRD: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-configuration.html
Re: --set driver.upgradePolicy.autoUpgrade=false. I set it as override when I do helm upgrade:

helm upgrade --wait gpu-operator \
    -n gpu-operator --create-namespace nvidia/gpu-operator \
    --set operator.defaultRuntime="containerd" \
    --set driver.nvidiaDriverCRD.enabled=true \
    --set driver.nvidiaDriverCRD.deployDefaultCR=false \
    --set driver.upgradePolicy.autoUpgrade=false \
 ...

[manrajag]

On openshift4 the only way I could avoid / bypass this issue was to use "oc drain " command with all the parameters and clear the workload located on the GPU node. Then ensure that when I uncordon this node, only the GPU operator pods will relocate back onto the GPU node first.

The Environment Variables seem to be unaffected or have no override or force-enforce or force-enabled. And the Validator of the GPU_Operator components seems to be broken for ensuring the GPU workloads are cleanly removed or stopped.

This is all in my opinion and based on my hands-on experience so there might be some mistakes or misinterpretations.

Thanks!

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[cdesiniotis]

this way I'm mimicking the actual node restart, e.g. after upgrade or for other reason.

On a node reboot, I would expect the NVIDIA driver modules to no longer be loaded (they do not persist a reboot) so the k8s-driver-manager init container should not have to clean up anything. This is why I would argue "killing the nvidia-driver-daemonset pod" does not actually mimic a node reboot.

@futurewasfree is my understanding correct that you want the k8s-driver-manager settings, in particular ENABLE_GPU_POD_EVICTION and ENABLE_AUTO_DRAIN, to be honored even when driver.upgradePolicy.autoUpgrade=true in Clusterpolicy? I guess this is something we could consider, but would need to evaluate further to see if there are any side effects.