Successfully overwrite the mig partition but cannot find the partition on node

[we1yq]

I changed the config from all-3g to all-7g
$ kubectl label node rtx1 nvidia.com/mig.config=all-7g.40gb --overwrite
node/rtx1 labeled
when I check this command, it shows successfully changed the mig config
$ nvidia-smi -L
GPU 0: NVIDIA A100-PCIE-40GB (UUID: GPU-03ca4983-f693-39d2-d7e0-25090fe07b2f)
MIG 7g.40gb Device 0: (UUID: MIG-a28fecf6-35ba-56a6-aab8-2be643b31249)
GPU 1: NVIDIA TITAN RTX (UUID: GPU-21058121-c040-c847-712d-da7a5cf48e4b)
GPU 2: NVIDIA TITAN RTX (UUID: GPU-edc7db6f-0fec-bc09-9cbe-5a8d2598a62e)
GPU 3: NVIDIA GeForce RTX 3090 (UUID: GPU-1e09b62e-bae8-23dd-a55d-03b34ee00182)
but I cannot find the new partition on the node
$ kubectl describe node rtx1 | grep nvidia.com/mig
nvidia.com/mig.capable=true
nvidia.com/mig.config=all-7g.40gb
nvidia.com/mig.config.state=success
nvidia.com/mig.strategy=single
nvidia.com/mig-3g.20gb: 0
nvidia.com/mig-3g.20gb: 0
nvidia.com/mig-3g.20gb 0 0

$ kubectl get pod -n gpu-operator
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-gb6f5 1/1 Running 0 5m8s
gpu-operator-5798b5b564-zw5tg 1/1 Running 2 (139m ago) 179m
gpu-operator-node-feature-discovery-gc-86f6495b55-ntp72 1/1 Running 1 (154m ago) 179m
gpu-operator-node-feature-discovery-master-694467d5db-pddls 1/1 Running 2 (139m ago) 179m
gpu-operator-node-feature-discovery-worker-g89fd 1/1 Running 2 (139m ago) 179m
nvidia-container-toolkit-daemonset-96vnv 1/1 Running 1 (154m ago) 167m
nvidia-cuda-validator-vcxkr 0/1 Completed 0 5m5s
nvidia-dcgm-exporter-sp7q9 1/1 Running 0 5m8s
nvidia-device-plugin-daemonset-pb98g 0/1 CrashLoopBackOff 5 (112s ago) 5m8s
nvidia-mig-manager-kcp8m 1/1 Running 1 (154m ago) 177m
nvidia-operator-validator-m8xkj 0/1 Init:3/4 1 (2m29s ago) 5m9s

[visheshtanksale]

Can you please provide the logs from the device plugin daemonset pod to triage why is it in CrashLoopBackOffstate?