Fix NAR/PR state toggling when BMH is in servicing error

When a BMH  enters servicing error during Day 2 configuration,
the NodeAllocationRequest and ProvisioningRequest were toggling
between Failed and ConfigurationUpdateRequested states.

Root cause: observedGeneration was only updated on configuration success.
When configuration failed due to BMH errors, the generation mismatch
persisted, causing the FSM to continuously re-trigger spec change handling.

Fix: Always update observedGeneration after processing a spec change,
regardless of success or failure.

Signed-off-by: Tao Liu <[email protected]>

Author: tliu2021

hwmgr-plugins/metal3/controller/metal3_nodeallocationrequest_controller.go

@@ -724,10 +724,18 @@ func (r *NodeAllocationRequestReconciler) handleNodeAllocationRequestSpecChanged
 				err = updateErr
 			}
 		}
-		if status == metav1.ConditionTrue && reason == string(hwmgmtv1alpha1.ConfigApplied) {
-			if err := hwmgrutils.UpdateNodeAllocationRequestPluginStatus(ctx, r.Client, nodeAllocationRequest); err != nil {
-				return hwmgrutils.RequeueWithShortInterval(), fmt.Errorf("failed to update hwMgrPlugin observedGeneration Status: %w", err)
-			}
+
+		// Update observedGeneration to acknowledge the spec change, regardless of success or failure.
+		// This prevents infinite retry loops when configuration fails due to hardware errors.
+		// If we only update on success, a persistent hardware error (like BMH servicing error)
+		// will cause the FSM to continuously detect a spec change and re-trigger configuration attempts.
+		if updateErr := hwmgrutils.UpdateNodeAllocationRequestPluginStatus(ctx, r.Client, nodeAllocationRequest); updateErr != nil {
+			r.Logger.ErrorContext(ctx, "Failed to update hwMgrPlugin observedGeneration Status",
+				slog.String("nodeAllocationRequest", nodeAllocationRequest.Name),
+				slog.String("error", updateErr.Error()))
+			// Return error to trigger requeue
+			return hwmgrutils.RequeueWithShortInterval(),
+				fmt.Errorf("failed to update hwMgrPlugin observedGeneration Status: %w", updateErr)
 		}
 	}