enhance resource interpreter test framework

zhzhuang-zju · zhzhuang-zju · commit 783012448de3 · 2025-11-20T14:51:35.000+08:00
Signed-off-by: zhzhuang-zju &lt;m17799853869@163.com&gt;
diff --git a/pkg/resourceinterpreter/default/thirdparty/README.md b/pkg/resourceinterpreter/default/thirdparty/README.md
@@ -0,0 +1,133 @@
+# Thirdparty Resource Interpreter
+
+This directory contains third-party resource interpreters for Karmada. These interpreters define how Karmada should handle custom resources from various third-party applications and operators.
+
+## Files
+
+- `thirdparty.go` - Main implementation of the third-party resource interpreter
+- `thirdparty_test.go` - Test suite for validating resource interpreter customizations
+- `resourcecustomizations/` - Directory containing resource customization definitions organized by API version and kind
+
+## Directory Structure
+
+The resource customizations are organized in the following structure:
+
+```
+resourcecustomizations/
+├── <group>/
+│   └── <version>/
+│       └── <kind>/
+│           ├── customizations.yaml          # Resource interpreter customization rules
+│           ├── customizations_tests.yaml    # Test cases for the customizations
+│           └── testdata/                    # Test input and expected output files
+│               ├── desired_xxx.yaml      # Input resource for desired state
+│               ├── observed_xxx.yaml     # Input resource for observed state  
+│               ├── status_xxx.yaml       # Input aggregated status items
+```
+
+## How to test
+
+### Running Tests
+
+To run all third-party resource interpreter tests:
+
+```bash
+cd pkg/resourceinterpreter/default/thirdparty
+go test -v
+```
+
+### Creating Test Cases
+
+#### 1. Create Test Structure
+
+For a new resource type, create the directory structure:
+
+```bash
+mkdir -p resourcecustomizations/<group>/<version>/<kind>/testdata
+```
+
+#### 2. Create Test Data Files
+
+Test data files are generally divided into three categories:
+
+- `desired_xxx.yaml`: Resource definitions deployed on the control plane
+- `observed_xxx.yaml`: Resource definitions observed in a member cluster
+- `status_xxx.yaml`: Status information of the resource on each member cluster, with structure `[]workv1alpha2.AggregatedStatusItem`
+
+Multiple test data files can be created for each category as needed. Pay attention to naming distinctions, as they will ultimately be referenced in `customizations_tests.yaml`.
+
+#### 3. Create Test Configuration
+
+Create `customizations_tests.yaml` to define test cases:
+
+```yaml
+tests:
+  - observedInputPath: testdata/observed-flinkdeployment.yaml
+    operation: InterpretReplica
+    desiredResults:
+      replica: 2
+```
+
+Where:
+- `operation` specifies the operation of resource interpreter
+- `desiredResults` defines the expected output, which is a key-value mapping where the key is the field name of the expected result and the value is the expected result
+
+The keys in `desiredResults` for different operations correspond to the Name field of the results returned by the corresponding resource interpreter operation `RuleResult.Results`.
+
+For example:
+```go
+func (h *healthInterpretationRule) Run(interpreter *declarative.ConfigurableInterpreter, args RuleArgs) *RuleResult {
+	obj, err := args.getObjectOrError()
+	if err != nil {
+		return newRuleResultWithError(err)
+	}
+	healthy, enabled, err := interpreter.InterpretHealth(obj)
+	if err != nil {
+		return newRuleResultWithError(err)
+	}
+	if !enabled {
+		return newRuleResultWithError(fmt.Errorf("rule is not enabled"))
+	}
+	return newRuleResult().add("healthy", healthy)
+}
+```
+
+The `desiredResults` for operation `InterpretHealth` should contain the `healthy` key.
+
+### Supported Operations
+
+The test framework supports the following operations:
+
+- `InterpretReplica` - Extract replica count from resource
+- `InterpretComponent` -  Extract component information from resource
+- `ReviseReplica` - Modify replica count in resource
+- `InterpretStatus` - Extract status information
+- `InterpretHealth` - Determine resource health status
+- `InterpretDependency` - Extract resource dependencies
+- `AggregateStatus` - Aggregate status from multiple clusters
+- `Retain` - Retain the desired resource template.
+
+### Test Validation
+
+The test framework validates:
+
+1. **Lua Script Syntax** - Ensures all Lua scripts are syntactically correct
+2. **Execution Results** - Compares actual results with expected results
+
+### Debugging Tests
+
+To debug failing tests:
+
+1. **Check Lua Script Syntax** - Ensure your Lua scripts are valid
+2. **Verify Test Data** - Confirm test input files are properly formatted
+3. **Review Expected Results** - Make sure expected results match the actual operation output
+4. **Use Verbose Output** - Run tests with `-v` flag for detailed output
+
+### Best Practices
+
+1. **Comprehensive Coverage** - Test all supported operations for your resource type
+2. **Edge Cases** - Include tests for edge cases and error conditions  
+3. **Realistic Data** - Use realistic resource definitions in test data
+4. **Clear Naming** - Use descriptive names for test files and cases
+
+For more information about resource interpreter customizations, see the [Karmada documentation](https://karmada.io/docs/userguide/globalview/customizing-resource-interpreter/).
diff --git a/pkg/resourceinterpreter/default/thirdparty/resourcecustomizations/flink.apache.org/v1beta1/FlinkDeployment/customizations_tests.yaml b/pkg/resourceinterpreter/default/thirdparty/resourcecustomizations/flink.apache.org/v1beta1/FlinkDeployment/customizations_tests.yaml
@@ -2,9 +2,116 @@ tests:
   - desiredInputPath: testdata/desired-flinkdeployment.yaml
     statusInputPath: testdata/status-file.yaml
     operation: AggregateStatus
-  - observedInputPath: testdata/observed-flinkdeployment.yaml
+    desiredResults:
+      aggregatedStatus:
+        apiVersion: flink.apache.org/v1beta1
+        kind: FlinkDeployment
+        metadata:
+          name: basic-example
+          namespace: test-namespace
+        spec:
+          flinkConfiguration:
+            taskmanager.numberOfTaskSlots: "2"
+          flinkVersion: v1_17
+          image: flink:1.17
+          job:
+            jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
+            parallelism: 2
+            upgradeMode: stateless
+          jobManager:
+            replicas: 1
+            resource:
+              cpu: 1
+              memory: 2048m
+          mode: native
+          serviceAccount: flink
+          taskManager:
+            resource:
+              cpu: 1
+              memory: 2048m
+        status:
+          clusterInfo:
+            flink-revision: 2750d5c @ 2023-05-19T10:45:46+02:00
+            flink-version: 1.17.1
+            total-cpu: "2.0"
+            total-memory: "4294967296"
+          jobManagerDeploymentStatus: READY
+          jobStatus:
+            checkpointInfo:
+              lastPeriodicCheckpointTimestamp: 0
+            jobId: 44cc5573945d1d4925732d915c70b9ac
+            jobName: Minimal Spec Example
+            savepointInfo:
+              lastPeriodicSavepointTimestamp: 0
+            startTime: "1717599166365"
+            state: RUNNING
+            updateTime: "1717599182544"
+          lifecycleState: STABLE
+          observedGeneration: 1
+          reconciliationStatus:
+            lastReconciledSpec: '{"spec":{"job":{"jarURI":"local:///opt/flink/examples/streaming/StateMachineExample.jar","parallelism":2,"entryClass":null,"args":[],"state":"running","savepointTriggerNonce":null,"initialSavepointPath":null,"checkpointTriggerNonce":null,"upgradeMode":"stateless","allowNonRestoredState":null,"savepointRedeployNonce":null},"restartNonce":null,"flinkConfiguration":{"taskmanager.numberOfTaskSlots":"2"},"image":"flink:1.17","imagePullPolicy":null,"serviceAccount":"flink","flinkVersion":"v1_17","ingress":null,"podTemplate":null,"jobManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":1,"podTemplate":null},"taskManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":null,"podTemplate":null},"logConfiguration":null,"mode":null},"resource_metadata":{"apiVersion":"flink.apache.org/v1beta1","metadata":{"generation":2},"firstDeployment":true}}'
+            lastStableSpec: '{"spec":{"job":{"jarURI":"local:///opt/flink/examples/streaming/StateMachineExample.jar","parallelism":2,"entryClass":null,"args":[],"state":"running","savepointTriggerNonce":null,"initialSavepointPath":null,"checkpointTriggerNonce":null,"upgradeMode":"stateless","allowNonRestoredState":null,"savepointRedeployNonce":null},"restartNonce":null,"flinkConfiguration":{"taskmanager.numberOfTaskSlots":"2"},"image":"flink:1.17","imagePullPolicy":null,"serviceAccount":"flink","flinkVersion":"v1_17","ingress":null,"podTemplate":null,"jobManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":1,"podTemplate":null},"taskManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":null,"podTemplate":null},"logConfiguration":null,"mode":null},"resource_metadata":{"apiVersion":"flink.apache.org/v1beta1","metadata":{"generation":2},"firstDeployment":true}}'
+            reconciliationTimestamp: 1717599148930
+            state: DEPLOYED
+          taskManager:
+            labelSelector: component=taskmanager,app=basic-example
+            replicas: 1
+  - observedInputPath: testdata/desired-flinkdeployment.yaml
     operation: InterpretReplica
+    desiredResults:
+      replica: 2
+      requires:
+        resourceRequest:
+          "cpu": "1"
+          "memory": "2048m"
+        namespace: "test-namespace"
+  - observedInputPath: testdata/desired-flinkdeployment.yaml
+    operation: InterpretComponent
+    desiredResults:
+      components:
+        - name: jobmanager
+          replicas: 1
+          replicaRequirements:
+            resourceRequest:
+              "cpu": "1"
+              "memory": "2048m"
+        - name: taskmanager
+          replicas: 1
+          replicaRequirements:
+            resourceRequest:
+              "cpu": "1"
+              "memory": "2048m"
   - observedInputPath: testdata/observed-flinkdeployment.yaml
     operation: InterpretHealth
+    desiredResults:
+      healthy: true
   - observedInputPath: testdata/observed-flinkdeployment.yaml
     operation: InterpretStatus
+    desiredResults:
+      status:
+        clusterInfo:
+          flink-revision: 2750d5c @ 2023-05-19T10:45:46+02:00
+          flink-version: 1.17.1
+          total-cpu: "2.0"
+          total-memory: "4294967296"
+        jobManagerDeploymentStatus: READY
+        jobStatus:
+          checkpointInfo:
+            lastPeriodicCheckpointTimestamp: 0
+          jobId: 44cc5573945d1d4925732d915c70b9ac
+          jobName: Minimal Spec Example
+          savepointInfo:
+            lastPeriodicSavepointTimestamp: 0
+          startTime: "1717599166365"
+          state: RUNNING
+          updateTime: "1717599182544"
+        lifecycleState: STABLE
+        observedGeneration: 1
+        reconciliationStatus:
+          lastReconciledSpec: '{"spec":{"job":{"jarURI":"local:///opt/flink/examples/streaming/StateMachineExample.jar","parallelism":2,"entryClass":null,"args":[],"state":"running","savepointTriggerNonce":null,"initialSavepointPath":null,"checkpointTriggerNonce":null,"upgradeMode":"stateless","allowNonRestoredState":null,"savepointRedeployNonce":null},"restartNonce":null,"flinkConfiguration":{"taskmanager.numberOfTaskSlots":"2"},"image":"flink:1.17","imagePullPolicy":null,"serviceAccount":"flink","flinkVersion":"v1_17","ingress":null,"podTemplate":null,"jobManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":1,"podTemplate":null},"taskManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":null,"podTemplate":null},"logConfiguration":null,"mode":null},"resource_metadata":{"apiVersion":"flink.apache.org/v1beta1","metadata":{"generation":2},"firstDeployment":true}}'
+          lastStableSpec: '{"spec":{"job":{"jarURI":"local:///opt/flink/examples/streaming/StateMachineExample.jar","parallelism":2,"entryClass":null,"args":[],"state":"running","savepointTriggerNonce":null,"initialSavepointPath":null,"checkpointTriggerNonce":null,"upgradeMode":"stateless","allowNonRestoredState":null,"savepointRedeployNonce":null},"restartNonce":null,"flinkConfiguration":{"taskmanager.numberOfTaskSlots":"2"},"image":"flink:1.17","imagePullPolicy":null,"serviceAccount":"flink","flinkVersion":"v1_17","ingress":null,"podTemplate":null,"jobManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":1,"podTemplate":null},"taskManager":{"resource":{"cpu":1.0,"memory":"2048m","ephemeralStorage":null},"replicas":null,"podTemplate":null},"logConfiguration":null,"mode":null},"resource_metadata":{"apiVersion":"flink.apache.org/v1beta1","metadata":{"generation":2},"firstDeployment":true}}'
+          reconciliationTimestamp: 1717599148930
+          state: DEPLOYED
+        taskManager:
+          labelSelector: component=taskmanager,app=basic-example
+          replicas: 1
diff --git a/pkg/resourceinterpreter/default/thirdparty/thirdparty_test.go b/pkg/resourceinterpreter/default/thirdparty/thirdparty_test.go
@@ -23,10 +23,13 @@ import (
 	"io"
 	"os"
 	"path/filepath"
+	"reflect"
 	"strings"
 	"testing"
 	"time"
 
+	"k8s.io/apimachinery/pkg/api/equality"
+	"k8s.io/apimachinery/pkg/api/resource"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/util/json"
 	"k8s.io/apimachinery/pkg/util/yaml"
@@ -39,6 +42,7 @@ import (
 )
 
 var rules interpreter.Rules = interpreter.AllResourceInterpreterCustomizationRules
+var checker = equality.Semantic.Copy()
 
 func checkScript(script string) error {
 	ctx, cancel := context.WithTimeout(context.TODO(), time.Second)
@@ -102,11 +106,11 @@ type TestStructure struct {
 }
 
 type IndividualTest struct {
-	DesiredInputPath  string `yaml:"desiredInputPath,omitempty"`
-	ObservedInputPath string `yaml:"observedInputPath,omitempty"`
-	StatusInputPath   string `yaml:"statusInputPath,omitempty"`
-	DesiredReplica    int64  `yaml:"desiredReplicas,omitempty"`
-	Operation         string `yaml:"operation"`
+	DesiredInputPath  string                 `yaml:"desiredInputPath,omitempty"`
+	ObservedInputPath string                 `yaml:"observedInputPath,omitempty"`
+	StatusInputPath   string                 `yaml:"statusInputPath,omitempty"`
+	DesiredResults    map[string]interface{} `yaml:"desiredResults,omitempty"`
+	Operation         string                 `yaml:"operation"`
 }
 
 func checkInterpretationRule(t *testing.T, path string, configs []*configv1alpha1.ResourceInterpreterCustomization) {
@@ -133,7 +137,7 @@ func checkInterpretationRule(t *testing.T, path string, configs []*configv1alpha
 			if err != nil {
 				t.Fatalf("checking %s of %s, expected nil, but got: %v", rule.Name(), customization.Name, err)
 			}
-			args := interpreter.RuleArgs{Replica: input.DesiredReplica}
+			args := interpreter.RuleArgs{}
 			if input.DesiredInputPath != "" {
 				args.Desired = getObj(t, dir+"/"+strings.TrimPrefix(input.DesiredInputPath, "/"))
 			}
@@ -143,10 +147,79 @@ func checkInterpretationRule(t *testing.T, path string, configs []*configv1alpha
 			if input.StatusInputPath != "" {
 				args.Status = getAggregatedStatusItems(t, dir+"/"+strings.TrimPrefix(input.StatusInputPath, "/"))
 			}
-			if result := rule.Run(ipt, args); result.Err != nil {
+			result := rule.Run(ipt, args)
+			if result.Err != nil {
 				t.Fatalf("execute %s %s error: %v\n", customization.Name, rule.Name(), result.Err)
 			}
+			for _, res := range result.Results {
+				expected, ok := input.DesiredResults[res.Name]
+				if !ok {
+					t.Logf("no expected result for %s", res.Name)
+					continue
+				}
+
+				if equal, err := deepEqual(expected, res.Value); err != nil || !equal {
+					t.Fatal("unexpected result for", res.Name, "expected:", expected, "got:", res.Value, "error:", err)
+				}
+			}
+		}
+	}
+}
+
+func deepEqual(expected, actualValue interface{}) (bool, error) {
+	err := checker.AddFuncs(
+		func(a, b resource.Quantity) bool {
+			return a.Equal(b)
+		})
+	if err != nil {
+		return false, fmt.Errorf("failed to add custom equality function: %w", err)
+	}
+
+	expectedJSONBytes, err := json.Marshal(expected)
+	if err != nil {
+		return false, fmt.Errorf("failed to marshal expected value: %w", err)
+	}
+
+	// Handle known types for semantic comparison
+	switch typedActual := actualValue.(type) {
+	case *workv1alpha2.ReplicaRequirements:
+		var unmarshaledExpected workv1alpha2.ReplicaRequirements
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			return false, fmt.Errorf("failed to unmarshal expected JSON into ReplicaRequirements: %w", err)
+		}
+		return checker.DeepEqual(&unmarshaledExpected, typedActual), nil
+
+	case *configv1alpha1.DependentObjectReference:
+		var unmarshaledExpected configv1alpha1.DependentObjectReference
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			return false, fmt.Errorf("failed to unmarshal expected JSON into DependentObjectReference: %w", err)
+		}
+		return reflect.DeepEqual(&unmarshaledExpected, typedActual), nil
+
+	case []workv1alpha2.Component:
+		var unmarshaledExpected []workv1alpha2.Component
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			return false, fmt.Errorf("failed to unmarshal expected JSON into []Component: %w", err)
+		}
+
+		return checker.DeepEqual(unmarshaledExpected, typedActual), nil
+
+	case *unstructured.Unstructured:
+		var unmarshaledExpected unstructured.Unstructured
+
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			fmt.Println(err)
+			return false, fmt.Errorf("failed to unmarshal expected JSON into Unstructured: %w", err)
+		}
+		return reflect.DeepEqual(&unmarshaledExpected, typedActual), nil
+
+	default:
+		// Fallback: marshal actualValue and do byte-wise comparison
+		actualJSON, err := json.Marshal(actualValue)
+		if err != nil {
+			return false, fmt.Errorf("failed to marshal actual value: %w", err)
 		}
+		return bytes.Equal(expectedJSONBytes, actualJSON), nil
 	}
 }
 
