enhance resource interpreter test framework

Signed-off-by: zhzhuang-zju <[email protected]>

Author: zhzhuang-zju

pkg/resourceinterpreter/default/thirdparty/README.md

@@ -0,0 +1,155 @@
+# Thirdparty Resource Interpreter
+
+This directory contains third-party resource interpreters for Karmada. These interpreters define how Karmada should handle custom resources from various third-party applications and operators.
+
+## Files
+
+- `thirdparty.go` - Main implementation of the third-party resource interpreter
+- `thirdparty_test.go` - Test suite for validating resource interpreter customizations
+- `resourcecustomizations/` - Directory containing resource customization definitions organized by API version and kind
+
+## Directory Structure
+
+The resource customizations are organized in the following structure:
+
+```
+resourcecustomizations/
+├── <group>/
+│   └── <version>/
+│       └── <kind>/
+│           ├── customizations.yaml          # Resource interpreter customization rules
+│           ├── customizations_tests.yaml    # Test cases for the customizations
+│           └── testdata/                    # Test input and expected output files
+│               ├── desired_xxx.yaml      # Input resource for desired state
+│               ├── observed_xxx.yaml     # Input resource for observed state  
+│               ├── status_xxx.yaml       # Input aggregated status items
+│               ├── output_xxx.yaml       # Expected output for various operations
+```
+
+## How to test
+
+### Running Tests
+
+To run all third-party resource interpreter tests:
+
+```bash
+cd pkg/resourceinterpreter/default/thirdparty
+go test -v
+```
+
+### Creating Test Cases
+
+#### 1. Create Test Structure
+
+For a new resource type, create the directory structure:
+
+```bash
+mkdir -p resourcecustomizations/<group>/<version>/<kind>/testdata
+```
+
+#### 2. Create Test Data Files
+
+Test data files are generally divided into four categories:
+
+- `desired_xxx.yaml`: Resource definitions deployed on the control plane.
+- `observed_xxx.yaml`: Resource definitions observed in a member cluster.
+- `status_xxx.yaml`: Status information of the resource on each member cluster, with structure `[]workv1alpha2.AggregatedStatusItem`.
+- `output_xxx.yaml`: Expected output for various operations, with structure `map[string]interface`.
+
+Multiple test data files can be created for each category as needed. Pay attention to naming distinctions, as they will ultimately be referenced in `customizations_tests.yaml`.
+
+#### 3. Create Test Configuration
+
+Test configuration is defined in `customizations_tests.yaml` within the resource customization directory. It specifies 
+the test cases to be executed. Its structure is as follows:
+```go
+type TestStructure struct {
+	Tests []IndividualTest `yaml:"tests"`
+}
+
+type IndividualTest struct {
+	DesiredInputPath  string `yaml:"desiredInputPath,omitempty"` // the path of desired_xxx.yaml
+	ObservedInputPath string `yaml:"observedInputPath,omitempty"` // the path of observed_xxx.yaml
+	StatusInputPath   string `yaml:"statusInputPath,omitempty"` // the path of status_xxx.yaml
+	InputReplicas     int64  `yaml:"inputReplicas,omitempty"` // the input replicas for revise operation
+	OutputResultsPath string `yaml:"outputResultsPath,omitempty"` // the path of output_xxx.yaml
+	Operation         string `yaml:"operation"` // the operation of resource interpreter
+}
+```
+
+Create `customizations_tests.yaml` to define test cases:
+
+```yaml
+tests:
+  - desiredInputPath: testdata/desired-flinkdeployment.yaml
+    statusInputPath: testdata/status-file.yaml
+    operation: AggregateStatus
+    outputResultsPath: testdata/output-flinkdeployment.yaml
+  - desiredInputPath: testdata/desired-flinkdeployment.yaml
+    operation: InterpretReplica
+    outputResultsPath: testdata/output-flinkdeployment.yaml
+```
+
+Where:
+- `operation` specifies the operation of resource interpreter
+- `outputResultsPath` defines the file path for expected output results. The output results are key-value mapping where the key is the field name of the expected result and the value is the expected result.
+
+The keys in output results for different operations correspond to the Name field of the results returned by the corresponding resource interpreter operation `RuleResult.Results`.
+
+For example:
+```go
+func (h *healthInterpretationRule) Run(interpreter *declarative.ConfigurableInterpreter, args RuleArgs) *RuleResult {
+	obj, err := args.getObjectOrError()
+	if err != nil {
+		return newRuleResultWithError(err)
+	}
+	healthy, enabled, err := interpreter.InterpretHealth(obj)
+	if err != nil {
+		return newRuleResultWithError(err)
+	}
+	if !enabled {
+		return newRuleResultWithError(fmt.Errorf("rule is not enabled"))
+	}
+	return newRuleResult().add("healthy", healthy)
+}
+```
+
+The output results for operation `InterpretHealth` should contain the `healthy` key.
+
+### Supported Operations
+
+The test framework supports the following operations:
+
+- `InterpretReplica` - Extract replica count from resource
+- `InterpretComponent` -  Extract component information from resource
+- `ReviseReplica` - Modify replica count in resource
+- `InterpretStatus` - Extract status information
+- `InterpretHealth` - Determine resource health status
+- `InterpretDependency` - Extract resource dependencies
+- `AggregateStatus` - Aggregate status from multiple clusters
+- `Retain` - Retain the desired resource template.
+
+### Test Validation
+
+The test framework validates:
+
+1. **Lua Script Syntax** - Ensures all Lua scripts are syntactically correct
+2. **Execution Results** - Compares actual results with expected results
+
+### Debugging Tests
+
+To debug failing tests:
+
+1. **Check Lua Script Syntax** - Ensure your Lua scripts are valid
+2. **Verify Test Data** - Confirm test input files are properly formatted
+3. **Review Expected Results** - Make sure expected results match the actual operation output
+4. **Use Verbose Output** - Run tests with `-v` flag for detailed output
+
+### Best Practices
+
+1. **Comprehensive Coverage** - Test all supported operations for your resource type
+2. **Edge Cases** - Include tests for edge cases and error conditions  
+3. **Realistic Data** - Use realistic resource definitions in test data
+4. **Clear Naming** - Use descriptive names for test files and cases
+
+For more information about resource interpreter customizations, see the [Karmada documentation](https://karmada.io/docs/userguide/globalview/customizing-resource-interpreter/).

pkg/resourceinterpreter/default/thirdparty/resourcecustomizations/flink.apache.org/v1beta1/FlinkDeployment/customizations_tests.yaml

@@ -2,9 +2,16 @@ tests:
   - desiredInputPath: testdata/desired-flinkdeployment.yaml
     statusInputPath: testdata/status-file.yaml
     operation: AggregateStatus
-  - observedInputPath: testdata/observed-flinkdeployment.yaml
+    outputResultsPath: testdata/output-flinkdeployment.yaml
+  - desiredInputPath: testdata/desired-flinkdeployment.yaml
     operation: InterpretReplica
+    outputResultsPath: testdata/output-flinkdeployment.yaml
+  - desiredInputPath: testdata/desired-flinkdeployment.yaml
+    operation: InterpretComponent
+    outputResultsPath: testdata/output-flinkdeployment.yaml
   - observedInputPath: testdata/observed-flinkdeployment.yaml
     operation: InterpretHealth
+    outputResultsPath: testdata/output-flinkdeployment.yaml
   - observedInputPath: testdata/observed-flinkdeployment.yaml
     operation: InterpretStatus
+    outputResultsPath: testdata/output-flinkdeployment.yaml

pkg/resourceinterpreter/default/thirdparty/thirdparty_test.go

@@ -27,7 +27,9 @@ import (
 	"testing"
 	"time"
 
+	"k8s.io/apimachinery/pkg/api/resource"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/conversion"
 	"k8s.io/apimachinery/pkg/util/json"
 	"k8s.io/apimachinery/pkg/util/yaml"
 
@@ -39,6 +41,10 @@ import (
 )
 
 var rules interpreter.Rules = interpreter.AllResourceInterpreterCustomizationRules
+var checker = conversion.EqualitiesOrDie(
+	func(a, b resource.Quantity) bool {
+		return a.Equal(b)
+	})
 
 func checkScript(script string) error {
 	ctx, cancel := context.WithTimeout(context.TODO(), time.Second)
@@ -102,11 +108,12 @@ type TestStructure struct {
 }
 
 type IndividualTest struct {
-	DesiredInputPath  string `yaml:"desiredInputPath,omitempty"`
-	ObservedInputPath string `yaml:"observedInputPath,omitempty"`
-	StatusInputPath   string `yaml:"statusInputPath,omitempty"`
-	DesiredReplica    int64  `yaml:"desiredReplicas,omitempty"`
-	Operation         string `yaml:"operation"`
+	DesiredInputPath  string `yaml:"desiredInputPath,omitempty"`  // the path of desired_xxx.yaml
+	ObservedInputPath string `yaml:"observedInputPath,omitempty"` // the path of observed_xxx.yaml
+	StatusInputPath   string `yaml:"statusInputPath,omitempty"`   // the path of status_xxx.yaml
+	InputReplicas     int64  `yaml:"inputReplicas,omitempty"`     // the input replicas for revise operation
+	OutputResultsPath string `yaml:"outputResultsPath,omitempty"` // the path of output_xxx.yaml
+	Operation         string `yaml:"operation"`                   // the operation of resource interpreter
 }
 
 func checkInterpretationRule(t *testing.T, path string, configs []*configv1alpha1.ResourceInterpreterCustomization) {
@@ -133,7 +140,7 @@ func checkInterpretationRule(t *testing.T, path string, configs []*configv1alpha
 			if err != nil {
 				t.Fatalf("checking %s of %s, expected nil, but got: %v", rule.Name(), customization.Name, err)
 			}
-			args := interpreter.RuleArgs{Replica: input.DesiredReplica}
+			args := interpreter.RuleArgs{Replica: input.InputReplicas}
 			if input.DesiredInputPath != "" {
 				args.Desired = getObj(t, dir+"/"+strings.TrimPrefix(input.DesiredInputPath, "/"))
 			}
@@ -143,10 +150,75 @@ func checkInterpretationRule(t *testing.T, path string, configs []*configv1alpha
 			if input.StatusInputPath != "" {
 				args.Status = getAggregatedStatusItems(t, dir+"/"+strings.TrimPrefix(input.StatusInputPath, "/"))
 			}
-			if result := rule.Run(ipt, args); result.Err != nil {
+			var outputResults = make(map[string]interface{})
+			if input.OutputResultsPath != "" {
+				outputResults = getObj(t, dir+"/"+strings.TrimPrefix(input.OutputResultsPath, "/")).Object
+			}
+			result := rule.Run(ipt, args)
+			if result.Err != nil {
 				t.Fatalf("execute %s %s error: %v\n", customization.Name, rule.Name(), result.Err)
 			}
+			for _, res := range result.Results {
+				expected, ok := outputResults[res.Name]
+				if !ok {
+					// TODO(@zhzhuang-zju): Once we have a complete set of test cases, change this to t.Fatal.
+					t.Logf("no expected result for %s of %s\n", res.Name, customization.Name)
+					continue
+				}
+
+				if equal, err := deepEqual(expected, res.Value); err != nil || !equal {
+					t.Fatal("unexpected result for", res.Name, "expected:", expected, "got:", res.Value, "error:", err)
+				}
+			}
+		}
+	}
+}
+
+func deepEqual(expected, actualValue interface{}) (bool, error) {
+	expectedJSONBytes, err := json.Marshal(expected)
+	if err != nil {
+		return false, fmt.Errorf("failed to marshal expected value: %w", err)
+	}
+
+	// Handle known types for semantic comparison
+	switch typedActual := actualValue.(type) {
+	case *workv1alpha2.ReplicaRequirements:
+		var unmarshaledExpected workv1alpha2.ReplicaRequirements
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			return false, fmt.Errorf("failed to unmarshal expected JSON into ReplicaRequirements: %w", err)
+		}
+		return checker.DeepEqual(&unmarshaledExpected, typedActual), nil
+
+	case *configv1alpha1.DependentObjectReference:
+		var unmarshaledExpected configv1alpha1.DependentObjectReference
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			return false, fmt.Errorf("failed to unmarshal expected JSON into DependentObjectReference: %w", err)
+		}
+		return checker.DeepEqual(&unmarshaledExpected, typedActual), nil
+
+	case []workv1alpha2.Component:
+		var unmarshaledExpected []workv1alpha2.Component
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			return false, fmt.Errorf("failed to unmarshal expected JSON into []Component: %w", err)
+		}
+
+		return checker.DeepEqual(unmarshaledExpected, typedActual), nil
+
+	case *unstructured.Unstructured:
+		var unmarshaledExpected unstructured.Unstructured
+
+		if err := json.Unmarshal(expectedJSONBytes, &unmarshaledExpected); err != nil {
+			return false, fmt.Errorf("failed to unmarshal expected JSON into Unstructured: %w", err)
+		}
+		return checker.DeepEqual(&unmarshaledExpected, typedActual), nil
+
+	default:
+		// Fallback: marshal actualValue and do byte-wise comparison
+		actualJSON, err := json.Marshal(actualValue)
+		if err != nil {
+			return false, fmt.Errorf("failed to marshal actual value: %w", err)
 		}
+		return bytes.Equal(expectedJSONBytes, actualJSON), nil
 	}
 }