Skip to content

[SPARK-56125][SQL] Simplify schema calculation for Merge Into Schema Evolution#55124

Open
szehon-ho wants to merge 2 commits intoapache:masterfrom
szehon-ho:cleanup/remove-dead-star-action-check
Open

[SPARK-56125][SQL] Simplify schema calculation for Merge Into Schema Evolution#55124
szehon-ho wants to merge 2 commits intoapache:masterfrom
szehon-ho:cleanup/remove-dead-star-action-check

Conversation

@szehon-ho
Copy link
Copy Markdown
Member

@szehon-ho szehon-ho commented Apr 1, 2026

What changes were proposed in this pull request?

Simplifies the schema evolution logic for MERGE INTO in two ways:

  1. Replace pre-filtering source schema with post-filtering schema changes: Instead of building a pruned source schema (sourceSchemaForSchemaEvolution) and then diffing it against the target, we now compute all schema diffs against the full source schema and filter the resulting TableChange objects to only those whose field paths fall under identity assignment paths. This eliminates the recursive filterSchema function, the isEqual helper (which was a duplicate of isSameColumnAssignment), and the isPrefix helper.

  2. Replace originalTarget/originalSource threading with onIncompatible callback in ResolveSchemaEvolution.computeSchemaChanges: The original target and source schemas were passed through every recursive call (7+ call sites across 3 methods) but only used once in the error case. A by-name error parameter replaces both, reducing parameter count and improving readability.

Also removes dead code: the containsStarAction check in sourceSchemaForSchemaEvolution was unreachable because actionsSchemaEvolutionReady returns false for star actions (via case _ => false), preventing pendingSchemaChanges from calling sourceSchemaForSchemaEvolution until ResolveReferences has expanded all star actions.

Why are the changes needed?

Code simplification and dead code removal. The existing logic was harder to follow due to the recursive schema pruning approach and redundant parameter threading.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests (the logic is semantically equivalent).

@szehon-ho szehon-ho force-pushed the cleanup/remove-dead-star-action-check branch from 389ffac to 5637683 Compare April 1, 2026 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant