Open
Description
While working on #1887 I looked at logic related to wrapping nodes with CopyExec (mostly for unpacking dictionaries. We have logic in planner.rs in several places that determines when to insert CopyExec nodes in the native physical plan. We should move that logic to the Spark side and define a new CopyExec node that we can translate 1:1 on the native side. This has a few benefits:
- CopyExec nodes will show up in the Spark UI representation of the plan
- More logic about plan translation is consolidated on the Spark side, which offers benefits of its own:
- Easier to understand what the final native plan will be during Comet rule execution on the Spark side
- Easier to cache serialized native plans
- Reduce redundant work across Spark workers in native code
Metadata
Metadata
Assignees
Labels
No labels