Test out early stop sort merge join to handle AS OF join?

[CTCC1]

I ran into some online benchmarks about AS OF join where in certain cases, "early stop sort merge join" can outperform UNION based AS OF join.

https://www.hopsworks.ai/post/a-spark-join-operator-for-point-in-time-correct-joins (fwiw, it mentioned tempo as the inspiration for the UNION based AS OF join)

open sourced implementations
https://github.com/Ackuq/spark-pit/blob/main/scala/src/main/scala/execution/Patterns.scala

Would be interested to see what the community / maintainers think.

[tnixon]

Thank you @CTCC1 for bringing this to our attention! We do plan to set up a more formal process for performance testing our functions and As-Of Joins are at the top of that list. We're also doing a big refactoring of the code that will make it easier to compare different implementations head-to-head, so this is very useful information, thanks!

[Tom-Newton]

I never tested this library but its implementation is quite similar to an in house implementation we previously used. Switching to spark PIT gave us a 20X speedup for the asof join stage on one of our workloads.

I will mention though that spark PIT is totally un-maintained and it does have quite a lot of bugs. I ended up creating a fork which fixed all the bugs I found https://github.com/Tom-Newton/spark-pit.

[R7L208]

Triage — March 2026

Status: Not resolved on v0.2-integration. The sort-merge join strategy in as_of_join does not implement early stopping. When the left and right series are already sorted, the algorithm could skip scanning the remainder of the right partition once a match is found, but this optimization is not present.

Jobs to be done:

• Research whether early-stop optimization is feasible within the pandas UDF join implementation
• Benchmark to determine if this is a meaningful performance improvement in practice
• If beneficial, implement early termination in the merge loop

Effort estimate: 🔴 Large (research) — requires benchmarking to justify, then algorithmic changes to the join kernel

Closure path: Low priority. Revisit after v0.2 stabilization if join performance becomes a reported bottleneck. May close as "won't fix" if benchmarks show negligible benefit.