Skip to content

ENH: Add RangeIndex union fast-path in union_indexes - #66293

Open
rajat315315 wants to merge 7 commits into
pandas-dev:mainfrom
rajat315315:perf/rangeindex-union-concat
Open

ENH: Add RangeIndex union fast-path in union_indexes#66293
rajat315315 wants to merge 7 commits into
pandas-dev:mainfrom
rajat315315:perf/rangeindex-union-concat

Conversation

@rajat315315

@rajat315315 rajat315315 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor
  • Resolves / Closes: PERF: union_indexes is slow for homogeneous RangeIndexes during DataFrame concatenation #66292

  • Summary of changes:

    • Added a fast-path inside union_indexes when all input indexes are RangeIndex instances.
    • Deduplicates incoming RangeIndexes to avoid redundant O(N) iterative set operations (e.g. when unioning identical column indices during pd.concat).
    • Automatically merges contiguous/overlapping RangeIndexes with matching steps into a single RangeIndex without calling union iteratively.
    • Reconciles names to None if they do not match.
    • Added unit tests in pandas/tests/indexes/ranges/test_setops.py to cover identical, contiguous, overlapping, non-contiguous, empty, and name-reconciliation cases.
  • Performance Impact:
    Benchmark results for concatenating 2,000 DataFrames (average of 5 runs):

    • Identical columns (axis=0 concat): ~37% speedup (0.0412s -> 0.0260s)
    • Identical rows (axis=1 concat): ~42% speedup (0.0329s -> 0.0191s)
  • closes #xxxx (Replace xxxx with the GitHub issue number)

  • Tests added and passed if fixing a bug or adding a new feature

  • All code checks passed.

  • Added type annotations to new arguments/methods/functions.

  • Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

  • I have reviewed and followed all the contribution guidelines

  • If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

@jbrockmendel

Copy link
Copy Markdown
Member

can you double-check this on

rngs1 = [range(0, 10), range(2, 5)]
rngs2 = [range(0, 10), range(0, 5)]
rngs3 = [range(0, 10, 2), range(2, 6, 2)]

@rajat315315

rajat315315 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Apologies, missed one case!

The initial implementation sorted the list of non-empty ranges by their start values (or start values descending if step < 0) but assumed the last sorted range had the largest stop value. If one range was entirely contained inside another, or if there were duplicate start values with different stop values, the final range bound returned was incorrect.

I have fixed it and also added new test-cases to cover these as well.

@rajat315315

rajat315315 commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Benchmark Results (10,000 Runs)
Here is a performance comparison between the Corrected Fast-Path and the standard Slow Path / Fallback (Index.union iteration):

  • rngs1:
    • Fast Path: 0.096769 seconds
    • Slow Path: 0.109222 seconds
    • Speedup: 1.13x
  • rngs2:
    • Fast Path: 0.082105 seconds
    • Slow Path: 0.125456 seconds
    • Speedup: 1.53x
  • rngs3:
    • Fast Path: 0.102104 seconds
    • Slow Path: 0.116838 seconds
    • Speedup: 1.14x

Comment thread doc/source/whatsnew/v3.1.0.rst Outdated
Comment thread pandas/tests/indexes/ranges/test_setops.py Outdated
Comment thread pandas/tests/indexes/ranges/test_setops.py Outdated
@jbrockmendel

Copy link
Copy Markdown
Member

please have your favorite AI fuzz this for cases you missed

@rajat315315
rajat315315 requested a review from jbrockmendel July 16, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants