Commit f8a4ac8
perf(sort-merge): cache current-row bytes in RowValues for SortPreservingMerge (#23990)
## Which issue does this PR close?
Part of the SortPreservingMerge cursor-cache work in #23840, split into
smaller PRs for easier review (per @alamb / @rluvaton). This one carries
the largest share of the win.
## Rationale for this change
`SortPreservingMerge` compares cursor heads in a loser tree; every
emitted row triggers `log2(k)` `compare` calls. For multi-column sort
keys the key is serialized into arrow `Rows` and wrapped in `RowValues`,
and each `compare` called `Rows::row(idx)` for both sides — walking
`Arc<Rows>` → `offsets[idx]` / `offsets[idx+1]` → buffer slice.
The offsets buffer (~65 KB per batch per partition, ×16 partitions) is
far too large to stay cache-resident, so those lookups typically land in
L2/L3 with a DRAM tail — and they are repeated for the same `idx` on
every compare of a stable loser-tree head, even though the answer never
changes until the cursor advances.
## What changes are included in this PR?
Cache the current row's `(ptr, len)` once per `Cursor::advance` (via the
existing `CursorValues::set_offset` hook) and read it in `compare`, so
the hot path resolves to two plain field loads plus a `memcmp` instead
of two Arc-chased offset walks.
- The pointer is into the Arc-owned buffer heap and stays valid across
struct moves (a cursor is written into a `Vec<Option<Cursor<..>>>`
slot), so `Send`/`Sync` are implemented by hand with a SAFETY note.
- `eq` / `eq_to_previous` take arbitrary cross-batch indices and
continue to index `Rows` directly (the cache only holds the current
offset).
- `compare` keeps a debug-only assert that the cache invariant holds
(indices equal the cursors' current offsets).
Only `datafusion/physical-plan/src/sorts/cursor.rs` changes. Follow-up
PRs will apply the same pattern to the single-column string cursors
(`ByteArrayValues`, `StringViewArray`) and add a null-wrapper fast path.
## Are these changes tested?
Yes:
- `test_row_values_cache_matches_rows_index` drives the cache across
every offset of a multi-row batch and asserts identical ordering to
per-row `Rows` indexing, plus the cross-batch `eq` path.
- `test_row_values_single_row_batch` covers the up-front row-0 cache and
the length snapshot.
- Verified red/green: breaking `set_offset` (skip the refresh) makes the
first test fail as expected.
- Existing `sorts::*` merge tests (83) pass.
## Are there any user-facing changes?
No API changes. The `sort_tpch10` benchmark from the combined series
(#23840) showed the multi-column queries driven by this cache: Q4
+1.23x, Q9 +1.15x, Q8 +1.13x, Q5 / Q6 / Q11 ~+1.12x; no regressions. CI
benchmark to confirm on this split.
---------
Co-authored-by: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>1 parent 35c56b0 commit f8a4ac8
1 file changed
Lines changed: 122 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
173 | 173 | | |
174 | 174 | | |
175 | 175 | | |
176 | | - | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
177 | 183 | | |
178 | 184 | | |
179 | 185 | | |
180 | 186 | | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
181 | 195 | | |
182 | 196 | | |
183 | 197 | | |
184 | 198 | | |
185 | 199 | | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
186 | 205 | | |
187 | 206 | | |
188 | 207 | | |
| |||
195 | 214 | | |
196 | 215 | | |
197 | 216 | | |
198 | | - | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
199 | 226 | | |
200 | 227 | | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
201 | 231 | | |
202 | 232 | | |
203 | 233 | | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
204 | 242 | | |
205 | 243 | | |
206 | 244 | | |
| |||
211 | 249 | | |
212 | 250 | | |
213 | 251 | | |
214 | | - | |
| 252 | + | |
215 | 253 | | |
216 | 254 | | |
217 | 255 | | |
218 | 256 | | |
219 | 257 | | |
220 | 258 | | |
| 259 | + | |
| 260 | + | |
221 | 261 | | |
222 | 262 | | |
223 | 263 | | |
| |||
227 | 267 | | |
228 | 268 | | |
229 | 269 | | |
230 | | - | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
231 | 290 | | |
232 | 291 | | |
233 | 292 | | |
| |||
638 | 697 | | |
639 | 698 | | |
640 | 699 | | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
641 | 759 | | |
642 | 760 | | |
643 | 761 | | |
| |||
0 commit comments