WIP: Attach `__ROW_VERSION` pseudo-field to types if the record meta-data stores versions #3800

alecgrieser · 2025-12-10T11:46:28Z

To address #3796, this pushes down the resolution of the pseudo-field __ROW_VERSION from within the plan generator to closer to the fetch. The goal there is to be able to then treat the __ROW_VERSION as a normal field elsewhere in planning, which would then allow us to remove the VersionValue from the planner, along with its somewhat odd semantics.

… stores versions

…bout-versions

…bably won't run anymore, but it was an easy enough fix - Make the meta-data translation lazy. This avoids some stack overflow problems if the schema doesn't form a graph, something that we don't really support in relational but that we have tests of in the Record Layer - Fix up tests so that they use the new format for the base record type (with pseudo-fields) - Version tests in the RL core tests now refer to the pseudo-field

…plans affected by merge skew

…hema called __ROW_VERSION

…able force continuations on some queries, though only starting with the current version This fixes FoundationDB#3734.

github-actions · 2025-12-17T18:23:39Z

📊 Metrics Diff Analysis Report

Summary

New queries: 37
Dropped queries: 0
Plan changed + metrics changed: 63
Plan unchanged + metrics changed: 52

ℹ️ About this analysis

This automated analysis compares query planner metrics between the base branch and this PR. It categorizes changes into:

New queries: Queries added in this PR
Dropped queries: Queries removed in this PR. These should be reviewed to ensure we are not losing coverage.
Plan changed + metrics changed: The query plan has changed along with planner metrics.
Metrics only changed: Same plan but different metrics

The last category in particular may indicate planner regressions that should be investigated.

New Queries

Count of new queries by file:

yaml-tests/src/test/resources/pseudo-field-clash.metrics.yaml: 20
yaml-tests/src/test/resources/versions-tests.metrics.yaml: 17

Plan and Metrics Changed

These queries experienced both plan and metrics changes. This generally indicates that there was some planner change
that means the planning for this query may be substantially different. Some amount of query plan metrics change is expected,
but the reviewer should still validate that these changes are not excessive.

Total: 63 queries

Statistical Summary (Plan and Metrics Changed)

task_count:

Average change: +20.6
Average regression: +54.0
Median change: +31
Median regression: +31
Standard deviation: 189.4
Standard deviation of regressions: 44.4
Range: -1055 to +222
Range of regressions: +6 to +222
Queries changed: 63
Queries regressed: 61

transform_count:

Average change: +3.7
Average regression: +9.7
Median change: +4
Median regression: +4
Standard deviation: 31.8
Standard deviation of regressions: 11.2
Range: -171 to +63
Range of regressions: +4 to +63
Queries changed: 63
Queries regressed: 58

transform_yield_count:

Average change: +1.5
Average regression: +3.2
Median change: +2
Median regression: +2
Standard deviation: 9.3
Standard deviation of regressions: 2.2
Range: -56 to +13
Range of regressions: +2 to +13
Queries changed: 60
Queries regressed: 57

insert_new_count:

Average change: +1.6
Average regression: +5.6
Median change: +3
Median regression: +3
Standard deviation: 22.7
Standard deviation of regressions: 4.5
Range: -137 to +23
Range of regressions: +1 to +23
Queries changed: 63
Queries regressed: 61

insert_reused_count:

Average change: -2.1
Median change: -1
Standard deviation: 4.1
Range: -24 to -1
Queries changed: 59
No regressions! 🎉

Significant Regressions (Plan and Metrics Changed)

There were 13 outliers detected. Outlier queries have a significant regression in at least one field. Statistically, this represents either an increase of more than two standard deviations above the mean or a large absolute increase (e.g., 100).

yaml-tests/src/test/resources/like.metrics.yaml:2: EXPLAIN select * from C WHERE C2 LIKE '%'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:13: EXPLAIN select * from C WHERE C2 LIKE 'a%'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:24: EXPLAIN select * from C WHERE C2 LIKE 'ap%'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:35: EXPLAIN select * from C WHERE C2 LIKE 'a%l%'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:46: EXPLAIN select * from C WHERE C2 LIKE 'ca%'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:57: EXPLAIN select * from C WHERE C2 LIKE 'ca_al'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:68: EXPLAIN select * from C WHERE C2 LIKE 'ca%al'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:79: EXPLAIN select * from C WHERE C2 LIKE 'ca_al%'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/like.metrics.yaml:90: EXPLAIN select * from C WHERE C2 LIKE 'ca%al%'
- old explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | FETCH
- new explain: COVERING(C2 <,> -> [C1: KEY[2], C2: KEY[0]]) | FILTER _.C2 LIKE @c8 ESCAPE 'null' | MAP (_.C1 AS C1, _.C2 AS C2)
- task_count: 338 -> 450 (+112)
- transform_count: 79 -> 95 (+16)
- transform_yield_count: 29 -> 35 (+6)
- insert_new_count: 38 -> 48 (+10)
- insert_reused_count: 5 -> 3 (-2)
yaml-tests/src/test/resources/null-extraction-tests.metrics.yaml:14: EXPLAIN select * from B where b3 = 4 and b2 = 'b' and (? is null or b1 < 20)
- old explain: COVERING(I1 [EQUALS promote(@c8 AS LONG), EQUALS promote(@c12 AS STRING)] -> [B1: KEY[3], B2: KEY[1], B3: KEY[0]]) | FILTER @c15 IS_NULL OR _.B1 LESS_THAN promote(@c21 AS INT) | FETCH
- new explain: COVERING(I1 [EQUALS promote(@c8 AS LONG), EQUALS promote(@c12 AS STRING)] -> [B1: KEY:[3], B2: KEY:[1], B3: KEY:[0]]) | FILTER @c15 IS_NULL OR _.B1 LESS_THAN promote(@c21 AS INT) | FETCH
- task_count: 1426 -> 1616 (+190)
- transform_count: 266 -> 300 (+34)
- transform_yield_count: 84 -> 93 (+9)
- insert_new_count: 175 -> 198 (+23)
yaml-tests/src/test/resources/uuid.metrics.yaml:453: EXPLAIN select * from tc order by b
- old explain: ISCAN(TC1 <,>)
- new explain: COVERING(TC1 <,> -> [A: KEY[2], B: KEY[0], C: VALUE[0]]) | MAP (_.A AS A, _.B AS B, _.C AS C)
- task_count: 277 -> 389 (+112)
- transform_count: 74 -> 94 (+20)
- transform_yield_count: 34 -> 42 (+8)
- insert_new_count: 25 -> 39 (+14)
- insert_reused_count: 8 -> 4 (-4)
yaml-tests/src/test/resources/valid-identifiers.metrics.yaml:367: EXPLAIN select struct "x$$" ("foo.tableA".*) from "foo.tableA"
- old explain: ISCAN(foo.tableA.idx3 <,>) | MAP (_ AS _0)
- new explain: COVERING(foo.tableA.idx <,> -> [foo__2tableA__2A1: KEY[0], foo__2tableA__2A2: KEY[1], foo__2tableA__2A3: KEY[2]]) | MAP ((_.foo.tableA.A1 AS foo.tableA.A1, _.foo.tableA.A2 AS foo.tableA.A2, _.foo.tableA.A3 AS foo.tableA.A3) AS _0)
- task_count: 301 -> 523 (+222)
- transform_count: 75 -> 138 (+63)
- transform_yield_count: 37 -> 50 (+13)
- insert_new_count: 35 -> 54 (+19)
- insert_reused_count: 6 -> 3 (-3)
yaml-tests/src/test/resources/versions-tests.metrics.yaml:14: EXPLAIN select t1.* from t1 where col1 = 10;
- old explain: ISCAN(I1 [EQUALS promote(@c10 AS LONG)])
- new explain: ISCAN(I1 [EQUALS promote(@c10 AS LONG)]) | MAP (_.ID AS ID, _.COL1 AS COL1, _.COL2 AS COL2)
- task_count: 571 -> 723 (+152)
- transform_count: 130 -> 180 (+50)
- transform_yield_count: 59 -> 58 (-1)
- insert_new_count: 63 -> 79 (+16)
- insert_reused_count: 8 -> 6 (-2)

Minor Changes (Plan and Metrics Changed)

In addition, there were 50 queries with minor changes.

Only Metrics Changed

These queries experienced only metrics changes without any plan changes. If these metrics have substantially changed,
then a planner change has been made which affects planner performance but does not correlate with any new outcomes,
which could indicate a regression.

Total: 52 queries

Statistical Summary (Only Metrics Changed)

task_count:

Average change: +52.9
Average regression: +52.9
Median change: +32
Median regression: +32
Standard deviation: 34.5
Standard deviation of regressions: 34.5
Range: +1 to +131
Range of regressions: +1 to +131
Queries changed: 52
Queries regressed: 52

transform_count:

Average change: +7.8
Average regression: +7.8
Median change: +6
Median regression: +6
Standard deviation: 5.3
Standard deviation of regressions: 5.3
Range: +2 to +22
Range of regressions: +2 to +22
Queries changed: 52
Queries regressed: 52

transform_yield_count:

Average change: +2.6
Average regression: +2.9
Median change: +2
Median regression: +2
Standard deviation: 1.8
Standard deviation of regressions: 1.6
Range: -1 to +6
Range of regressions: +1 to +6
Queries changed: 45
Queries regressed: 42

insert_new_count:

Average change: +5.3
Average regression: +5.3
Median change: +5
Median regression: +5
Standard deviation: 2.5
Standard deviation of regressions: 2.5
Range: +3 to +10
Range of regressions: +3 to +10
Queries changed: 49
Queries regressed: 49

insert_reused_count:

Average change: -1.8
Average regression: +2.0
Median change: -1
Median regression: +2
Standard deviation: 1.8
Standard deviation of regressions: 0.0
Range: -5 to +2
Range of regressions: +2 to +2
Queries changed: 41
Queries regressed: 3

Significant Regressions (Only Metrics Changed)

There were 2 outliers detected. Outlier queries have a significant regression in at least one field. Statistically, this represents either an increase of more than two standard deviations above the mean or a large absolute increase (e.g., 100).

yaml-tests/src/test/resources/distinct-from.metrics.yaml:46: EXPLAIN select * from t1 WHERE null is distinct from null
- explain: COVERING(I1 <,> -> [COL1: KEY[0], ID: KEY[2]]) | FILTER NULL IS_DISTINCT_FROM NULL | FETCH
- task_count: 473 -> 604 (+131)
- transform_count: 112 -> 134 (+22)
- transform_yield_count: 48 -> 54 (+6)
- insert_new_count: 56 -> 65 (+9)
- insert_reused_count: 8 -> 3 (-5)
yaml-tests/src/test/resources/distinct-from.metrics.yaml:57: EXPLAIN select * from t1 WHERE 10 is distinct from 10
- explain: COVERING(I1 <,> -> [COL1: KEY[0], ID: KEY[2]]) | FILTER @c6 IS_DISTINCT_FROM @c6 | FETCH
- task_count: 473 -> 604 (+131)
- transform_count: 112 -> 134 (+22)
- transform_yield_count: 48 -> 54 (+6)
- insert_new_count: 56 -> 65 (+9)
- insert_reused_count: 8 -> 3 (-5)

Minor Changes (Only Metrics Changed)

In addition, there were 50 queries with minor changes.

wip: attach ROW_VERSION pseudo-field to types if the record meta-data…

667c5b7

… stores versions

alecgrieser added Draft bug fix Change that fixes a bug labels Dec 10, 2025

alecgrieser added 11 commits December 10, 2025 15:35

Update star expansion so that it always asks for the fields explicitly

ea2fa4a

get versions-tests.yamsql to a working state

43b2db5

Merge remote-tracking branch 'upstream/main' into 03796-be-explicit-a…

85db0ef

…bout-versions

use more appropriate type for data modification plans ; fix covering …

03e6ef5

…plans affected by merge skew

update null-extraction-tests.yamsql to reflect new explains

ecdf04b

remove now unused private method

d038dbb

add test for what happens if there is a column in the user-defined sc…

a4aec01

…hema called __ROW_VERSION

Remove commented out code. As this fixes FoundationDB#3734, we can en…

972a09c

…able force continuations on some queries, though only starting with the current version This fixes FoundationDB#3734.

Some clean up to make things more uniform or avoid useless changes

ed8fa1f

align error message of casting in the different match candidates

e712692

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Attach `__ROW_VERSION` pseudo-field to types if the record meta-data stores versions #3800

WIP: Attach `__ROW_VERSION` pseudo-field to types if the record meta-data stores versions #3800

Uh oh!

alecgrieser commented Dec 10, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

WIP: Attach __ROW_VERSION pseudo-field to types if the record meta-data stores versions #3800

Are you sure you want to change the base?

WIP: Attach __ROW_VERSION pseudo-field to types if the record meta-data stores versions #3800

Uh oh!

Conversation

alecgrieser commented Dec 10, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

📊 Metrics Diff Analysis Report

Summary

New Queries

Plan and Metrics Changed

Statistical Summary (Plan and Metrics Changed)

Significant Regressions (Plan and Metrics Changed)

Minor Changes (Plan and Metrics Changed)

Only Metrics Changed

Statistical Summary (Only Metrics Changed)

Significant Regressions (Only Metrics Changed)

Minor Changes (Only Metrics Changed)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

WIP: Attach `__ROW_VERSION` pseudo-field to types if the record meta-data stores versions #3800

WIP: Attach `__ROW_VERSION` pseudo-field to types if the record meta-data stores versions #3800