[lowering] Support SDPA where the value head dim differs from query/key - #117
Merged
gokulkrishna98 merged 2 commits intoSep 25, 2026
Merged
Conversation
The SDPA lowering assumed the output had the same type as query (D_v == D_k) and asserted on it, so attention with a differing value head dim failed to convert even though eager PyTorch handles it. The result type is now derived from the value head dim. The composite op cannot represent an output head dim other than the query's, so it is only emitted when D_v is provably equal to D_k; otherwise the decomposition is lowered directly. The default 1/sqrt(D_k) scale still uses the query head dim.
gokulkrishna98
marked this pull request as ready for review
September 21, 2026 18:15
jakesabathia2
approved these changes
Sep 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scaled_dot_product_attentionlowering assumed the output had the same type asquery(D_v == D_k) and asserted on it, so attention with a differing value head dim failed to convert even though eager PyTorch handles it.[B, n_q_heads, T_q, D_v]) in_sdpa_decomposeand in all three rank paths ofreplace_sdpa.D_vis provably equal toD_k; its interface cannot represent another output head dim, soD_v != D_kand dynamic head dims lower the decomposition directly.D_k; it now uses the query's leading dims with the value head dim.1/sqrt(D_k)scale still uses the query head dim.Repro
Asserted before this change; converts now.
Tests
TestSDPA::test_sdpa_value_head_dim_differs— rank 4 and rank 6, causal / explicit-mask / maskless, static and dynamic, GQA.TestSDPA::test_sdpa_rank3_value_head_dim_differs,TestSDPA::test_sdpa_dynamic_head_dim.D_v == D_kand thatD_v != D_kand dynamic head dims lower inline.pytest tests/ops/test_ops.py -k SDPA tests/ops/test_ops_ir.py tests/composite_ops/test_sdpa.py— 2083 passed.