Skip to content

[lowering] Support SDPA where the value head dim differs from query/key - #117

Merged
gokulkrishna98 merged 2 commits into
apple:mainfrom
gokulkrishna98:fix-sdpa-value-head-dim
Sep 25, 2026
Merged

gokulkrishna98 merged 2 commits into
apple:mainfrom
gokulkrishna98:fix-sdpa-value-head-dim

Conversation

@gokulkrishna98

Copy link
Copy Markdown
Contributor

scaled_dot_product_attention lowering assumed the output had the same type as query (D_v == D_k) and asserted on it, so attention with a differing value head dim failed to convert even though eager PyTorch handles it.

  • Derive the result type from the value head dim ([B, n_q_heads, T_q, D_v]) in _sdpa_decompose and in all three rank paths of replace_sdpa.
  • Emit the composite op only when D_v is provably equal to D_k; its interface cannot represent another output head dim, so D_v != D_k and dynamic head dims lower the decomposition directly.
  • The rank > 4 restore path rebuilt the shape from the query, which carries D_k; it now uses the query's leading dims with the value head dim.
  • The default 1/sqrt(D_k) scale still uses the query head dim.

Repro

class SDPA(nn.Module):
    def forward(self, q, k, v):
        return F.scaled_dot_product_attention(q, k, v, enable_gqa=True)

# q: [8, 4, 1, 32], k: [8, 1, 36, 32], v: [8, 1, 36, 24]

Asserted before this change; converts now.

Tests

  • TestSDPA::test_sdpa_value_head_dim_differs — rank 4 and rank 6, causal / explicit-mask / maskless, static and dynamic, GQA.
  • TestSDPA::test_sdpa_rank3_value_head_dim_differs, TestSDPA::test_sdpa_dynamic_head_dim.
  • IR tests asserting the composite is emitted for D_v == D_k and that D_v != D_k and dynamic head dims lower inline.

pytest tests/ops/test_ops.py -k SDPA tests/ops/test_ops_ir.py tests/composite_ops/test_sdpa.py — 2083 passed.

The SDPA lowering assumed the output had the same type as query
(D_v == D_k) and asserted on it, so attention with a differing value
head dim failed to convert even though eager PyTorch handles it.

The result type is now derived from the value head dim. The composite
op cannot represent an output head dim other than the query's, so it is
only emitted when D_v is provably equal to D_k; otherwise the
decomposition is lowered directly. The default 1/sqrt(D_k) scale still
uses the query head dim.
@gokulkrishna98 gokulkrishna98 changed the title Support SDPA where the value head dim differs from query/key [lowering] Support SDPA where the value head dim differs from query/key Sep 21, 2026
@gokulkrishna98
gokulkrishna98 marked this pull request as ready for review September 21, 2026 18:15
@gokulkrishna98
gokulkrishna98 merged commit 65ec629 into apple:main Sep 25, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants