docs(spike): S3 query-aware fusion — validate as already-shipped + neutral#78
Merged
Merged
Conversation
…utral S3's feature (query-type weighted RRF: _classify_query + _fusion_weights, identifier -> bm25 up-weight) was already implemented, unit-tested, and default-on — shipped in b81d57b (2026-04-27, before the roadmap was written) and live since hybrid default-on (#58). Same stale-roadmap pattern as R1/R2/R4/C7. The missing piece was the isolated A/B the gate calls for. Adds benchmarks/query_aware_fusion_ab.py: per reader_comparison case (3 identifier / 3 concept — real identifier coverage, so not vacuous), fetch end-to-end twice flipping only _fusion_weights between the shipped query-aware values and equal weights via monkeypatch (no production toggle); graded metric = top-1 identity + first-fact rank, measured rerank-on and rerank-off. Result: end-to-end neutral (net facts delta +0, top1 changed 1/6 marginal). At the pure-fusion layer the identifier cases — the weighting's target — are identical between weighted and equal RRF (rankers concur at top, so the 5x swing can't reorder; the lever does fire elsewhere, confirming this is real invariance not a dead patch). Gate (coding net >= +1) not met. Decision: keep as-is. No toggle (unrequested config, no rollback need), no simplification (touches the load-bearing retrieval path for zero end-to-end benefit; tested + harmless), no weight tuning (the structure, not the magnitude, makes it inert here). S3 closed as already-implemented + validated-neutral; retrieval.py unchanged. Design + outcome + cited-paper reconciliation in docs/superpowers/specs/2026-06-06-query-aware-fusion-validation.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
S3's feature (query-type weighted RRF: _classify_query +
_fusion_weights, identifier -> bm25 up-weight) was already implemented,
unit-tested, and default-on — shipped in b81d57b (2026-04-27, before
the roadmap was written) and live since hybrid default-on (#58). Same
stale-roadmap pattern as R1/R2/R4/C7. The missing piece was the
isolated A/B the gate calls for.
Adds benchmarks/query_aware_fusion_ab.py: per reader_comparison case
(3 identifier / 3 concept — real identifier coverage, so not vacuous),
fetch end-to-end twice flipping only _fusion_weights between the
shipped query-aware values and equal weights via monkeypatch (no
production toggle); graded metric = top-1 identity + first-fact rank,
measured rerank-on and rerank-off.
Result: end-to-end neutral (net facts delta +0, top1 changed 1/6
marginal). At the pure-fusion layer the identifier cases — the
weighting's target — are identical between weighted and equal RRF
(rankers concur at top, so the 5x swing can't reorder; the lever does
fire elsewhere, confirming this is real invariance not a dead patch).
Gate (coding net >= +1) not met.
Decision: keep as-is. No toggle (unrequested config, no rollback need),
no simplification (touches the load-bearing retrieval path for zero
end-to-end benefit; tested + harmless), no weight tuning (the
structure, not the magnitude, makes it inert here). S3 closed as
already-implemented + validated-neutral; retrieval.py unchanged. Design
docs/superpowers/specs/2026-06-06-query-aware-fusion-validation.md.
🤖 Generated with Claude Code