Skip to content

docs(spike): S3 query-aware fusion — validate as already-shipped + neutral#78

Merged
dongwhee merged 1 commit into
developfrom
spike/s3-query-aware-fusion-validation
Jun 6, 2026
Merged

docs(spike): S3 query-aware fusion — validate as already-shipped + neutral#78
dongwhee merged 1 commit into
developfrom
spike/s3-query-aware-fusion-validation

Conversation

@dongwhee

@dongwhee dongwhee commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

S3's feature (query-type weighted RRF: _classify_query +
_fusion_weights, identifier -> bm25 up-weight) was already implemented,
unit-tested, and default-on — shipped in b81d57b (2026-04-27, before
the roadmap was written) and live since hybrid default-on (#58). Same
stale-roadmap pattern as R1/R2/R4/C7. The missing piece was the
isolated A/B the gate calls for.

Adds benchmarks/query_aware_fusion_ab.py: per reader_comparison case
(3 identifier / 3 concept — real identifier coverage, so not vacuous),
fetch end-to-end twice flipping only _fusion_weights between the
shipped query-aware values and equal weights via monkeypatch (no
production toggle); graded metric = top-1 identity + first-fact rank,
measured rerank-on and rerank-off.

Result: end-to-end neutral (net facts delta +0, top1 changed 1/6
marginal). At the pure-fusion layer the identifier cases — the
weighting's target — are identical between weighted and equal RRF
(rankers concur at top, so the 5x swing can't reorder; the lever does
fire elsewhere, confirming this is real invariance not a dead patch).
Gate (coding net >= +1) not met.

Decision: keep as-is. No toggle (unrequested config, no rollback need),
no simplification (touches the load-bearing retrieval path for zero
end-to-end benefit; tested + harmless), no weight tuning (the
structure, not the magnitude, makes it inert here). S3 closed as
already-implemented + validated-neutral; retrieval.py unchanged. Design

  • outcome + cited-paper reconciliation in
    docs/superpowers/specs/2026-06-06-query-aware-fusion-validation.md.

🤖 Generated with Claude Code

…utral

S3's feature (query-type weighted RRF: _classify_query +
_fusion_weights, identifier -> bm25 up-weight) was already implemented,
unit-tested, and default-on — shipped in b81d57b (2026-04-27, before
the roadmap was written) and live since hybrid default-on (#58). Same
stale-roadmap pattern as R1/R2/R4/C7. The missing piece was the
isolated A/B the gate calls for.

Adds benchmarks/query_aware_fusion_ab.py: per reader_comparison case
(3 identifier / 3 concept — real identifier coverage, so not vacuous),
fetch end-to-end twice flipping only _fusion_weights between the
shipped query-aware values and equal weights via monkeypatch (no
production toggle); graded metric = top-1 identity + first-fact rank,
measured rerank-on and rerank-off.

Result: end-to-end neutral (net facts delta +0, top1 changed 1/6
marginal). At the pure-fusion layer the identifier cases — the
weighting's target — are identical between weighted and equal RRF
(rankers concur at top, so the 5x swing can't reorder; the lever does
fire elsewhere, confirming this is real invariance not a dead patch).
Gate (coding net >= +1) not met.

Decision: keep as-is. No toggle (unrequested config, no rollback need),
no simplification (touches the load-bearing retrieval path for zero
end-to-end benefit; tested + harmless), no weight tuning (the
structure, not the magnitude, makes it inert here). S3 closed as
already-implemented + validated-neutral; retrieval.py unchanged. Design
+ outcome + cited-paper reconciliation in
docs/superpowers/specs/2026-06-06-query-aware-fusion-validation.md.
@dongwhee dongwhee merged commit 550ee24 into develop Jun 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant