docs(spike): S4 PDF backend — keep PyMuPDF (staged, trigger not met)#79
Merged
Conversation
S4's harness (pdf_backend_comparison.py) existed but had one case with no table queries and PyMuPDF already at recall 1.0 — it could not test the gate. Ran staged per user's choice: curate representative table-heavy fixtures, verify PyMuPDF end-to-end via fetch_relevant first, install a torch-scale contender (Docling) only if PyMuPDF demonstrably fails. It did not. Two methodology corrections: (1) measure markdown fact recall end-to-end through fetch_relevant, not the harness's structured `table_hit` — trawl's PDF path never consumes the `tables` field; (2) representative fixtures, not reverse-engineered ones. Result: three real paper results tables (Transformer BLEU 28.4/41.8, LoRA WikiSQL/MNLI, BERT GLUE) all retrieve their cell values end-to-end through PyMuPDF. The only miss is a pathological 18-language BGE-M3 matrix — and at the extraction level even that scores recall 1.0 (facts are in PyMuPDF's markdown), so the miss is retrieval-ranking + query-vocab (ISO code ko vs "Korean"), not extraction; swapping the backend cannot fix it, and any structured-table win would land in the unused `tables` field absent table-aware chunking (separate scope). Decision: keep PyMuPDF, no structured backend, no torch dependency. Adds 4 representative table cases + 1 documented pathological matrix to pdf_backend_cases.yaml as a durable corpus; .[pdf-backends] extra and pdf_backends.py unchanged. Re-spike only if a representative PDF drops table facts end-to-end AND table-aware chunking is built. Outcome: docs/superpowers/specs/2026-06-06-pdf-backend-comparison-outcome.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
S4's harness (pdf_backend_comparison.py) existed but had one case with
no table queries and PyMuPDF already at recall 1.0 — it could not test
the gate. Ran staged per user's choice: curate representative
table-heavy fixtures, verify PyMuPDF end-to-end via fetch_relevant
first, install a torch-scale contender (Docling) only if PyMuPDF
demonstrably fails. It did not.
Two methodology corrections: (1) measure markdown fact recall
end-to-end through fetch_relevant, not the harness's structured
table_hit— trawl's PDF path never consumes thetablesfield; (2)representative fixtures, not reverse-engineered ones.
Result: three real paper results tables (Transformer BLEU 28.4/41.8,
LoRA WikiSQL/MNLI, BERT GLUE) all retrieve their cell values end-to-end
through PyMuPDF. The only miss is a pathological 18-language BGE-M3
matrix — and at the extraction level even that scores recall 1.0
(facts are in PyMuPDF's markdown), so the miss is retrieval-ranking +
query-vocab (ISO code ko vs "Korean"), not extraction; swapping the
backend cannot fix it, and any structured-table win would land in the
unused
tablesfield absent table-aware chunking (separate scope).Decision: keep PyMuPDF, no structured backend, no torch dependency.
Adds 4 representative table cases + 1 documented pathological matrix to
pdf_backend_cases.yaml as a durable corpus; .[pdf-backends] extra and
pdf_backends.py unchanged. Re-spike only if a representative PDF drops
table facts end-to-end AND table-aware chunking is built. Outcome:
docs/superpowers/specs/2026-06-06-pdf-backend-comparison-outcome.md
🤖 Generated with Claude Code