Skip to content

docs(spike): S4 PDF backend — keep PyMuPDF (staged, trigger not met)#79

Merged
dongwhee merged 1 commit into
developfrom
spike/s4-pdf-backend-comparison
Jun 6, 2026
Merged

docs(spike): S4 PDF backend — keep PyMuPDF (staged, trigger not met)#79
dongwhee merged 1 commit into
developfrom
spike/s4-pdf-backend-comparison

Conversation

@dongwhee

@dongwhee dongwhee commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

S4's harness (pdf_backend_comparison.py) existed but had one case with
no table queries and PyMuPDF already at recall 1.0 — it could not test
the gate. Ran staged per user's choice: curate representative
table-heavy fixtures, verify PyMuPDF end-to-end via fetch_relevant
first, install a torch-scale contender (Docling) only if PyMuPDF
demonstrably fails. It did not.

Two methodology corrections: (1) measure markdown fact recall
end-to-end through fetch_relevant, not the harness's structured
table_hit — trawl's PDF path never consumes the tables field; (2)
representative fixtures, not reverse-engineered ones.

Result: three real paper results tables (Transformer BLEU 28.4/41.8,
LoRA WikiSQL/MNLI, BERT GLUE) all retrieve their cell values end-to-end
through PyMuPDF. The only miss is a pathological 18-language BGE-M3
matrix — and at the extraction level even that scores recall 1.0
(facts are in PyMuPDF's markdown), so the miss is retrieval-ranking +
query-vocab (ISO code ko vs "Korean"), not extraction; swapping the
backend cannot fix it, and any structured-table win would land in the
unused tables field absent table-aware chunking (separate scope).

Decision: keep PyMuPDF, no structured backend, no torch dependency.
Adds 4 representative table cases + 1 documented pathological matrix to
pdf_backend_cases.yaml as a durable corpus; .[pdf-backends] extra and
pdf_backends.py unchanged. Re-spike only if a representative PDF drops
table facts end-to-end AND table-aware chunking is built. Outcome:
docs/superpowers/specs/2026-06-06-pdf-backend-comparison-outcome.md

🤖 Generated with Claude Code

S4's harness (pdf_backend_comparison.py) existed but had one case with
no table queries and PyMuPDF already at recall 1.0 — it could not test
the gate. Ran staged per user's choice: curate representative
table-heavy fixtures, verify PyMuPDF end-to-end via fetch_relevant
first, install a torch-scale contender (Docling) only if PyMuPDF
demonstrably fails. It did not.

Two methodology corrections: (1) measure markdown fact recall
end-to-end through fetch_relevant, not the harness's structured
`table_hit` — trawl's PDF path never consumes the `tables` field; (2)
representative fixtures, not reverse-engineered ones.

Result: three real paper results tables (Transformer BLEU 28.4/41.8,
LoRA WikiSQL/MNLI, BERT GLUE) all retrieve their cell values end-to-end
through PyMuPDF. The only miss is a pathological 18-language BGE-M3
matrix — and at the extraction level even that scores recall 1.0
(facts are in PyMuPDF's markdown), so the miss is retrieval-ranking +
query-vocab (ISO code ko vs "Korean"), not extraction; swapping the
backend cannot fix it, and any structured-table win would land in the
unused `tables` field absent table-aware chunking (separate scope).

Decision: keep PyMuPDF, no structured backend, no torch dependency.
Adds 4 representative table cases + 1 documented pathological matrix to
pdf_backend_cases.yaml as a durable corpus; .[pdf-backends] extra and
pdf_backends.py unchanged. Re-spike only if a representative PDF drops
table facts end-to-end AND table-aware chunking is built. Outcome:
docs/superpowers/specs/2026-06-06-pdf-backend-comparison-outcome.md
@dongwhee dongwhee merged commit a24d03f into develop Jun 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant