diff --git a/CHANGELOG.md b/CHANGELOG.md index 2ec2d78..5d85a45 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -93,6 +93,14 @@ _No changes yet._ suffix-less API/RSS paths like `news.ycombinator.com/rss` still route through raw passthrough (`fetcher_used: passthrough-probed`). +- **Pluggable PDF extraction backends** + (`src/trawl/fetchers/pdf_backends.py`) — the PDF fetcher now routes + through a backend layer (`pymupdf` / `markitdown` / `unstructured` / + `docling` / `mineru`, heavy parsers behind lazy imports). The + production default is unchanged (PyMuPDF); the layer powers the R5 + backend comparison harness, which concluded keep-PyMuPDF (see + Tests / research below). + ### Fixed - **Wikipedia fetcher heading preservation** — modern MediaWiki HTML diff --git a/CLAUDE.md b/CLAUDE.md index 9878c09..cc39f1e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -37,8 +37,9 @@ trawl directory. Humans should read `README.md` first, then - **Longform retrieval cost (default on, 2026-04-22)**: `TRAWL_CHUNK_BUDGET` default flipped from `0` to `100` after re-validating on curl.se manpage (275 KB / 760 chunks, p95 25149 ms → 3065 ms). Parity 15/15 - + agent_patterns coding 23/24 (pre-existing unrelated `arxiv_pdf_lora` - fetcher fail) preserved. Opt out via `TRAWL_CHUNK_BUDGET=0`. + + agent_patterns coding 23/24 at that time preserved (the unrelated + `arxiv_pdf_lora` fetcher flake has since recovered — coding is 24/24 + as of v0.4.5). Opt out via `TRAWL_CHUNK_BUDGET=0`. ### What a new session should do first