feat(tools): research_to_file — two-phase research/synthesis pipeline tool - #22
Open
kuroko1t wants to merge 2 commits into
Open
feat(tools): research_to_file — two-phase research/synthesis pipeline tool#22kuroko1t wants to merge 2 commits into
kuroko1t wants to merge 2 commits into
Conversation
… tool A single tool that wraps the validated phase 1 (gather sources) + phase 2 (single-shot synthesis) pipeline so the user-facing UX stays Claude- compatible — describe the task in natural language; the LLM picks the tool; orchestration is in Rust code, not in the LLM's recipe-following. Background. Dog-food on 2026-05-22 (see convergence_triangulated_ capabilities_problem.md) showed three orthogonal interventions — prompt pressure, model swap to qwen3.5:9b, pre-scaffolded TODO file — all produced 0 write_file calls on q3 for the same JP research task. The phase 1 / phase 2 split was validated independently: with an explicit "save sources, do not summarise" directive the subagent reliably reached write_file × 3, and a single-shot synthesis call produced a coherent deliverable with URLs cited inline. Wrapping the split in a tool puts the only LLM decision at "use research_to_file or not", which is the same tool-selection class q3 has demonstrably gotten right for web_search / web_fetch / write_file. Tool surface: - Args: topic, output_file, instructions, source_count (default 3). - Phase 1 spawns a subagent that searches → fetches → write_files exactly N source-K.md files into <stem>.sources/. Bounded by "do NOT summarise" rule so the subagent's convergence problem reduces to a counting condition the model can satisfy. - Phase 2 reads the source files back, fires a single non-tool LLM call with the user's verbatim instructions, writes the deliverable to output_file. Stats include the synthesis call's token usage. - Workspace persists next to the deliverable (debuggable + re-synthesis without re-fetching). - Same Moderate risk class as write_file so approval gates still apply in non-yolo modes. The dispatch lives in Agent because it needs &mut self for the subagent and the LLM client. The Tool::execute path returns an explicit error rather than no-op'ing, so misuse outside the agent loop is loud. 740 lib tests pass (+5 for the new tool's schema, description, risk, and standalone-call error). cargo fmt + clippy --release -D warnings clean. Live smoke against ollama not run in this commit — defer to a follow-up validation pass.
Live smoke of v1 found the subagent inside phase 1 reproduces exactly the convergence problem the tool was meant to bypass: 30 iterations, 0 write_file calls, 37 fetches against hallucinated URLs. See auto-memory pr22_v1_subagent_phase1_failed.md. Wrapping the pipeline in a tool moved the problem one level deeper instead of solving it. v2 takes the agent loop out of phase 1 entirely. Rust drives: - one web_search call (via a new pub(crate) helper search_top_urls in web_search.rs that returns structured (title, url) pairs instead of the formatted markdown that the Tool::execute path returns), - a fetch-and-extract loop over the top N candidate URLs (each web_fetch with prompt is a single bounded LLM call to extract facts from one page), - std::fs::write of each source-K.md. LLM judgement is preserved only where it's well-scoped: per-fetch extraction and the final synthesis. Both were independently validated on 2026-05-22 (phase1-validation wrote 3 files cleanly; phase2-validation synthesised 498 chars of coherent JP prose with URL citations). Drive-by: search_top_urls fails loudly on non-200 status. DuckDuckGo's anti-bot path returns HTTP 202 with a "checking your browser" HTML body that parse_ddg_results turns into 0 results. The Tool::execute path still silently swallows this as "No results found", but the new helper surfaces "Search returned HTTP 202 (DuckDuckGo may be rate-limiting…)" so callers get a usable diagnostic. Cf. ddg_rate_limit_blocker.md. Live smoke is currently blocked by exactly that DDG rate-limit (triggered mid-dog-food today after ~8 consecutive runs); re-run when DDG clears. The architectural change is unit-test-covered and the two phases are independently validated as above. 740 lib tests pass. cargo fmt + clippy --release -D warnings clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
research_to_filetool: single CLI entry-point UX (the user describes the task in natural language, the LLM picks the tool) wrapping the validated phase 1 / phase 2 pipeline. Orchestration is in Rust — no agent loop inside the tool.Design (v2)
topic,output_file,instructions(verbatim shape requirements), optionalsource_count(default 3, clamped 1-10).web_searchcall via a newpub(crate)helpersearch_top_urlsthat returns structured(title, url)pairs. Then a Rust loop over the topsource_count * 2candidates, each iteration callingdispatch_web_fetch_call(LLM only used for per-page extraction) andstd::fs::writetosource-K.md. Stops atsource_countsuccesses.instructions, writes the deliverable tooutput_file.<output_stem>.sources/next to the deliverable.Moderate(same class aswrite_file).v1 → v2 history
The first commit on this branch (088fc46) ran phase 1 inside a subagent. Live smoke 2026-05-23 found the subagent reproduces exactly the convergence problem the tool was meant to bypass: LLM picked the tool, subagent started, but wrote 0 source files (out of 5 requested) and burned ~70 wasted tool calls. Wrapping the pipeline in a tool moved the convergence problem one level deeper. v2 (617be8d) takes the agent loop out of phase 1 entirely.
Drive-by
search_top_urlsreturns a hard error on non-200 status. DuckDuckGo's anti-bot path returns HTTP 202 with a "checking your browser" body thatparse_ddg_resultsturns into 0 results — theTool::executepath still silently swallows this as "No results found", but the new helper surfaces a clearer diagnostic. Worth fixing theexecutepath the same way in a follow-up.Validation
cargo fmt --check+cargo clippy --release --lib -- -D warningsclean.std::fs::writeprimitive.Test plan