Keep architect tool data across turns, show reviews in full

[nicolai-rhesis]

Purpose

#2427 stopped annotations arriving at the LLM as a column of - ?, but it did not stop the architect inventing review content. Two paths were left open, and both are data starvation rather than creativity — the agent is told reviews exist, shown nothing usable, and asked to report on them.

Tool results died with the turn. prepare_and_load_session rehydrates only conversation_history from the DB, so every user message builds a fresh agent with an empty _execution_history. A follow-up question about something already fetched — the exact words of a review, who left it, which turn it was on — had nothing but the agent's own earlier prose to answer from, and older messages are cut to 500 chars. This is the likeliest source of invented reviewer names and turn numbers in a real conversation, and #2427 does not touch it.

Review comments arrived clipped to 80 characters. desc_chars is a summary width for a named entity's description, where the untruncated version is one get_* call away. Since #2427 it was also the clip width for every string inside an unnamed record, where this rendering is the only place the content is ever shown. A reviewer's comment reached the model as half a sentence ending in …, next to an instruction to "expand on the seed with appropriate detail" — the setup for finishing someone else's sentence and presenting it as their review. Arguably worse than the bare - ? it replaced, because a partial quote reads as a real one.

What Changed

Tool data survives the turn

• dump_state emits carried_tool_results: the rendered tool results from this and earlier turns, newest-first within two caps, then back in chronological order. restore_state loads it; persist_state stores it on agent_state and build_agent reads it back. No migration — the key is additive and existing sessions fall back to [].
• The digest goes into both prompts (_format_history for the reasoning call, _format_tool_results_for_streaming for the response writer) under a header that tells the agent to quote from it rather than recall it, and to re-read the tool if the user needs current state.
• Failures and the internal pseudo-tools are excluded. An error is not data, and a finish result is the answer text, which is already the assistant message in conversation history.
• Bounded by carried_result_max_entries (12) and carried_result_max_chars (12_000). One review costs about 415 bytes of JSONB.

Unnamed records are shown whole

• _compact_list_result_for_history now takes record_chars (4_000) and record_block_chars (20_000) separately from desc_chars, which keeps its 80 for named entities. An annotation, a test result, anything without a label is dumped in full.
• The block budget is shared across the unnamed records on the page, so a page of small records is never clipped at all and a page of fat ones degrades in detail instead of losing rows. Dropping rows to fit would read as "there were only four", which is the same lie as the - ?. Anything actually cut carries a … [record truncated] marker.

Measured across three page shapes:

page	raw	rendered	rows kept	full comment kept
1 annotation	553	534	1	yes
20 annotations, ~440-char comments	9,865	9,675	20	yes
20 test results with full outputs	88,825	20,555	20	n/a — all 20 marked truncated

Incidental

• _format_tool_results_for_streaming now slices from _turn_start_step. Execution history accumulates in-process, so a third-turn response was handed turn one's results under the heading "Tool Results (data you collected)" as if freshly fetched. Stale data presented as current is its own fabrication source. No effect on the backend path, where the agent is fresh each turn, but it is required for correctness now that data crosses turns at all.

Additional Context

• Refs Architect returns no reviews for a test run and hallucinates reviews #2402. Follows Let the architect read annotations and create complete metrics #2374 (made annotations reachable) and Fix annotations list hiding every pre-project-scoping review #2427 (stopped them rendering as - ?).
• test_long_comment_is_clipped asserted exactly the behaviour this removes, so it is replaced by three tests covering the new contract.
• Not addressed here, from the same analysis of Architect returns no reviews for a test run and hallucinates reviews #2402: streaming_response.j2 still instructs the model to "expand on the seed with appropriate detail" with no grounding rule for quoted human text; get_test_result still returns test_reviews inline as an undocumented second shape for the same reviews; blank comments/user fields still have no vocabulary, so "no comment left" cannot be said; and total_count is returned as an X-Total-Count header, so the tool payload cannot state a real total. Each is a separate change.
• Also noticed, unrelated and untouched: workflow_path is in the snapshot and emitted by dump_state, but persist_state never writes it and build_agent never reads it, so it resets to unset every turn.
• There is still no eval that asserts what the model says matches what the tool returned. That is what would have caught Let the architect read annotations and create complete metrics #2374 not working, and it also covers Architect reports creating objects it did not create, and substitutes its own rubric #2516.

Testing

cd sdk && uv run pytest ../tests/sdk/agents/
cd apps/backend && uv run pytest ../../tests/backend/tasks/architect/ ../../tests/backend/services/architect/

443 SDK tests and 100 backend tests pass. Docker must be running for the backend suite.

New coverage:

• A two-turn round trip: turn 1 fetches annotations, the digest goes through json.dumps/loads the way the JSONB column does, and turn 2 — a fresh agent with an empty execution history — has the reviewer, the verbatim comment and the turn index in its prompt.
• Failed calls and finish results stay out of the digest; both caps keep the newest entries; earlier data survives a turn that fetched nothing; reset() clears it.
• The streaming prompt shows this turn's results and not the previous turn's.
• A long review comment survives whole; a runaway field is still bounded by record_chars; a 20-row page of fat records keeps all 20 rows with the clipping marked and every review still identifiable.

Worth a manual check: open the architect on a run with reviews, ask what people flagged, then ask a follow-up about one specific review without naming it again. The follow-up should quote the real comment rather than paraphrasing from its own previous answer.

[peqy]

The core approach (persisting a rendered, bounded tool-result digest across turns + showing unnamed records whole) directly addresses the hallucination sources described and the tests look on-point.

Main things to consider before merge:

• If the newest carried entry is too large for carried_result_max_chars, _build_carried_tool_results() currently drops the entire digest (including older smaller entries) due to a break.
• The per-record record_block_chars budget for unnamed records isn’t strictly enforced because overhead isn’t counted (and there’s no aggregate counter), so worst-case prompt growth can exceed the intended bound.

Found 2 issues (0 critical, 2 improvements).

[peqy]

Improvement: _build_carried_tool_results stops building the digest entirely when the newest entry doesn’t fit the remaining char budget (break on used + len(rendered) > ...). If a single large rendered result (e.g. a compacted 20-row page) exceeds carried_result_max_chars, this will produce an empty digest and also drop any older smaller carried entries.

Fix: consider continue-skipping oversized entries (so older data still survives), and/or truncating an oversized newest entry to fit with a clear marker (similar to _render_tool_result’s fallback truncation). A regression test for “one oversized newest entry doesn’t wipe existing digest” would lock this in.

[peqy]

Fixed in 693b4be: oversized newest entries are now truncated/skipped instead of break-ing the whole build, and there’s a regression test for the “fat page doesn’t empty/wipe digest” case. 👍

[peqy]

Improvement: allowance is computed per unnamed record as min(record_chars, max(_MIN_RECORD_CHARS, record_block_chars // unnamed_count)), but the loop then appends f" - {compact_item}" without accounting for the JSON overhead/newline/prefix. That means the actual block size can exceed record_block_chars (sometimes by a lot), defeating the “shared across unnamed records” guarantee.

Fix: either (a) include overhead in the per-record budget (e.g. allowance - len(" - ")), or (b) enforce record_block_chars as an aggregate counter while iterating and truncate/mark once exceeded.

[peqy]

With the new per-record truncation marker, I’m less worried about strict record_block_chars adherence: worst-case overhead (bullet + marker) is small vs the 20k default, and the important contract (don’t drop rows; mark truncation) is covered by tests. I’d treat this as non-blocking unless you need a hard cap.

[peqy]

Main idea looks solid and the new tests capture the intended “quote real tool data across turns” contract well.

Blocking: the two open threads still apply:

• _build_carried_tool_results breaks out if the newest rendered entry doesn’t fit the remaining carried_result_max_chars, which can yield an empty carried digest when a single compacted page exceeds 12k (exactly the “fat page” case). This undermines the MR’s goal.
• _compact_list_result_for_history’s per-record allowance doesn’t account for per-line overhead / named-item lines, so record_block_chars isn’t a hard bound.

Once those are addressed, ship it.

[peqy]

Looks good. The carried tool-result digest + full rendering for unnamed records should materially reduce invented review content, and the new tests cover the risky cases (fresh-agent follow-up, fat pages, streaming turn-slice). Ship it.