Make Architect responses concise and facts-oriented - #2551
Conversation
4bbaba5 to
2be1198
Compare
![peqy[bot]](https://avatars.githubusercontent.com/in/1930020?s=60&v=4){kind=small}
There was a problem hiding this comment.
Critical Seed/streaming contract looks inconsistent for RUN_ANALYZE: the seed is still constrained to ~1–2 sentences, but the streaming pass is told to “add nothing to it” (no new sections). That likely prevents generating the phase-mandated run-summary structure (tables/rows/links) unless the seed already includes it.
Improvement The new “no emoji/marks anywhere” prompt scan is useful but brittle across all refs/templates; expect churn as docs evolve unless exemptions/scoping are made more flexible.
Found 2 issues (1 critical, 1 improvement).
|
|
||
| ## Instructions | ||
| Write a natural, complete response to the user based on your analysis, the tool results, and the seed response above. Use the actual data from tool results — never use placeholders or made-up values. Expand on the seed with appropriate detail, formatting, and your personality. Use markdown formatting where helpful. Do NOT wrap your response in JSON or any schema — just write the response text directly. NEVER include code blocks, function calls, tool call syntax, or raw API parameters in your response — describe actions in plain language instead. | ||
| Write the user-facing response. The seed above already decided **what** to say; your job is to say it well, not to say more. Keep the seed's content and add nothing to it — no new sections, no extra commentary, no restating a point the seed makes once. Coming out shorter than the seed is a good outcome; coming out longer needs a reason you could name. |
There was a problem hiding this comment.
Critical Possible seed/streaming mismatch: the system prompt still constrains final_answer (seed) to ~1–2 sentences (per telemachus-security.j2), but this streaming instruction says “add nothing to it — no new sections”. In RUN_ANALYZE, the required tables/rows/links in phases/analysis.md won’t fit in a 1–2 sentence seed, and the streaming pass would be prohibited from adding them.
Fix: either (a) relax this to “add no new claims/analysis beyond the seed, but you may populate the phase-required structure using tool results (tables/links/failure samples)”, or (b) change the first pass so the seed already contains the full phase-mandated shape and data (and keep streaming as purely tightening/rewriting).
There was a problem hiding this comment.
Addressed in 76d8dd5: the streaming prompt now includes the phase guidance (phase_knowledge_text), and the instruction explicitly allows populating the phase-required structure (tables/sections) from tool results without treating that as “adding content”. This removes the 1–2 sentence seed vs. required run-summary shape mismatch.
There was a problem hiding this comment.
Fixed in 7138181: _build_streaming_prompt() now injects phase_knowledge_text into streaming_response.j2, and the template explicitly allows populating the phase-required structure (tables/sections) from tool results without counting that as “adding content”. This resolves the 1–2 sentence seed vs. run-summary shape mismatch in RUN_ANALYZE.
| """ | ||
|
|
||
| # Emoji, arrows and check marks the old personality encouraged. | ||
| _DECORATION = re.compile("[✅❌✓✔✖✗\U0001f300-\U0001faff\U0001f600-\U0001f64f]") |
There was a problem hiding this comment.
Improvement The _DECORATION scan is intentionally strict, but it’s also brittle: it applies to all templates and skills/rhesis/references/**/*.md and will fail if any future reference includes ✓/✗ or emoji in a code block / worked example (like bracketfeld already did).
Fix: consider scoping this to the actual “example output” sections (or allow marks inside fenced code blocks), or expand the exemption mechanism to be path-based / comment-based so docs can evolve without test churn.
There was a problem hiding this comment.
Still think the intent is good, but +1 that this will be churny as soon as a future reference includes ✓/✗/emoji as data in a fenced example. If you want to keep the strict scan, one low-cost mitigation is to ignore matches inside ``` fenced blocks (or make exemptions path-based rather than by basename).
There was a problem hiding this comment.
Still agree with the intent, but this remains brittle: the regex applies across all templates + skills/rhesis/references/**/*.md, so any future reference that includes ✓/✗/emoji as data inside a fenced example will fail. Low-cost mitigations: strip/ignore ``` fenced blocks before scanning, or make exemptions path-based (rather than by basename) so future doc examples don’t cause churn.
|
Looks good. The streaming writer now gets Found 1 issue (0 critical, 1 improvement): the emoji/decoration scan test is intentionally strict but likely brittle long-term; consider ignoring decorations inside fenced code blocks or expanding exemptions to be path-based. |
The streaming pass told the agent to "expand on the seed with appropriate detail", so every answer grew past what the reasoning pass intended. Flip it to keep the seed's content and add nothing. Results analysis was specified in three overlapping places that all load in the same turn — streaming_response.j2, result-analysis.md and phases/analysis.md — so the agent tried to satisfy all of them at once. Give phases/analysis.md sole ownership of the output shape, with caps on failures and next steps, and drop the depth ladder that made a mostly-failing run produce the longest message. Personality drops the mythology, humor and emoji paragraphs along with "you think out loud", and gains a response-shape section: lead with the answer, no preamble, no closing recap, ~150 words of prose. Signed-off-by: Nicolai Bohn <nicolai@rhesis.ai>
Prompt wording has no runtime assertions, so nothing stopped the verbosity from creeping back. Cover the length budget, the ban on narration and emoji, the streaming pass not expanding the seed, and exactly one file owning the run-summary shape. Signed-off-by: Nicolai Bohn <nicolai@rhesis.ai>
The run-summary shape moved out of streaming_response.j2 and into phases/analysis.md, which only ever reached the iteration prompt. _build_streaming_prompt passed no phase knowledge and the call runs with an empty system prompt, so the pass that writes what the user reads was left with a one-or-two-sentence seed, an instruction to add no sections, and no description of the tables it was supposed to produce. Render the same phase knowledge into the streaming prompt, and say that laying out the required structure is not adding to the seed. Signed-off-by: Nicolai Bohn <nicolai@rhesis.ai>
76d8dd5 to
7138181
Compare
There was a problem hiding this comment.
Looks good. Injecting phase_knowledge_text into the streaming writer fixes the seed-vs-shape mismatch and keeps phases/analysis.md as the single format owner.
Improvement: the _DECORATION scan will likely be churny as soon as a future reference includes ✓/✗/emoji as data in fenced examples; consider ignoring fenced code blocks or expanding exemptions to be path-based.
1 participant
Purpose
Architect answers are too long, and results analysis is the worst case. The cause is structural rather than a matter of tuning. The Architect answers in two LLM calls: the first reasons and produces a short
final_answerseed, the second rewrites that seed into the message the user sees. The instruction on the second call was literally "Expand on the seed with appropriate detail", so every answer was inflated by design.On top of that, the run-summary format was specified in three places that all load together in the
EXECUTING/RUN_ANALYZEturn —streaming_response.j2,result-analysis.md, andphases/analysis.md. The agent tried to satisfy all three at once. The "calibrate depth" ladder instreaming_response.j2also had it backwards: a run below 40% passing got a full breakdown, so the worse the result, the longer the message that buried it.This makes the Architect factual and short by default, while keeping it a helpful assistant rather than a terse one.
What Changed
streaming_response.j2— "Expand on the seed with appropriate detail" becomes "The seed already decided what to say; your job is to say it well, not to say more… coming out shorter than the seed is a good outcome; coming out longer needs a reason you could name." The duplicated 5-section run summary, the depth ladder, and the comparison structure are removed in favour of one line deferring to the phase guidance. Link rules and the boundaries block are unchanged.personality.j2— drops the Telemachus mythology paragraph, the humor paragraph, the emoji paragraph, and "you think out loud" from the Planning bullet. Adds an explicit Response shape section: lead with the answer, no preamble, no closing recap, tables over prose, one question and only at the end, and a budget of ~150 words of prose per message excluding tables and code. Emoji are banned outright. Guiding principle 4 changes from "Make testing feel productive, not punitive" to "Facts before framing — do not soften a bad result or editorialize a good one."phases/analysis.md— becomes the single owner of output shape, with caps: overall is one line; requirements and metrics are one table row each with no commentary; failures are capped at 3 at two lines each, thenN more failures in this run; next steps are at most 3 one-line bullets, skipped entirely when everything passed. States explicitly that the shape does not change with the pass rate.result-analysis.md— 199 to 161 lines. Both "Presenting…" sections and the scripted "Offering next steps" sentences are gone; the file is retrieval-only now and says so at the top.insights-summary.md— step 4 of the tool sequence points atphases/analysis.mdinstead of restating the format.telemachus-guidelines.j2— "Be thorough but concise in your questions" was not actionable; it becomes "Ask one question at a time, and only when a tool call cannot answer it". The metric-richness rule keeps its substance but adds that richness belongs in the tool call, not in the chat message.TestConciseOutputGuidanceclass with 9 tests covering the length budget, the removal of narration and emoji guidance, the streaming pass not being told to expand, the caps inphases/analysis.md, and exactly one file prescribing the run-summary shape.Additional Context
skills/rhesis/references/**is mirrored publicly torhesis-ai/skillson merge tomain, so four of these files ship to users.python scripts/skill/validate.pypasses.personality.j2stays a.j2rather than becoming a.md. Six of the nine files inprompt_templates/contain zero Jinja tags, so static prose in a.j2is the local convention; renaming only this one would make it the inconsistent file for no behavioural gain.use-case-bracketfeld.mdis exempted from the no-emoji test with a comment explaining why: its ✓/✗ marks are cells in a scope-compatibility table the agent reads but never reproduces.Testing
Verified that the rendered
EXECUTING/RUN_ANALYZEturn now injects exactly one presentation spec:Still needs a live check. Prompt changes are not verified by unit tests. Worth driving one real analysis turn against a completed test run and comparing the output length and shape against the caps in
phases/analysis.md— particularly a run with more than 3 failures, to confirm theN more failures in this runcap holds instead of the agent listing all of them.