Skip to content

Add bounded final-answer acceptance gate in regular runner#311

Open
mmprotest wants to merge 3 commits into
mainfrom
mmprotest/add-finalization-gate-to-runner
Open

Add bounded final-answer acceptance gate in regular runner#311
mmprotest wants to merge 3 commits into
mainfrom
mmprotest/add-finalization-gate-to-runner

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Prevent premature finalization by the regular runner when the assistant claims a concrete edit, when edits were made but not verified, or when the last verification failed and no corrective action followed.
  • Keep the change small, conservative, one-shot, and non-benchmark-specific while exempting analysis/plan-only requests.

Description

  • Added a centralized helper final_answer_block_reason(...) in villani_code/state.py that returns a one-off blocking nudge reason (or None) for three conservative cases: concrete edit claimed with no edits, changed files with a known verification command but no verification since the last edit, and failed verification with no corrective follow-up; it respects plan/analysis-only instructions and uses a per-run nudge_state to avoid repeated blocking.
  • Wired that helper into the runner completion path in villani_code/state.py, tracking minimal per-run bookkeeping (last_edit_turn, last_verification_turn, last_failure_followup_turn, finalization_nudge_state) and updating those markers when Write/Patch/Bash/inspection tools run so the gate has conservative inputs.
  • Improved verification feedback in villani_code/state_runtime.py by extracting focused visible_failure_lines (assertions, traceback/file refs, pytest E lines, and keywords like expected/actual) and appending a generic guidance sentence: "Treat visible test assertions as authoritative evidence. Do not guess replacement values when the expected value appears in the failure output."
  • Added unit tests tests/test_final_answer_gate.py that exercise the helper for the required behaviors and one-shot semantics.

Testing

  • Ran the new unit tests with pytest -q tests/test_final_answer_gate.py tests/test_state_runtime.py::test_verification_detail_event_keeps_raw_trace, and the test run passed (8 passed).
  • The added tests directly exercise final_answer_block_reason(...) for: concrete edit claim without edits (blocks once), vague suggestions (no block), plan-only exemption (no block), changed files without verification when verification is known (blocks), changed files without a known verification command (no block), failed verification with and without corrective follow-up (blocks/does not block respectively).

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant