Dynamic executor model routing with cost-aware escalation

[saikethan27]

Closes #13.

Adds a second, orthogonal routing dimension to the executor: a tier (cheap / strong) alongside the existing type (gui / cli). Routine work stays on a cheaper backend; the Manager can ask for a stronger one per subtask, and the harness escalates on its own when verification keeps failing.

Everything goes through the existing AgentAdapter abstraction, so it is backend-agnostic — a Codex manager can dispatch to a Claude Code executor and back.

Rebased onto 53bc678 (0.1.4). Conflicts resolved in favour of upstream wherever the two overlapped; details at the end.

Note on scope: this branch carries two independent commits. The first is the feature for #13. The second is an unrelated Windows fix that I hit while trying to verify the feature — the harness could not complete a single round on Windows. Happy to split it into its own PR if you would prefer to review them separately.

---

1. Executor tiers and cost-aware escalation (#13)

Configuration

[run.roles.executor.cheap]
agent = "codex"
model = "gpt-5.6-sol"

[run.roles.executor.strong]
agent = "claude_code"
model = "claude-opus-5"

[run.executor_routing]
default_tier = "cheap"
escalate_after_failures = 1
escalate_after_stalled_rounds = 3
escalation_tier = "strong"

Tier tables nest inside the existing role tables, so resolution keeps the current fallback-chain shape. Naming a tier is a deliberate cost decision, so it outranks the older type-level section:

gui_executor.cheap → executor.cheap → gui_executor → executor → [run].agent/.model

Per-type overrides ([run.roles.gui_executor.strong]) are supported, so all four of CLI+cheap, CLI+strong, GUI+cheap and GUI+strong are expressible without duplicating the executor implementation. Every field has a matching CLI flag, as elsewhere in the project.

Backward compatible. A config with no tier tables resolves every tier to the same executor it uses today. A test asserts a legacy [run.roles.executor] config produces a byte-identical defaults dict.

Manager-selected tier

The Manager may add one optional line after its route:

Next: cli
Executor tier: cheap

The issue sketched this as JSON, but the manager protocol is deliberately JSON-free throughout (MANAGER_INSTRUCTIONS: "Output plain natural language, never JSON"), so it follows the existing Next: line convention instead. The keys map one-to-one. The prompt instructs the Manager to judge the subtask in front of it and explicitly not to decide from task category. Omitting the line accepts the configured default.

Escalation

cheap executor → Auditor → FAIL → strong executor → Auditor

Escalation never bypasses verification: the strong result goes through the normal Auditor flow. A passing audit clears the escalation, so routing returns to the default tier and the expensive model is scoped to the stretch of the run that is actually struggling.

What counts as a failure matters more than it looks. Auditors are instructed to report incomplete whenever the whole contract is unsatisfied, "even if the local subtask succeeded" — so a perfectly good mid-run round comes back incomplete / clean / aligned. An earlier version of this PR treated that as a failure and escalated on round one of every task, which threw the saving away. Two distinct signals now:

Signal	Counts toward	Meaning
blocked, suspect/violation, needs_revision/invalid, or an executor episode that errored	escalate_after_failures	The round actually went wrong
Consecutive clean rounds naming the same unclosed gap	escalate_after_stalled_rounds	The cheap tier is spinning
incomplete + clean + aligned with a new gap each round	neither	Ordinary progress

Escalation briefing

Every role episode is a one-shot process — there is no session to resume, and resuming would not be coherent across an escalation that changes backend. A previous round's executor_output never otherwise reaches a later executor, and the Auditor's findings only arrive if the Manager happened to cite those rounds.

So an escalated executor is briefed with what the previous tier attempted and why it was rejected. The prior attempt is labelled as that executor's own unaudited claim; the Auditor report is labelled authoritative — the same trust boundary MANAGER_INSTRUCTIONS already draws. Bounded to the three most recent failures and char-clipped. Disable with escalation_briefing = false.

The Manager is told too, through the existing harness_feedback channel, since repeated failure can mean the decomposition is wrong rather than the executor being too weak.

Observability

• Startup summary resolving every (type, tier) cell to its agent and model
• executor_tier + tier_source on the executor events; a new executor_escalation event carrying the reason and the failed rounds
• tier=cheap on the console role line, and a dedicated escalation line
• executor_tier per round in report.json, plus an executor_routing block recording the policy, per-tier round counts, and the full escalation history (kept even after a run recovers)
• executor_tier exposed by DashboardState for finished and in-progress rounds (written before the executor starts, so a live round already shows it). The badge itself needs re-adding to the new frontend/, see below.

---

2. Windows support (independent of #13)

The harness could not complete a single round on Windows. Agent commands were POSIX shell strings handed to create_subprocess_shell, which is cmd.exe there, so mkdir -p failed on the first call. Process control used os.killpg/SIGKILL/SIGHUP, none of which exist on Windows, so every timeout raised AttributeError.

Rather than translate shell strings per platform, this removes the shell from the agent path. Everything it was doing is a real subprocess argument:

Was	Now
cd '<ws>' && …	cwd=
VAR=value <cmd>	env= layered onto os.environ
… < prompt.md	prompt written to the child's stdin
mkdir -p, chmod	native pathlib calls

Commands are argv lists, so shlex.quote is gone and no model-supplied value can reach a shell parser. Environment.exec survives as an explicit escape hatch.

Also fixed, all found by running it for real:

• Process control — platform split: CREATE_NEW_PROCESS_GROUP + taskkill /T, named operations instead of POSIX-only signal numbers
• MAX_PATH — run directories reach 260 characters easily, and the failure is deceptive: mkdir succeeds on the shorter parent while open() on the file inside raises FileNotFoundError. New utils/paths applies the \\?\ prefix, but only to paths that actually exceed the limit
• screenshot() — PowerShell on Windows, screencapture on macOS, existing X11 tools on Linux
• Console — cp1252 no longer mangles output
• Claude deny rules — now also emit the drive-letter form, so harness-owned paths are genuinely hidden from the auditor on Windows

The 0.1.4 hardening layer is POSIX-only in the same way, so the run tree could not be written on Windows at all: _ensure_dir_nofollow, _atomic_bytes_write, _append_jsonl and the event append all require O_NOFOLLOW, O_DIRECTORY and dir_fd. Each grows a Windows branch that keeps what the platform can express — reparse-point refusal, atomic replace, the hard-link check before truncation, and the same event-id sequencing — and documents the one guarantee that cannot be reproduced without directory descriptors: the check-then-use window is not anchored. POSIX behaviour is untouched.

Breaking: CommandAgentAdapter takes argv=/env= instead of command_template=. EpisodeResult.metadata["command"] is now the argv list rather than a string; the existing reader in manager.py already accepts both. The vendored cua_harness copies under eval/ are separate packages and are untouched.

---

Testing

The lh_harness package had no test suite; this adds one — 208 tests, no paid model calls (scripted adapters and a recording environment).

Area	Cover
Configuration	Both tiers, per-tier backends, type×tier, routing table, legacy config unchanged, and every invalid-config error path
Resolution	All four (type, tier) cells across five config shapes, tier-before-type precedence, and a regression lock on the eight pre-existing roles
Manager routing	Tier parsing (en/zh, bold, unknown → default), all four type×tier combinations, auditor and budget unaffected by tier
Escalation	Threshold, stickiness, reset on pass, disabled, both executor types, and the clean-but-incomplete regression
Briefing	Content and trust labelling, prompt byte-identical without one, survives a Manager citing nothing
E2E	Full flow asserted on persisted state — rounds.jsonl, report.json, events.jsonl, Dashboard
Windows	LocalEnvironment argv/cwd/env/stdin, timeouts, long lines, MAX_PATH, process-group primitives, adapter argv construction

Verified against real models on Windows

Run	Result
Full run, codex / gpt-5.6-luna	complete — file produced, all trajectories written
Cross-backend	codex.CMD manager + auditor, claude.EXE executor, one run, complete
Tier really changes the model	tier=cheap → --model=gpt-5.6-luna, tier=strong → --model=gpt-5.6-sol, read from the recorded argv
Manager tier selection	Emitted Executor tier: cheap unprompted
Escalation + briefing	Fired, briefing embedded verbatim in the executor prompt, escalated round audited and passed
Backward compatibility	Legacy single-executor config completed normally

Rebased onto 0.1.4

Rebased onto 53bc678, resolving conflicts in favour of upstream wherever the two overlapped — the supervised-run plumbing, bootstrap-failure handling, resolve_codex_binary(), the LH_HARNESS_WEB_TOKEN scrub, bounded stdout capture, and the symlink-safe trajectory writer are all preserved. The tier badge that used to live in dashboard/static/app.js went away with that file; executor_tier is still exposed by DashboardState, so re-adding it to the new frontend/ is a small follow-up.

Measured against origin/main on the same machine:

	origin/main	this branch
Failed	74	68
Passed	94	308

No regressions. The six newly-passing tests are upstream's own (test_cli_isolation, test_manager_hardening, test_trajectory_artifacts), repaired by the Windows branches above. The remaining 68 are pre-existing Windows breakage in the webapi/supervisor POSIX-only paths, which this PR does not touch — on Linux CI they are unaffected either way.

Known limitations

• A same-tier retry (when escalate_after_failures is raised above 1) is not briefed; only escalated rounds are. Documented at the knob.
• Tiers are a closed set (cheap, strong).
• The stall detector compares the Auditor's gap text by similarity — a heuristic, tunable, and disabled with 0.
• The Claude drive-letter deny rule emits both forms because I could not confirm upstream which one the matcher accepts on Windows. Worth a maintainer's eye.