feat(benchmark): pre-bake hermes agent into dataset images for closed-book runs

[gburachas]

Closes #259.

What

Bakes the Hermes agent into the dataset image at prepare time, pinned to a full commit SHA, so closed-book Terminal-Bench runs do not fetch it at task time.

• benchmark/agent-versions.env — the pin, in one place
• benchmark/prepare_harbor_dataset.py — install the agent during dataset prep and record the pin in the manifest
• benchmark/patches/harbor-agent-patches.diff — skip the runtime install when the agent is already present
• tests/test_prepare_harbor_dataset.py — assert the pin reaches the manifest

Why

Closed-book runs deny network egress to the task container. An agent installed at task time therefore fails, and the failure looks like a task failure rather than a harness problem — it shows up as a score, not an error. Pre-baking moves the fetch into dataset preparation, where the network is available and a failure is loud.

Pinning is the other half: an agent that silently changes between runs makes two runs incomparable while both appear valid.

The pin is a full commit SHA, not a tag. A tag reads as immutable but can be deleted or repointed, so two builds could record the same string in the manifest while installing different code — the manifest would then assert a reproducibility it does not have. Requiring one shape is complete by construction, where a deny-list of moving names (main, master, ...) never can be, since any branch name passes it. The installer script is fetched from the same commit for the same reason.

The pin is applied with the installer's --commit, not --branch. --branch reaches git clone --branch, which accepts only branch and tag names and rejects a SHA outright. --force-commit is required alongside it: without it the installer skips the pin whenever the commit is an ancestor of the freshly cloned HEAD, logs a warning, and leaves the image on the tip of main — the drift this pin exists to prevent, arriving as a warning rather than a build failure. test_the_hermes_pin_is_applied_by_commit_and_forced pins that flag combination so it cannot be quietly dropped.

Scope

Benchmark tooling only. No Rust, no library or server behaviour, no public API. Kept minimal per @ayushag-nv's note on #259.

How tested

• uv run pytest tests/test_prepare_harbor_dataset.py -v — 17 passed
• uv run pytest tests/ -m "not integration" — 876 passed, 9 skipped
• all five affected tests re-run against the pre-fix code and fail there, so none passes vacuously
• uv run ruff check .
• SPDX headers present on both touched .py files
• Commits signed off per the DCO

Rebased onto current main with no conflicts.

Note: the first commit is authored by @SubhashreeR, carrying their original sign-off.

Summary by CodeRabbit

• New Features

  • Added Hermes agent support to benchmark environments using a fixed, verified release.
  • Added support for custom OpenAI-compatible endpoints, including configurable base URLs and optional API keys.
  • Reuses an existing local Hermes installation when available, reducing unnecessary setup work.

• Bug Fixes

  • Prevents installations from using mutable or unverified source references.

• Tests

  • Added coverage confirming release pin validation and installer behavior.

[coderabbitai]

Walkthrough

The change pins Hermes to v2026.8.3, pre-installs it in dataset images, and reuses local installations. Runtime setup now supports custom OpenAI-compatible endpoints. Tests validate ref pinning and manifest metadata.

Changes

Hermes dataset runtime

Layer / File(s)	Summary
Pinned Hermes installation benchmark/agent-versions.env, benchmark/prepare_harbor_dataset.py, benchmark/patches/harbor-agent-patches.diff, tests/test_prepare_harbor_dataset.py	The build validates the Hermes ref, installs Hermes from the pinned ref, verifies its version, records it in manifest metadata, and skips installation when a local binary exists. Tests cover moving refs and pinned installer references.
Custom endpoint routing benchmark/patches/harbor-agent-patches.diff	Runtime setup passes OPENAI_BASE_URL and OPENAI_API_KEY into YAML generation. Custom endpoints use a bare model name and provider configuration.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

I’m a rabbit with a pinned little friend,
Hermes installs where the image will send.
Custom endpoints now follow the trail,
Local binaries make installs fail-safe.
Tests guard each immutable detail. 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes pre-bake Hermes, pin its version, record the manifest version, and skip runtime installation when Hermes is already installed [#259].
Out of Scope Changes check	✅ Passed	All changes remain within benchmark tooling and support Hermes installation, pinning, runtime reuse, and endpoint configuration.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes pre-installing the Hermes agent in benchmark dataset images for closed-book runs.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

benchmark/prepare_harbor_dataset.py (1)

210-210: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Wrap the changed Python source lines.

Lines 210 and 261 exceed 100 characters. Extract the prebaked-version value and split the installer command across adjacent string literals.

As per coding guidelines: "Keep lines within 100 characters."

Also applies to: 261-262

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@benchmark/prepare_harbor_dataset.py` at line 210, In the dataset preparation
command construction, extract the prebaked agent version value into a local
variable and reuse it in the environment assignment. Wrap the long Python source
lines, including the installer command around the referenced command-building
logic, by splitting adjacent string literals while preserving the generated
command exactly.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmark/prepare_harbor_dataset.py`:
- Around line 195-206: Require HERMES_VERSION in the prepare_harbor_dataset
validation flow to be a full commit SHA, or resolve configured refs to one
before generating the Docker layer and manifest; reject tags and branch
references. In benchmark/prepare_harbor_dataset.py lines 195-206, update the
HERMES_VERSION validation accordingly; in benchmark/agent-versions.env lines
8-13, replace v2026.8.3 with the resolved full SHA; in
tests/test_prepare_harbor_dataset.py lines 331-346, use a full-SHA fixture and
assert that tag and branch references are rejected.
- Around line 258-262: Update the Alpine package installation branch in the
generated setup command to include the bash package alongside git, ripgrep, and
xz, ensuring the subsequent installer pipeline to bash succeeds on Alpine.
- Around line 201-202: Add "HERMES_VERSION" to the required pin names validated
by prepare_dataset before the pins["HERMES_VERSION"] access, preserving the
existing validation error behavior for missing pins.

---

Nitpick comments:
In `@benchmark/prepare_harbor_dataset.py`:
- Line 210: In the dataset preparation command construction, extract the
prebaked agent version value into a local variable and reuse it in the
environment assignment. Wrap the long Python source lines, including the
installer command around the referenced command-building logic, by splitting
adjacent string literals while preserving the generated command exactly.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: af463a69-0464-4e0c-9a01-b8bff70b1b94

📥 Commits

Reviewing files that changed from the base of the PR and between fb3fc30 and 4fcdde2.

📒 Files selected for processing (4)

• benchmark/agent-versions.env
• benchmark/patches/harbor-agent-patches.diff
• benchmark/prepare_harbor_dataset.py
• tests/test_prepare_harbor_dataset.py

[ayushag-nv]

@gburachas Can you please address coderabbit comments first. On high level looks good, please address the comments, I will approve

[gburachas]

@ayushag-nv all three CodeRabbit findings are addressed in f4052d9 — this is ready for your review.

Finding	Fix
Pin can be repointed	HERMES_VERSION is now a full commit SHA, validated by shape
Missing pin crashes unhelpfully	HERMES_VERSION added to the shared required pin set
Alpine build fails before install	bash added to the apk line

One thing worth your attention rather than a rubber stamp. The SHA suggestion could not be applied as written: the value was also passed to the installer's --branch, which reaches git clone --branch and accepts only branch and tag names — a SHA is rejected outright, so the literal fix would have failed every build. The pin now goes through the installer's --commit flag instead, with --force-commit.

--force-commit is the non-obvious part: without it the installer skips the pin whenever the commit is an ancestor of the freshly cloned HEAD, warns, and leaves the image on the tip of main — the exact drift the pin exists to prevent, arriving as a warning instead of a build failure. There is a test pinning that flag combination so it cannot be quietly dropped.

Validation

• uv run pytest tests/ -m "not integration" — 876 passed, 9 skipped (baseline 873 + 9; the 3 new tests are the delta)
• uv run ruff check . — clean
• SPDX headers present on all touched files; commit carries the DCO sign-off
• All five affected tests were re-run against the pre-fix code and fail there, so none passes vacuously

Scope is unchanged: benchmark/ only, no Rust, no library or server behaviour, no public API.