diff --git a/README.md b/README.md index 84b574f..13fb1e9 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ [![mutation](https://img.shields.io/endpoint?url=https%3A%2F%2Fraw.githubusercontent.com%2Frxdt%2Floopgate_harness%2Fmain%2Fmutation-score.json)](https://github.com/rxdt/loopgate_harness/actions/workflows/mutation.yml) + --- @@ -72,7 +73,8 @@ Each run starts fresh, has clear limits, saves its logs, protects key files, and --- -## Default Tools +## Default Tool Dependencies +_Edit at will_ - [ruff](https://docs.astral.sh/ruff/) lints and formats Python code, fast - [pylint](https://pypi.org/project/pylint/) catches code errors and style problems - [pydoclint](https://pypi.org/project/pydoclint/0.9.1/) checks docstrings match function signatures @@ -88,14 +90,20 @@ Each run starts fresh, has clear limits, saves its logs, protects key files, and - [preferences.py](preferences/preferences.py) A custom AST-parser to optionally expand. It catches e.g. a [style preference](https://google.github.io/styleguide/pyguide) that tools don't. - Forbidden paths set in [[tool.harness]](pyproject.toml) - Update `[tool.harness.gate]` or `[tool.harness.gate]` in [pyproject](pyproject.toml) to change what is checked before a commit or push. +- Failing checks block by default. If a fresh drop-in fails across the board, set [`behavior = "warn"`](pyproject.toml#L87) to report without blocking, then flip back to `"fail"`. -#### The Gate: Tiered Checks +### The Gate: Tiered Checks -⚑ `harness preflight` _(pre-commit)_ are the fast checks to run often. Lint + check format for everyone, _plus_ **containment** for the agents. > [!NOTE] -> **Self-heals by un-staging forbidden files.** +> **A pre-commit or gate phase self-heals by un-staging forbidden files.** + +⚑ `harness preflight` _(pre-commit)_ + +are the fast checks to run often. Lint + check format for everyone, _plus_ **containment** for the agents. -βœ… `harness gate` _(pre-push)_ = _(pre-commit)_ checks **+** type-checks, security audit, dependency audit, complexity analysis, full test coverage, prompt to run mutmut +βœ… `harness gate` _(pre-push)_ = _(pre-commit)_ checks **+** + +adds type-checks, security audit, dependency audit, AST-scan, complexity analysis, full test coverage, prompt to run mutmut Only humans can bypass triggered gates and commit, always. Only humans can use flag `--no-verify`. @@ -109,13 +117,14 @@ Note that `semgrep --config auto` needs network for semgrep registry rules. ## Details -[docs/plan.md](docs/plan.md) is where you define what you want the end product to be. You must be _very_ clear on what the finished product should and should **not** contain. Do **not** let agents guess. +[plan.md](docs/plan.md) is where you define what you want the end product to be. You must be _VERY_ clear on what the finished product should and should **not** contain. Do **not** let agents guess. Tell agents *exactly* what the finished "product" should be. -`docs/PROMPT.md` tells each agent to pick a `spec` and build. `docs/specs/` say _what_ to build. The agent decides _what next_. You keep `docs/plan.md` current, and specs get rewritten from it (agent is told in `docs/PROMPT.md` to update the specs). Each iteration the agent updates its spec and `PROJECT_STATUS`. +[`PROMPT`](docs/PROMPT.md) tells each agent to pick a spec.md and build. [`specs/`](docs/specs) say _what_ to build. The agent decides _what next_. You keep [`plan.md`](docs/plan.md) current, and specs get rewritten from it. The agent is told in `docs/PROMPT.md` to update the specs. Each iteration the agent updates its spec and [`PROJECT_STATUS`](docs/PROJECT_STATUS.md). -> [!IMPORTANT] -> Default configuration is in [`pyproject.toml`](pyproject.toml). Update tool settings, add agent commands, change checks, or leave it as is. +### In summary, your job, the bare minimum: +write something into the plan +## Start a project ## Start a project 1. `gh repo create / --template rxdt/loopgate_harness --private --clone` **or** @@ -133,6 +142,9 @@ Note that `semgrep --config auto` needs network for semgrep registry rules. 10. Configurations for Ruff linting, type-checking Pyright, Complexipy, Pytest coverage, etcetera are set in [`pyproject.toml`](pyproject.toml). 11. Your coding quirks go in [`preferences/preferences.py`](preferences/preferences.py). Delete functions that don't serve you. Add your own. +> [!IMPORTANT] +> Default configuration is in [`pyproject.toml`](pyproject.toml). Update tool settings, add agent commands, change checks, or leave it as is. + ### Works with `uv`, `poetry`, or `pip` ```sh @@ -202,10 +214,6 @@ harness run copilot 2 20 ``` #### To run LoopGate with any agent, the worker must be installed and authenticated separately. -### Run logs - -Every run is saved as a log file in `scratchpad/runs/`. `harness status` shows how many logs you have and the path to the newest one. Open that file to read what the agent thought and did. _(Metrics and audited logs coming soon.)_ - ### Add a mutation score badge Run `uv run mutmut run && uv run mutmut export-cicd-stats`, then use [check_mutmut.py](mutation/check_mutmut.py) to write `mutation-score.json` for the Shields badge. @@ -234,15 +242,21 @@ harness/ the gate, loop runner, CLI (πŸ€– forbi cli.py command-line entry point js-scaffold javascript example to build upon preferences/ user-defined preferences not covered by tools (πŸ€– forbidden directory) +mutation/ get your mutation score and learn how to run mutmut (πŸ€– forbidden directory) + check_mutant.py tests/ preferences/ (πŸ€– tests/preferences is forbidden directory) .githooks/ pre-commit / pre-push gate hooks (πŸ€– forbidden directory) -pyproject.toml project + tooling config (πŸ€– forbidden) +pyproject.toml project + tooling config (πŸ€– forbidden file) docs/ PROMPT, specs/, your plan (agent and human maintained) scratchpad/ scratch dir agents can use for temp files (For the πŸ€– to play) src/ your product/source code (add to coverage source) ``` +### Run logs + +Every run is saved as a log file in `scratchpad/runs/`. `harness status` shows how many logs you have and the path to the newest one. Open that file to read what the agent thought and did. _(Metrics and audited logs coming soon.)_ + [`pyproject.toml`](pyproject.toml) is the single source of harness configuration. Humans own it and [`preferences/`](preferences/); both are agent-protected. If an agent edits a forbidden file, the file will be unstaged (not allowed to commit). A forbidden pattern by an agent (e.g. `# noqa` or `nosemgrep` will also prevent their commit and force them to fix it.) @@ -265,7 +279,7 @@ If an agent edits a forbidden file, the file will be unstaged (not allowed to co ```yaml - run: uv sync && uv run harness gate ``` -- Add or remove coding preferences [preferences.py](preferences/preferences.py) that only agents in loops **must** respect. Current preferences: +- Add or remove coding preferences [preferences.py](preferences/preferences.py) that only agents **must** respect. Current "preferences": ```py function_argument_assignment_has_star # agents use non-specific `def fun(*)` @@ -300,9 +314,17 @@ LoopGate does not install or log in agent CLIs. Install and authenticate the wor A **gate** is a workflow checkpoint that evaluates code and decides whether it is allowed to land in your commits. A **sandbox** is an isolated OS-level environment designed to prevent code from modifying your underlying machine. LoopGate uses gates to control your git history, but it does _not_ provide a secure OS sandbox. +- **What if I don't want to use those agents?** + +Remove or add commands to enable using different agents. Let's say you're going _exclusive_ with Mistral and will _only_ use Mistral 3. Set this and only this in [`[tool.harness.agents]`](pyproject.toml#L91-L123) +``` +vibe = ["vibe", "--auto-approve", "--output", "streaming"] +``` +Then run it with `harness run vibe ## Read before Infinity Loops +## Read before Infinity Loops 1. **This harness does not sandbox agents.** It tries to harness bad code in loops via gates. Sandboxing agents will, e.g. prevent them from maintaining git, running Playwright, being seen as trustworthy by semgrep leading to cyclical failures, etc. diff --git a/README.template.md b/README.template.md index 1aca937..f41f1e2 100644 --- a/README.template.md +++ b/README.template.md @@ -98,6 +98,7 @@ harness run copilot 2 20 Every run is saved as a log file in `scratchpad/runs/`. `harness status` shows how many logs you have and the path to the newest one β€” open that file to read everything the agent did. Tool commands are defined in `[tool.harness]` in [pyproject.toml](pyproject.toml). The gate and CI both derive them from there. +Failing checks block by default. If your project fails across the board at first, set [`behavior = "warn"`](pyproject.toml#L87) to report without blocking, then flip back to `"fail"`. #### The Gate: Tiered Checks diff --git a/harness/cli.py b/harness/cli.py index 24b3825..7532101 100644 --- a/harness/cli.py +++ b/harness/cli.py @@ -253,7 +253,7 @@ def check_for_timeout_and_prompt() -> str | None: """ if IS_WINDOWS: return None - if not which("gtimeout") or not which("timeout"): + if not (which("gtimeout") or which("timeout")): rprint("\n[yellow]macOS harness needs timeout/gtimeout from coreutils to run loops[/yellow]") if not which("brew"): rprint("Get Homebrew https://brew.sh then run `brew install coreutils` or `sudo port install`") diff --git a/harness/gate.py b/harness/gate.py index d08c6f3..7502ae3 100644 --- a/harness/gate.py +++ b/harness/gate.py @@ -45,15 +45,16 @@ def __init__(self, root: Path) -> None: Path(__file__).with_name("temp.pyproject.toml").read_text(encoding="utf-8") ) harness = defaults["tool"]["harness"] + self.settings = harness.get( + "settings", {"behavior": "fail", "error_diff_lines": 500, "languages": ["py"]} + ) self.forbidden: dict[str, list[str]] = harness.get("FORBIDDEN", {}) - self.languages: tuple[str, ...] = harness.get("languages", {}) self.agents: dict[str, list[str]] = harness.get("agents", {}) self.commit_checks: dict[str, list[str]] = harness.get("preflight", {}) self.gate_checks: dict[str, list[str]] = harness.get("gate", {}) | self.commit_checks self.forbidden_files: tuple[str, ...] = tuple(self.forbidden.get("FILES", [])) self.forbidden_dirs: tuple[str, ...] = tuple(self.forbidden.get("DIRS", [])) self.forbidden_patterns: tuple[str, ...] = tuple(self.forbidden.get("PATTERNS", [])) - self.error_diff_lines: int = harness.get("error_diff_lines") def run_checks(self, checks: dict[str, list[str]]) -> dict[str, list[str]]: """Run each named command, streaming its output live under a phase header. @@ -80,7 +81,7 @@ def run_checks(self, checks: dict[str, list[str]]) -> dict[str, list[str]]: elif "format" in name: results["warn"].append(name) else: - results["fail"].append(name) + results[self.settings["behavior"]].append(name) key = "fail" if os.environ.get("RALPH_LOOP") else "warn" colorize("AGENT CHECKs", "running non-human agent checks") self._run_non_human_checks(results, key) @@ -173,16 +174,16 @@ def _check_diff_size(self, ref: str, results: dict[str, list[str]], key: str): inserted, deleted, path = line.split("\t", 2) if not (inserted == "-" or path.lower().endswith(".lock")): # binary or lockfile total += int(inserted) + int(deleted) - warn_at_75: int = round(self.error_diff_lines * 0.75) + warn_at_75: int = round(self.settings["error_diff_lines"] * 0.75) msg = ( f"{total} lines of code modified (insertions + deletions in staged files). Agents get WARN at " - f"75% {warn_at_75}, ERROR at {self.error_diff_lines}." + f"75% {warn_at_75}, ERROR at {self.settings['error_diff_lines']}." ) do_better = ( "\nRefactor bloat, reduce mis-direction, re-use fixtures, cut duplication, slim down " "code. More code does not mean good code." ) - if total > self.error_diff_lines: + if total > self.settings["error_diff_lines"]: results[key].append(msg + do_better) elif total > warn_at_75: results["warn"].append(msg + do_better) @@ -196,7 +197,7 @@ def _check_for_preferences(self) -> str: The preferences violations and filepath found in staged files. """ problems: list[str] = [] - if "py" in self.languages: + if "py" in self.settings["languages"]: staged = run_git([ "diff", "--cached", diff --git a/harness/ralph.ps1 b/harness/ralph.ps1 index dc325d8..a345f30 100644 --- a/harness/ralph.ps1 +++ b/harness/ralph.ps1 @@ -1,52 +1,84 @@ # Ralph (Windows twin of ralph.sh). Hand docs/PROMPT.md to a fresh-context agent and loop. # Keep Ralph Dumb: start the worker, give it the prompt, print a line, repeat. Nothing else. -# Windows has no POSIX `timeout`, so this uses Wait-Process + taskkill /T to bound each iteration. +# Windows has no POSIX `timeout`, so this uses Process.WaitForExit + taskkill /T. # -# Usage: pwsh -File ralph.ps1 +# Usage: +# powershell.exe -File ralph.ps1 $ErrorActionPreference = "Stop" [Console]::InputEncoding = [System.Text.UTF8Encoding]::new($false) -$env:RALPH_LOOP = "1" # mark loop commits so the gate applies containment to the worker +[Console]::OutputEncoding = [System.Text.UTF8Encoding]::new($false) + +# Mark loop commits so the gate (run by the git hooks) applies containment to the worker. +$env:RALPH_LOOP = "1" function ConvertTo-WindowsArgument([string]$argument) { $escaped = [regex]::Replace($argument, '(\\*)"', '$1$1\"') return '"' + $escaped + [regex]::Match($argument, '(\\*)$').Groups[1].Value + '"' } +function Write-RalphEvent([System.Collections.IDictionary]$payload) { + [Console]::Out.WriteLine(($payload | ConvertTo-Json -Compress)) + [Console]::Out.Flush() +} + +if ($args.Count -lt 3) { + [Console]::Error.WriteLine( + "Usage: ralph.ps1 " + ) + exit 2 +} + $maxIterations = [int]$args[0] $maxMinutes = [double]$args[1] -$rest = @($args | Select-Object -Skip 2) +$worker = @($args | Select-Object -Skip 2) +$timeoutMilliseconds = [int][Math]::Ceiling($maxMinutes * 60 * 1000) -for ($i = 1; $i -le $maxIterations; $i++) { - # Receipt line, same stdout contract as ralph.sh: `harness run` saves stdout as the run's .jsonl. - # The worker inherits this handle, so flush before starting it or the records interleave. - $timestamp = (Get-Date).ToString("yyyy-MM-ddTHH:mm") - [Console]::Out.WriteLine("{""type"":""ralph"",""iteration"":$i,""max_iterations"":$maxIterations,""max_minutes"":$maxMinutes,""timestamp"":""$timestamp""}") - [Console]::Out.Flush() - $stdin = "$($env:RALPH_PROMPT)`n`nRALPH_ITERATION=$i/$maxIterations`n" - $psi = [System.Diagnostics.ProcessStartInfo]::new() - $psi.FileName = $rest[0] - if ($rest.Count -gt 1) { - $psi.Arguments = (($rest[1..($rest.Count - 1)] | ForEach-Object { ConvertTo-WindowsArgument $_ }) -join ' ') +$iteration = 1 +while ($iteration -le $maxIterations) { + Write-RalphEvent ([ordered]@{ + type = "ralph" + iteration = $iteration + max_iterations = $maxIterations + max_minutes = $maxMinutes + timestamp = (Get-Date).ToString("yyyy-MM-ddTHH:mm") + }) + + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = $worker[0] + if ($worker.Count -gt 1) { + $startInfo.Arguments = (($worker[1..($worker.Count - 1)] | ForEach-Object { + ConvertTo-WindowsArgument $_ + }) -join ' ') } - $psi.RedirectStandardInput = $true - $psi.UseShellExecute = $false - $proc = [System.Diagnostics.Process]::Start($psi) - # feed the prompt on stdin, then bound the run; taskkill /T kills the agent AND its children - $proc.StandardInput.Write($stdin); $proc.StandardInput.Close() - # Bound the run. Like ralph.sh's `set -e` + timeout: a timeout or a nonzero worker exit stops the - # loop and propagates failure, so `harness run` never reports success for a failed iteration. - $timeoutMilliseconds = [int][Math]::Ceiling($maxMinutes * 60 * 1000) - if (-not $proc.WaitForExit($timeoutMilliseconds)) { - taskkill.exe /F /T /PID $proc.Id | Out-Null - $proc.WaitForExit() - exit 124 # match GNU timeout's exit code + $startInfo.RedirectStandardInput = $true + $startInfo.UseShellExecute = $false + + $process = [System.Diagnostics.Process]::Start($startInfo) + $prompt = "$($env:RALPH_PROMPT)`n`nRALPH_ITERATION=$iteration/$maxIterations`n" + $process.StandardInput.Write($prompt) + $process.StandardInput.Close() + + if (-not $process.WaitForExit($timeoutMilliseconds)) { + taskkill.exe /F /T /PID $process.Id | Out-Null + $process.WaitForExit() + $process.Dispose() + exit 124 } - if ($proc.ExitCode -ne 0) { - exit $proc.ExitCode + + $process.WaitForExit() + $exitCode = $process.ExitCode + $process.Dispose() + if ($exitCode -ne 0) { + exit $exitCode } + + $iteration += 1 } -$timestamp = (Get-Date).ToString("yyyy-MM-ddTHH:mm") -[Console]::Out.WriteLine("{""type"":""ralph"",""completed"":$maxIterations,""max_minutes"":$maxMinutes,""timestamp"":""$timestamp""}") -[Console]::Out.Flush() +Write-RalphEvent ([ordered]@{ + type = "ralph" + completed = $iteration - 1 + max_minutes = $maxMinutes + timestamp = (Get-Date).ToString("yyyy-MM-ddTHH:mm") +}) diff --git a/harness/temp.pyproject.toml b/harness/temp.pyproject.toml index 931545c..0210aea 100644 --- a/harness/temp.pyproject.toml +++ b/harness/temp.pyproject.toml @@ -22,6 +22,8 @@ dev = [ # Humans own this file (it is agent-forbidden below). # ============================================================================== [tool.harness] +[tool.harness.settings] +behavior = "fail" # "fail" or "warn": change to "warn" if your project has too many blocking failures to start languages = ["py"] error_diff_lines = 500 # 400 ~90th percentile of pull requests, 200 LOC generally ok @@ -96,12 +98,12 @@ test = [ ] [tool.harness.FORBIDDEN] -DIRS = ["harness/", ".githooks/", ".github/", ".git/", "preferences/", "tests/preferences/"] +DIRS = ["harness/", ".githooks/", ".github/", ".git/", "preferences/", "tests/preferences/", "mutation", "tests/mutation/"] FILES = [ "agents.md", "docs/prompt.md", "docs/plan.md", # delete/comment line if you want agents to manage the core plan - # tooling/config files that would weaken checks in this file + # tooling/config files that would weaken checks: "tox.ini", "setup.cfg", "pytest.ini", diff --git a/harness/tests/test_cli.py b/harness/tests/test_cli.py index 9a6364a..bbd13d1 100644 --- a/harness/tests/test_cli.py +++ b/harness/tests/test_cli.py @@ -11,10 +11,9 @@ import sys from datetime import datetime, timezone from pathlib import Path -from shutil import which from types import SimpleNamespace from typing import TYPE_CHECKING -from unittest.mock import DEFAULT, Mock, call +from unittest.mock import DEFAULT, Mock, call, create_autospec import pytest import tomlkit as tomllib @@ -276,6 +275,7 @@ def test_cli_summaries_report_complete_agent_check_results( exit_code, summary = expected monkeypatch.setenv("RALPH_LOOP", "1") monkeypatch.setattr(gates(), "commit_checks" if command == "preflight" else "gate_checks", {}) + monkeypatch.setattr(cli.console, "print", Mock(wraps=cli.console.print)) source_path = git_repo / "src" / "mod.py" source_path.parent.mkdir() source_path.write_text(source, encoding="utf-8") @@ -291,7 +291,7 @@ def test_cli_summaries_report_complete_agent_check_results( ) diff_size_output = ( "PHASE: DIFF SIZE 1 lines of code modified (insertions + deletions in staged files). " - "Agents get WARN at 75% 300, ERROR at 400. " + "Agents get WARN at 75% 375, ERROR at 500. " ) mutation_output = ( "killed 132 survived 0 total 133 no_tests 0 skipped 0 suspicious 0 timeout 1 " @@ -302,6 +302,16 @@ def test_cli_summaries_report_complete_agent_check_results( assert diff_size_output in output assert mutation_output in output assert output.endswith(unstyle(summary)) + assert ( + cli.console.print.call_args_list[-2].args[0].title_style, + cli.console.print.call_args_list[-2].args[0].box, + cli.console.print.call_args_list[-2].args[0].padding, + [(column.header, column.style) for column in cli.console.print.call_args_list[-2].args[0].columns], + ) == ("bold grey82", None, (0, 5, 0, 5), [("RESULT", ""), ("CHECK", "bold dim white")]) + assert [printed.kwargs for printed in cli.console.print.call_args_list[-2:]] == [ + {"justify": "center"}, + {"justify": "center"}, + ] def test_status_counts_run_receipts_and_names_the_newest(git_repo: Path) -> None: @@ -338,10 +348,15 @@ def test_setup_git_hooks_records_exact_posix_commands( monkeypatch.setattr(cli.subprocess, "run", run) monkeypatch.setattr(cli, "rprint", print_message) + write_text = create_autospec(Path.write_text, wraps=Path.write_text) + monkeypatch.setattr(Path, "write_text", write_text) recorded = cli.setup_git_hooks(env_bin) assert recorded == git_dir / "harness-path" assert recorded.read_text(encoding="utf-8") == f"{(env_bin / 'harness').as_posix()}\n" + write_text.assert_called_once_with( + recorded, f"{(env_bin / 'harness').as_posix()}\n", encoding="utf-8", newline="\n" + ) run_git.assert_called_once_with(["rev-parse", "--git-common-dir"]) assert run.call_args_list == [ call(["git", "config", "core.hooksPath", ".githooks"], cwd=str(tmp_path), check=True), @@ -470,10 +485,18 @@ def test_write_harness_config_aborts_when_existing_wiring_is_declined( original = '[tool.harness.gate]\ntest = ["pytest"]\n' pyproject.write_text(original, encoding="utf-8") monkeypatch.setattr(cli, "TOOLS", {}) + monkeypatch.setattr(Path, "is_file", create_autospec(Path.is_file, wraps=Path.is_file)) + monkeypatch.setattr(Path, "read_text", create_autospec(Path.read_text, wraps=Path.read_text)) + monkeypatch.setattr(cli, "confirm", Mock(side_effect=Abort())) - with runner.isolation(input="n\n"), pytest.raises(Abort): + with pytest.raises(Abort): cli.write_harness_config() + Path.is_file.assert_called_once_with(pyproject) + Path.read_text.assert_called_once_with(pyproject, encoding="utf-8") + cli.confirm.assert_called_once_with( + cli.style("Seems loopgate may be wired already. Continue?", fg=10), default=True, abort=True + ) assert pyproject.read_text(encoding="utf-8") == original @@ -485,7 +508,22 @@ def test_write_harness_config_selects_installed_user_tools( monkeypatch.setattr(cli.util, "find_spec", Mock(side_effect=installed.get)) monkeypatch.setattr(cli, "which", Mock(return_value=None)) (git_repo / "pyproject.toml").write_text("[tool.ruff]\nline-length = 99\n", encoding="utf-8") + monkeypatch.setattr(Path, "with_name", create_autospec(Path.with_name, wraps=Path.with_name)) + monkeypatch.setattr(Path, "read_text", create_autospec(Path.read_text, wraps=Path.read_text)) + monkeypatch.setattr(Path, "write_text", create_autospec(Path.write_text, wraps=Path.write_text)) + monkeypatch.setattr( + cli.ConfigParser, "read", create_autospec(cli.ConfigParser.read, wraps=cli.ConfigParser.read) + ) cli.write_harness_config() + Path.with_name.assert_called_once_with(Path(cli.__file__).resolve(), "temp.pyproject.toml") + assert Path.read_text.call_args_list == [ + call(git_repo / "pyproject.toml", encoding="utf-8"), + call(Path(cli.__file__).resolve().parent / "temp.pyproject.toml", encoding="utf-8"), + ] + assert Path.write_text.call_args.args[0] == git_repo / "pyproject.toml" + assert Path.write_text.call_args.kwargs == {"encoding": "utf-8"} + assert cli.ConfigParser.read.call_args.args[1:] == ([git_repo / "tox.ini", git_repo / "setup.cfg"],) + assert cli.ConfigParser.read.call_args.kwargs == {"encoding": "utf-8"} ruff_lint_no_format = tomllib.parse((git_repo / "pyproject.toml").read_text(encoding="utf-8")) assert ruff_lint_no_format["tool"]["harness"]["preflight"] == { "complexity": ["complexipy", ".", "--suggest-refactors"], @@ -651,139 +689,20 @@ def test_init_declines_without_mutating_repo(git_repo: Path) -> None: assert "Run `harness init`" in fresh.stdout -def test_init_hoists_and_records_the_installed_harness(tmp_path: Path) -> None: - """The installed console command initializes a disposable repo and home from packaged assets.""" - git_repo = tmp_path - gate.run_git(["init", "-q"], git_repo) - (git_repo / ".githooks").mkdir() - (git_repo / "uv.lock").touch() - (git_repo / "pyproject.toml").write_text( - f"{INIT_PROJECT_COMMENT}\n[project]\nname = 'existing'\n{INIT_PROJECT_COMMENT}", encoding="utf-8" - ) - (git_repo / ".githooks" / "pre-commit").write_bytes(b"#!/bin/sh\nprintf '%s\\n' existing-pre-commit\n") - (git_repo / ".githooks" / "pre-commit").chmod(0o755) - executable = Path(sys.executable).with_name("harness.exe" if sys.platform == "win32" else "harness") - git_path = which("git") - assert executable.is_file() - assert git_path is not None - home = git_repo / "home" - bin_path = git_repo / "bin" - bin_path.mkdir() - (bin_path / "timeout").write_text("#!/bin/sh\nexit 0\n", encoding="utf-8") - (bin_path / "timeout").chmod(0o755) - environment = {key: value for key, value in os.environ.items() if not key.startswith("UV_")} | { - "GIT_CONFIG_GLOBAL": os.devnull, - "GIT_CONFIG_NOSYSTEM": "1", - "HOME": str(home), - "PATH": os.pathsep.join(( - str(bin_path), - str(executable.parent), - str(Path(git_path).parent), - os.defpath, - )), - "USERPROFILE": str(home), - "VIRTUAL_ENV": str(executable.parent.parent), - "XDG_CONFIG_HOME": str(home / ".config"), - } - environment.pop("PYTHONPATH", None) - installed_assets = assert_installed_config_paths(environment, git_repo) - packaged_pre_push = (installed_assets["githooks"][0] / "pre-push").read_bytes() - (git_repo / ".githooks" / "pre-push").write_bytes( - packaged_pre_push.partition(b"\n")[0] - + b'\n"$(dirname "$0")/loopgate-pre-push" "$@" || exit # loopgate\n' - + packaged_pre_push.partition(b"\n")[2] - ) - (git_repo / ".githooks" / "loopgate-pre-push").write_bytes(packaged_pre_push) - - result = subprocess.run( - [str(executable), "init"], - cwd=git_repo, - input="y\n" * 5, - capture_output=True, - text=True, - env=environment, - check=False, - ) - - assert (result.returncode, result.stderr) == (0, "") - assert "RESULT:" in unstyle(result.stdout) - assert "Can likely run loops: True" in unstyle(result.stdout) - assert "Ensure your environemnt is activated" in unstyle(result.stdout) - assert ( - subprocess.run( - [ - git_path, - "-c", - 'alias.loopgate-hoist=!f() { sh "$1"; }; f', - "loopgate-hoist", - (installed_assets["githooks"][0] / "hoist").as_posix(), - ], - cwd=git_repo, - capture_output=True, - text=True, - env=environment, - check=False, - ).returncode - == 0 - ) - written_pyproject = (git_repo / "pyproject.toml").read_text(encoding="utf-8") - assert tomllib.loads(written_pyproject)["project"] == {"name": "existing"} - assert written_pyproject.count(INIT_PROJECT_COMMENT) == 2 - assert normalized_path( - (git_repo / ".git" / "harness-path").read_text(encoding="utf-8").strip() - ) == normalized_path(executable) - assert gate.run_git(["config", "--get", "core.hooksPath"], git_repo).strip() == ".githooks" - assert (git_repo / "scratchpad" / "runs" / ".gitkeep").is_file() - assert (git_repo / ".githooks" / "pre-commit").read_bytes() == ( - b"#!/bin/sh\n(\n" - b' . "$(dirname "$0")/_resolve"\n' - b' exec "$HARNESS" preflight\n' - b") || exit # loopgate\n" - b"printf '%s\\n' existing-pre-commit\n" - ) - for name in ("pre-push", "prepare-commit-msg"): - assert (git_repo / ".githooks" / name).read_bytes() == ( - installed_assets["githooks"][0] / name - ).read_bytes() - assert (git_repo / ".githooks" / "_resolve").read_bytes() == ( - installed_assets["githooks"][0] / "_resolve" - ).read_bytes() - assert all( - os.access(git_repo / ".githooks" / name, os.X_OK) - for name in ( - "pre-commit", - "pre-push", - "prepare-commit-msg", - ) - ) - assert not list((git_repo / ".githooks").glob("loopgate-*")) - assert not list((git_repo / ".githooks").glob(".loopgate-original-*")) - assert { - "docs/PROMPT.md": (git_repo / "docs" / "PROMPT.md").read_bytes(), - "mutation/check_mutmut.py": (git_repo / "mutation" / "check_mutmut.py").read_bytes(), - "preferences/preferences.py": (git_repo / "preferences" / "preferences.py").read_bytes(), - "tests/mutation/test_check_mutmut.py": ( - git_repo / "tests" / "mutation" / "test_check_mutmut.py" - ).read_bytes(), - "tests/preferences/test_preferences.py": ( - git_repo / "tests" / "preferences" / "test_preferences.py" - ).read_bytes(), - } == { - "docs/PROMPT.md": (installed_assets["docs"][0] / "PROMPT.md").read_bytes(), - "mutation/check_mutmut.py": (installed_assets["mutation"][0] / "check_mutmut.py").read_bytes(), - "preferences/preferences.py": (installed_assets["preferences"][0] / "preferences.py").read_bytes(), - "tests/mutation/test_check_mutmut.py": ( - installed_assets["mutation_tests"][0] / "test_check_mutmut.py" - ).read_bytes(), - "tests/preferences/test_preferences.py": ( - installed_assets["pref_tests"][0] / "test_preferences.py" - ).read_bytes(), - } - - -def test_init_writes_config_before_hook_consent(monkeypatch: pytest.MonkeyPatch, git_repo: Path) -> None: - """Declining required hooks aborts after config generation and before any asset or hook mutation.""" +def test_init_hoists_and_records_the_installed_harness( + monkeypatch: pytest.MonkeyPatch, git_repo: Path +) -> None: + """Init handles hook consent, hoisting, and recording the installed harness.""" monkeypatch.setattr(cli, "TOOLS", {}) + bin_dir = git_repo / "bin" + bin_dir.mkdir() + write_executable(bin_dir / "gtimeout", "#!/bin/sh\nexit 0\n") + monkeypatch.setenv("PATH", f"{bin_dir}{os.pathsep}{os.environ['PATH']}") + monkeypatch.setattr(Path, "home", fake_home(git_repo / "home")) + installed_assets = assert_installed_config_paths( + {key: value for key, value in os.environ.items() if key != "PYTHONPATH"}, git_repo + ) + assert (installed_assets["githooks"][0] / "hoist").is_file() hoist = Mock(return_value=False) monkeypatch.setattr(cli, "hoist", hoist) @@ -794,29 +713,37 @@ def test_init_writes_config_before_hook_consent(monkeypatch: pytest.MonkeyPatch, assert "harness" in tomllib.loads((git_repo / "pyproject.toml").read_text(encoding="utf-8"))["tool"] hoist.assert_not_called() - retry = runner.invoke(cli.app, ["init"], input="y\n" * 5) + result = runner.invoke(cli.app, ["init"], input="y\n" * 5) - assert retry.exit_code == 0, retry.output - assert "quality checks run?" in retry.output - assert "Can likely run loops: False" in unstyle(retry.output) + assert result.exit_code == 0, result.output + assert "2. Can we wire githooks so quality checks run?" in result.output + retry_output = unstyle(result.output) + assert "Can likely run loops: False" in retry_output + assert "Success. Try running loops with `harness run `" not in retry_output + assert "macOS harness needs timeout/gtimeout from coreutils" not in retry_output + assert "Install `brew install coreutils` now?" not in retry_output hoist.assert_called_once_with() + recorded = git_repo / ".git" / "harness-path" + assert ( + recorded.read_text(encoding="utf-8") + == (Path(sys.executable).parent / ("harness.exe" if cli.IS_WINDOWS else "harness")).as_posix() + "\n" + ) hoist.return_value = True - (git_repo / ".git" / "harness-path").write_text("harness\n", encoding="utf-8") - setup_hooks = Mock(return_value=git_repo / ".git" / "harness-path") - timeout = Mock() + setup_hooks = Mock(return_value=recorded) configure = Mock() monkeypatch.setattr(cli, "setup_git_hooks", setup_hooks) - monkeypatch.setattr(cli, "check_for_timeout_and_prompt", timeout) monkeypatch.setattr(cli, "configure_agents", configure) - success = runner.invoke(cli.app, ["init"], input="y\n" * 3) + result = runner.invoke(cli.app, ["init"], input="y\n" * 3) - assert success.exit_code == 0, success.output - assert "Can likely run loops: True" in unstyle(success.output) + assert result.exit_code == 0, result.output + success_output = unstyle(result.output) + assert "files added: True" in success_output + assert "Can likely run loops: True" in success_output + assert hoist.call_args_list == [call(), call()] setup_hooks.assert_called_once_with(Path(sys.executable).parent) - timeout.assert_called_once_with() configure.assert_called_once_with() @@ -854,6 +781,9 @@ def test_hoist_rejects_missing_required_assets(monkeypatch: pytest.MonkeyPatch, assert cli.hoist() is False message.assert_called_once_with("Harness is missing required assets: `docs/` and `githooks/`") + (tmp_path / "docs").mkdir() + (tmp_path / "missing-hooks").mkdir() + assert cli.hoist() is False repo = tmp_path / "repo" package_docs = tmp_path / "package" / "docs" @@ -868,23 +798,47 @@ def test_hoist_rejects_missing_required_assets(monkeypatch: pytest.MonkeyPatch, monkeypatch.setattr( cli, "ASSETS", - { - "docs": (package_docs, repo / "docs"), - "githooks": (package_hooks, repo / ".githooks"), - }, + {"docs": (package_docs, repo / "docs"), "githooks": (package_hooks, repo / ".githooks")}, ) monkeypatch.setattr(cli, "REPO_ROOT", repo) confirm = Mock(return_value=True) run_git = Mock() + message.reset_mock() monkeypatch.setattr(cli, "confirm", confirm) monkeypatch.setattr(cli, "run_git", run_git) + monkeypatch.setattr(Path, "mkdir", create_autospec(Path.mkdir, wraps=Path.mkdir)) + monkeypatch.setattr(Path, "touch", create_autospec(Path.touch, wraps=Path.touch)) + monkeypatch.setattr(cli.console, "print", Mock(wraps=cli.console.print)) assert cli.hoist() is True assert (repo / "docs" / "nested" / "new.txt").read_text(encoding="utf-8") == "new\n" assert (repo / "docs" / "existing.txt").read_text(encoding="utf-8") == "existing\n" assert (repo / "scratchpad" / "runs" / ".gitkeep").is_file() - confirm.assert_called_once() - assert confirm.call_args.kwargs == {"default": True, "abort": True} + confirm.assert_called_once_with( + cli.style( + "3. Confirm, loopgate can add those files? Pre-existing files in the expected paths will remain " + "and loopgate will skip adding them.", + fg=10, + ), + default=True, + abort=True, + ) + Path.mkdir.assert_any_call(repo / ".githooks", parents=True, exist_ok=True) + assert Path.mkdir.call_args_list.count(call(repo / "docs" / "nested", parents=True, exist_ok=True)) == 2 + Path.mkdir.assert_any_call(repo / "scratchpad" / "runs", parents=True, exist_ok=True) + Path.touch.assert_called_once_with(repo / "scratchpad" / "runs" / ".gitkeep") + cli.console.print.assert_called_once_with(f"[green]\n4. Ran {(package_hooks / 'hoist').as_posix()}[/]\n") + assert message.call_args_list[0] == call( + "\n[bold yellow]We will need to add these files[/]\n* Git hooks are what ensure quality checks run" + "\n* Mutation tests promote good tests.\n* `preferences` allow for checks beyond what tooling catches" + "and demonstrate Hypothesis property tests\n* `docs` contain the instructions and memory for loops " + "\n*`scratchpad/` allows local agent use and contains a `runs/` directory for logs." + ) + assert message.call_args_list[1:3] == [ + call(f"`docs/` exists {(repo / 'docs').exists()}"), + call(f"`githooks/` exists {(repo / '.githooks').exists()}"), + ] + assert message.call_args_list[-1] == call(f"`scratchpad/` and `runs/` also added at {repo}") run_git.assert_called_once_with( [ "-c", @@ -906,10 +860,7 @@ def test_hoist_aborts_before_writing_when_declined(monkeypatch: pytest.MonkeyPat monkeypatch.setattr( cli, "ASSETS", - { - "docs": (package_docs, repo / "docs"), - "githooks": (package_hooks, repo / ".githooks"), - }, + {"docs": (package_docs, repo / "docs"), "githooks": (package_hooks, repo / ".githooks")}, ) with runner.isolation(input="n\n"), pytest.raises(Abort): @@ -969,6 +920,10 @@ def test_installing_the_template_cleans_the_repo_and_sets_hooks( toolchain = stub_toolchain(git_repo / ".git") monkeypatch.setattr(subprocess, "run", toolchain) + monkeypatch.setattr(Path, "is_file", create_autospec(Path.is_file, wraps=Path.is_file)) + monkeypatch.setattr(Path, "replace", create_autospec(Path.replace, wraps=Path.replace)) + monkeypatch.setattr(Path, "unlink", create_autospec(Path.unlink, wraps=Path.unlink)) + monkeypatch.setattr(cli, "rmtree", Mock(wraps=cli.rmtree)) result = runner.invoke(cli.app, ["install"]) assert result.exit_code == 0 @@ -983,6 +938,15 @@ def test_installing_the_template_cleans_the_repo_and_sets_hooks( assert not (git_repo / "harness" / "tests").exists() assert (git_repo / "preferences").is_dir() assert (git_repo / "tests" / "preferences").is_dir() + Path.is_file.assert_any_call(git_repo / "README.template.md") + Path.is_file.assert_any_call(git_repo / "temp.pyproject.toml") + Path.replace.assert_any_call(git_repo / "README.template.md", git_repo / "README.md") + Path.replace.assert_any_call(git_repo / "temp.pyproject.toml", git_repo / "pyproject.toml") + Path.unlink.assert_any_call(git_repo / ".github" / "workflows" / "publish.yml", missing_ok=True) + Path.unlink.assert_any_call(git_repo / "CONTRIBUTING.md", missing_ok=True) + cli.rmtree.assert_any_call(git_repo / "dist") + cli.rmtree.assert_any_call(git_repo / "harness" / "tests") + cli.rmtree.assert_any_call(git_repo / ".assets") toolchain.assert_any_call(("uv", "sync"), cwd=str(git_repo), check=True) recorded_harness = (git_repo / ".git" / "harness-path").read_text(encoding="utf-8").strip() env_bin = git_repo / ".venv" / ("Scripts" if sys.platform == "win32" else "bin") @@ -1076,15 +1040,20 @@ def test_install_picks_the_package_manager_from_project_signals( """ scripts = "Scripts" if sys.platform == "win32" else "bin" python_name = "python.exe" if sys.platform == "win32" else "python" - interpreter = git_repo / ".pyenv" / scripts + interpreter = git_repo / (virtual_env or ".pyenv") / scripts poetry_bin = git_repo / ".poetry" / "virtualenvs" / "project" / scripts monkeypatch.setenv("UV_TESTING", "1") for name in tuple(os.environ): if name.startswith("UV_"): monkeypatch.delenv(name) - monkeypatch.setenv("VIRTUAL_ENV", str(git_repo / virtual_env) if virtual_env else "") + assert not any(name.startswith("UV_") for name in os.environ) + if virtual_env == "uv-managed": + monkeypatch.setenv("UV_TESTING", "1") + monkeypatch.delenv("VIRTUAL_ENV", raising=False) + monkeypatch.setattr(cli.os.environ, "get", Mock(wraps=cli.os.environ.get)) + monkeypatch.setattr(Path, "is_file", create_autospec(Path.is_file, wraps=Path.is_file)) monkeypatch.setattr(cli.sys, "executable", str(interpreter / python_name)) - monkeypatch.setattr(cli, "which", which_finds(("timeout",))) + monkeypatch.setattr(cli, "which", Mock(wraps=which_finds(("timeout",)))) monkeypatch.setattr(cli, "REPO_ROOT_STR", str(git_repo)) (git_repo / "pyproject.toml").write_text('[project]\nname = "x"\n', encoding="utf-8") if lockfile: @@ -1108,29 +1077,36 @@ def test_install_picks_the_package_manager_from_project_signals( assert [call for call in calls if call in managers.values()] == [managers[manager]] installed = (git_repo / ".git" / "harness-path").read_text(encoding="utf-8").strip() assert normalized_path(installed) == normalized_path(recorded[manager]) + cli.os.environ.get.assert_any_call("VIRTUAL_ENV", "") + Path.is_file.assert_any_call(git_repo / "uv.lock") + if lockfile != "uv.lock": + Path.is_file.assert_any_call(git_repo / "poetry.lock") + if not lockfile and virtual_env != "uv-managed": + cli.which.assert_any_call("uv") @pytest.mark.parametrize( - ("on_path", "answer", "outcome"), + ("on_path", "answer", "installs_coreutils", "offers_coreutils", "shows_homebrew_hint"), [ - pytest.param(("timeout",), None, (False, ""), id="timeout-present"), - pytest.param(("gtimeout",), None, (False, ""), id="gtimeout-present"), - pytest.param((), None, (False, "brew.sh"), id="no-timeout-no-homebrew"), - pytest.param(("brew",), True, (True, ""), id="confirmed"), - pytest.param(("brew",), False, (False, ""), id="declined"), + pytest.param(("timeout",), None, False, False, False, id="timeout-present"), + pytest.param(("gtimeout",), None, False, False, False, id="gtimeout-present"), + pytest.param((), None, False, True, True, id="no-timeout-no-homebrew"), + pytest.param(("brew",), True, True, True, False, id="confirmed"), + pytest.param(("brew",), False, False, True, False, id="declined"), ], ) def test_install_offers_coreutils_only_when_no_timeout_tool_exists( on_path: tuple[str, ...], answer: bool | None, - outcome: tuple[bool, str], + installs_coreutils: bool, + offers_coreutils: bool, + shows_homebrew_hint: bool, monkeypatch: pytest.MonkeyPatch, git_repo: Path, ) -> None: """macOS needs coreutils to time out a loop iteration, so install probes for it and offers the install only when Homebrew can do it. It never prompts when a timeout tool is already there. """ - installs_coreutils, hint = outcome prompts: list[str] = [] def confirm(prompt: str, abort: bool = False) -> bool: @@ -1148,11 +1124,13 @@ def confirm(prompt: str, abort: bool = False) -> bool: result = runner.invoke(cli.app, ["install"]) calls = [tuple(record.args[0]) for record in toolchain.call_args_list] + output = unstyle(result.stdout) assert result.exit_code == 0 assert (("brew", "install", "coreutils") in calls) is installs_coreutils assert prompts == ([] if answer is None else ["\nInstall `brew install coreutils` now?"]) - assert hint in result.stdout + assert ("macOS harness needs timeout/gtimeout from coreutils" in output) is offers_coreutils + assert ("Get Homebrew https://brew.sh" in output) is shows_homebrew_hint def test_windows_skips_posix_steps_and_launches_the_powershell_twin( @@ -1416,10 +1394,18 @@ def test_run_worker_logs_every_line_and_streams_only_when_verbose( streaming_worker = [sys.executable, "-c", 'print(\'{ "type" : "result" }\'); print("not json")'] real_popen = subprocess.Popen popen = Mock(wraps=real_popen) + monkeypatch.setattr(Path, "open", create_autospec(Path.open, wraps=Path.open)) monkeypatch.setattr(cli.subprocess, "Popen", popen) + monkeypatch.setattr(cli, "JSON", Mock(wraps=cli.JSON)) + monkeypatch.setattr(cli.console, "print", Mock(wraps=cli.console.print)) assert cli.run_worker(streaming_worker, log, verbose=True) == 0 popen.assert_called_once_with(streaming_worker, cwd=str(tmp_path), stdout=subprocess.PIPE, text=True) + assert cli.JSON.call_args_list == [ + call('{ "type" : "result" }\n', indent=None), + call("not json\n", indent=None), + ] + assert [printed.kwargs for printed in cli.console.print.call_args_list] == [{"end": "\n"}, {"end": "\n"}] streamed = capsys.readouterr().out assert '"type"' in streamed @@ -1440,6 +1426,7 @@ def test_run_worker_logs_every_line_and_streams_only_when_verbose( assert not capsys.readouterr().out assert log.read_text(encoding="utf-8") == "worker output\n" + assert Path.open.call_args_list[::2] == [call(log, "w", encoding="utf-8")] * 2 def test_claude_preset_runs_two_real_loop_iterations(monkeypatch: pytest.MonkeyPatch, git_repo: Path) -> None: diff --git a/harness/tests/test_gate.py b/harness/tests/test_gate.py index 2badf27..d31f2ab 100644 --- a/harness/tests/test_gate.py +++ b/harness/tests/test_gate.py @@ -21,7 +21,7 @@ from harness.tests.conftest import REPO_ROOT, fake_popen from mutation import check_mutmut -WARNING_THRESHOLD = round(gates().error_diff_lines * 0.75) +WARNING_THRESHOLD = round(gates().settings["error_diff_lines"] * 0.75) def stage(repo: Path, name: str, content: str) -> None: @@ -158,10 +158,12 @@ def test_pre_commit_hook_dispatches_preflight_and_controls_commit( pytest.param((WARNING_THRESHOLD, WARNING_THRESHOLD, True, None), id="at-warning-threshold"), pytest.param((WARNING_THRESHOLD + 1, WARNING_THRESHOLD + 1, True, "WARNED"), id="warn"), pytest.param( - (gates().error_diff_lines, gates().error_diff_lines, True, "WARNED"), id="at-error-threshold" + (gates().settings["error_diff_lines"], gates().settings["error_diff_lines"], True, "WARNED"), + id="at-error-threshold", ), pytest.param( - (gates().error_diff_lines + 1, WARNING_THRESHOLD, True, None), id="unstaged-lines-ignored" + (gates().settings["error_diff_lines"] + 1, WARNING_THRESHOLD, True, None), + id="unstaged-lines-ignored", ), ], ) @@ -487,19 +489,18 @@ def test_gate_runs_exactly_what_pyproject_configures( ) -> None: """A root owns its complete configuration, Git target, command dispatch, and containment results.""" monkeypatch.setenv("RALPH_LOOP", "1") - raw_toml = tomllib.loads((REPO_ROOT / "pyproject.toml").read_bytes().decode())["tool"]["harness"] + raw_harness_toml = tomllib.loads((REPO_ROOT / "pyproject.toml").read_bytes().decode())["tool"]["harness"] configured = Gate(REPO_ROOT) assert vars(configured) == { "repo_root": REPO_ROOT, - "forbidden": raw_toml["FORBIDDEN"], - "languages": raw_toml["languages"], - "agents": raw_toml["agents"], - "commit_checks": raw_toml["preflight"], - "gate_checks": raw_toml["gate"] | raw_toml["preflight"], - "forbidden_files": tuple(raw_toml["FORBIDDEN"]["FILES"]), - "forbidden_dirs": tuple(raw_toml["FORBIDDEN"]["DIRS"]), - "forbidden_patterns": tuple(raw_toml["FORBIDDEN"]["PATTERNS"]), - "error_diff_lines": raw_toml["error_diff_lines"], + "settings": raw_harness_toml["settings"], + "forbidden": raw_harness_toml["FORBIDDEN"], + "agents": raw_harness_toml["agents"], + "commit_checks": raw_harness_toml["preflight"], + "gate_checks": raw_harness_toml["gate"] | raw_harness_toml["preflight"], + "forbidden_files": tuple(raw_harness_toml["FORBIDDEN"]["FILES"]), + "forbidden_dirs": tuple(raw_harness_toml["FORBIDDEN"]["DIRS"]), + "forbidden_patterns": tuple(raw_harness_toml["FORBIDDEN"]["PATTERNS"]), } assert vars(gates()) == {**vars(configured), "repo_root": git_repo} (git_repo / "pyproject.toml").write_text("[project]\nname = 'x'\n", encoding="utf-8") @@ -507,6 +508,7 @@ def test_gate_runs_exactly_what_pyproject_configures( "tool" ]["harness"] assert Gate(git_repo).gate_checks == fallback["preflight"] | fallback["gate"] + assert Gate(git_repo).settings == fallback["settings"] expected_repo_root = Path( subprocess.check_output(["git", "rev-parse", "--show-toplevel"], cwd=REPO_ROOT, text=True).strip() ) @@ -611,7 +613,7 @@ def test_gate_runs_exactly_what_pyproject_configures( diff_size_output = ( "\nPHASE: DIFF SIZE" "\n28 lines of code modified (insertions + deletions in staged files). " - "Agents get WARN at 75% 300, ERROR at 400.\n" + "Agents get WARN at 75% 375, ERROR at 500.\n" ) assert printed.startswith(phase_output) assert diff_size_output in printed @@ -627,6 +629,18 @@ def test_gate_runs_exactly_what_pyproject_configures( assert "Run this inside a git repository" in capfd.readouterr().out gates.cache_clear() + monkeypatch.chdir(git_repo) + gates.cache_clear() + gate.run_git(["reset", "-q"], git_repo) + monkeypatch.setitem(gates().settings, "behavior", "warn") + live = gates().run_checks({ + "ruff lint": [sys.executable, "-c", "print('hello from the check')"], + "pyright types": [sys.executable, "-c", "raise SystemExit(7)"], + "ruff format": [sys.executable, "-c", "raise SystemExit(1)"], + }) + assert live == {"pass": ["ruff lint", "mutmut"], "fail": [], "warn": ["pyright types", "ruff format"]} + gates.cache_clear() + @pytest.mark.parametrize( ("score", "loop", "bucket"), @@ -689,8 +703,8 @@ def test_diff_size_counts_only_relevant_changed_lines( [ pytest.param(WARNING_THRESHOLD, "quiet", id="at-warn"), pytest.param(WARNING_THRESHOLD + 1, "advised", id="over-warn"), - pytest.param(gates().error_diff_lines, "advised", id="at-cap"), - pytest.param(gates().error_diff_lines + 1, "blocked", id="over-cap"), + pytest.param(gates().settings["error_diff_lines"], "advised", id="at-cap"), + pytest.param(gates().settings["error_diff_lines"] + 1, "blocked", id="over-cap"), ], ) def test_diff_size_warns_then_blocks_as_the_change_grows( @@ -710,7 +724,7 @@ def test_diff_size_warns_then_blocks_as_the_change_grows( message = ( f"{lines} lines of code modified (insertions + deletions in staged files). " - f"Agents get WARN at 75% {WARNING_THRESHOLD}, ERROR at {gates().error_diff_lines}." + f"Agents get WARN at 75% {WARNING_THRESHOLD}, ERROR at {gates().settings['error_diff_lines']}." ) advisory = message + ( "\nRefactor bloat, reduce mis-direction, re-use fixtures, cut duplication, slim down code. " @@ -751,7 +765,7 @@ def test_diff_size_ignores_changes_with_nothing_staged( stage(git_repo, filepaths[0], "value = 1\n") gate.run_git(["commit", "-q", "-m", "seed the file"], git_repo) (git_repo / "src" / filepaths[1]).write_text( - "value = 1\n" * (gates().error_diff_lines + 2), encoding="utf-8" + "value = 1\n" * (gates().settings["error_diff_lines"] + 2), encoding="utf-8" ) monkeypatch.setattr(gates(), "commit_checks", {}) @@ -930,12 +944,12 @@ def test_preferences_only_ever_read_python( assert gates().run_preflight() == {"pass": ["mutmut"], "fail": [], "warn": []} recorder.assert_not_called() - monkeypatch.setattr(gates(), "languages", ("rb",)) + monkeypatch.setitem(gates().settings, "languages", ("rb",)) stage(git_repo, "app.rb", "def foo; end\n") assert gates().run_preflight() == {"pass": ["mutmut"], "fail": [], "warn": []} recorder.assert_not_called() - monkeypatch.setattr(gates(), "languages", ("py",)) + monkeypatch.setitem(gates().settings, "languages", ("py",)) stage(git_repo, "src/gone.py", "value = 1\n") gate.run_git(["commit", "-q", "-m", "add gone"], git_repo) gate.run_git(["rm", "-q", "src/gone.py"], git_repo) diff --git a/harness/tests/test_ralph_ps1.py b/harness/tests/test_ralph_ps1.py index aa4eedf..f542be4 100644 --- a/harness/tests/test_ralph_ps1.py +++ b/harness/tests/test_ralph_ps1.py @@ -72,7 +72,7 @@ def test_two_iterations_pass_prompt_marker_and_environment(tmp_path: Path) -> No def test_explicit_one_iteration_completes(tmp_path: Path) -> None: - """An explicit one-iteration loop runs the worker once.""" + """Direct completion receipts report the exact number of workers run.""" worker = write_worker(tmp_path, "import sys\nsys.stdin.read()\n") result = run_ralph(tmp_path, ["1", "1", sys.executable, str(worker)]) @@ -82,6 +82,20 @@ def test_explicit_one_iteration_completes(tmp_path: Path) -> None: assert (events[0]["iteration"], events[-1]["completed"]) == (1, 1) assert all(event["max_minutes"] == 1 for event in events) + no_iterations = run_ralph(tmp_path, ["0", "1", sys.executable, str(worker)]) + no_iteration_events = [json.loads(line) for line in no_iterations.stdout.splitlines()] + assert no_iterations.returncode == 0 + assert len(no_iteration_events) == 1 + assert ( + no_iteration_events[0]["type"], + no_iteration_events[0]["completed"], + no_iteration_events[0]["max_minutes"], + ) == ( + "ralph", + 0, + 1, + ) + def test_worker_arguments_are_preserved_exactly(tmp_path: Path) -> None: """Literal -p, model flags, and spaced values reach the worker as distinct unchanged arguments.""" @@ -102,11 +116,15 @@ def test_worker_arguments_are_preserved_exactly(tmp_path: Path) -> None: def test_command_without_additional_arguments_runs(tmp_path: Path) -> None: - """A worker executable with no argv tail does not trigger an invalid PowerShell array slice.""" + """The worker is required, while a worker without its own arguments runs unchanged.""" result = run_ralph(tmp_path, ["1", "1", "sort.exe"]) assert result.returncode == 0 + missing_worker = run_ralph(tmp_path, ["1", "1"]) + assert missing_worker.returncode == 2 + assert "" in missing_worker.stderr + def test_nonzero_worker_exit_propagates_and_stops(tmp_path: Path) -> None: """The first worker failure reaches the caller and prevents later iterations.""" diff --git a/mutation-score.json b/mutation-score.json index b1de836..4fee1ba 100644 --- a/mutation-score.json +++ b/mutation-score.json @@ -1,6 +1,6 @@ { "schemaVersion": 1, "label": "mutation", - "message": "84.2%", + "message": "83.6%", "color": "#177445" } diff --git a/pyproject.toml b/pyproject.toml index 2f2f2d7..6311e58 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "harness" requires-python = ">=3.10" -version = "0.2.0" +version = "0.1.0" description = "Gated, harnessed, autonomy loop with quality checks." authors = [{ name = "rxdt" }] readme = "README.md" @@ -83,8 +83,10 @@ include = [ # Humans own this file (it is agent-forbidden below). # ============================================================================== [tool.harness] +[tool.harness.settings] +behavior = "fail" # "fail" or "warn": change to "warn" if your project has too many blocking failures to start languages = ["py"] -error_diff_lines = 400 # ~90th percentile of pull requests, 200 LOC generally ok +error_diff_lines = 500 # 400 is ~90th percentile of pull requests, 200 LOC generally ok # argv preset per agent for `harness run ` [tool.harness.agents] @@ -142,7 +144,7 @@ complexity = ["complexipy", ".", "--suggest-refactors"] pylint = ["pylint", "."] [tool.harness.gate] -audit = ["pip-audit", ".", "--strict"] +audit = ["pip-audit", "--strict"] security = [ "semgrep", "scan", "--error", # exit nonzero on findings so the gate blocks the commit, not just reports @@ -155,28 +157,52 @@ types = ["pyright", "--outputjson"] test = ["pytest", "-p", "no:cacheprovider", "-n", "auto", "--cov", "--cov-report=term-missing", "--cov-fail-under=100", "--hypothesis-show-statistics", "--durations=5"] [tool.harness.FORBIDDEN] -DIRS = ["harness/", ".githooks/", ".github/", ".git/", "preferences/", "tests/preferences/"] +DIRS = ["harness/", ".githooks/", ".github/", ".git/", "preferences/", "tests/preferences/", "mutation", "tests/mutation/"] FILES = [ "agents.md", "docs/prompt.md", "docs/plan.md", # delete/comment line if you want agents to manage the core plan - # tooling/config files that would weaken checks in this file - "pytest.ini", + # tooling/config files that would weaken checks: "tox.ini", "setup.cfg", + "pytest.ini", + ".pytest.ini", + "pytest.toml", + ".pytest.toml", ".coveragerc", + ".coveragerc.toml", "ruff.toml", ".ruff.toml", - ".semgrepignore", - "pyrightconfig.json", + ".flake8", + "pylintrc", ".pylintrc", + "pylintrc.toml", + ".pylintrc.toml", + "pyrightconfig.json", + "mypy.ini", + ".mypy.ini", + "ty.toml", + "pyrefly.toml", + ".pyrefly.toml", + "complexipy.toml", + ".complexipy.toml", + "radon.cfg", + ".semgrepignore", + "semgrep.yml", + "semgrep.yaml", + ".semgrep.yml", + ".semgrep.yaml", + "semgrep.config.yml", + "semgrep.config.yaml", + ".bandit", + "sonar-project.properties", + ".snyk", ".gitmodules", ".gitattributes", "pyproject.toml", # agents will claim inability to add dependencies with this forbidden "requirements.txt", "uv.lock", # if *.lock remains, agents will leave it uncommitted if file changes - "poetry.lock", - "mypy.ini" + "poetry.lock" ] PATTERNS = [ "# noqa", @@ -437,7 +463,7 @@ banned-api."re".msg = "Regex is slow. Use Python built-in string manipulation op "D104", # Test package boundaries do not need package docstrings, undocumented-public-package "DOC201", # Tests explain behavior through assertions, docstring-missing-returns "S101", # Allow 'assert' statements (needed for pytest) - "PLR0913", # Tests may need larger setup functions, aka max-positional-args + "PLR0913", # Tests may need larger setup functions, aka too-many-arguments "PLR0915", # Tests may need larger setup functions, aka too-many-statements "PLR0917", # Tests may need larger setup function definitions, aka max-positional-args ] diff --git a/tests/mutation/test_check_mutmut.py b/tests/mutation/test_check_mutmut.py index 12e5b2c..4a0e629 100644 --- a/tests/mutation/test_check_mutmut.py +++ b/tests/mutation/test_check_mutmut.py @@ -31,10 +31,11 @@ def test_module_output_from_mutation_directory_is_exact( runpy.run_path(str(checker), run_name="__main__") assert exc_info.value.code == 1 - assert " ".join(unstyle(capsys.readouterr().out).split()) == ( - f"{'─' * 27} MUTMUT MUTATION RESULTS {'─' * 28} " - "killed 1 survived 1 total 2 skipped 0 timeout 0 Mutation Score: 50.0" - ) + output = " ".join(unstyle(capsys.readouterr().out).split()) + leading_rule, title, results = output.partition("MUTMUT MUTATION RESULTS") + assert title == "MUTMUT MUTATION RESULTS" + assert not leading_rule.strip("─ ") + assert results.strip("─ ") == ("killed 1 survived 1 total 2 skipped 0 timeout 0 Mutation Score: 50.0") monkeypatch.chdir(module_dir) monkeypatch.setattr(check_mutmut, "__file__", str(module_dir / "check_mutmut.py"))