test(compat): scaffold cross-implementation compat-test harness#1249
Merged
mergify[bot] merged 1 commit intomainfrom Apr 21, 2026
Conversation
Member
Author
|
This pull request is part of a Mergify stack:
|
This was referenced Apr 20, 2026
Contributor
Merge ProtectionsYour pull request matches the following merge protections and will not be merged until they are valid. 🟢 🤖 Continuous IntegrationWonderful, this rule succeeded.
🟢 👀 Review RequirementsWonderful, this rule succeeded.
🟢 Enforce conventional commitWonderful, this rule succeeded.Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 🔎 ReviewsWonderful, this rule succeeded.
🟢 📕 PR descriptionWonderful, this rule succeeded.
|
jd
force-pushed
the
devs/jd/worktree-rust-port/scaffold-cross-impl-compat-test-harness--011cf832
branch
from
8858672 to
260a37d
Compare
jd
force-pushed
the
devs/jd/worktree-rust-port/formalize-json-stdout-stderr-discipline--5a37d5c1
branch
from
a2e8e33 to
e03dc48
Compare
Member
Author
Revision history
|
jd
force-pushed
the
devs/jd/worktree-rust-port/scaffold-cross-impl-compat-test-harness--011cf832
branch
from
260a37d to
b5f2dfe
Compare
jd
force-pushed
the
devs/jd/worktree-rust-port/scaffold-cross-impl-compat-test-harness--011cf832
branch
from
b5f2dfe to
a651c13
Compare
Base automatically changed from
devs/jd/worktree-rust-port/formalize-json-stdout-stderr-discipline--5a37d5c1
to
main
April 21, 2026 07:59
Contributor
|
@jd this pull request is now in conflict 😩 |
jd
force-pushed
the
devs/jd/worktree-rust-port/scaffold-cross-impl-compat-test-harness--011cf832
branch
from
a651c13 to
6703ebf
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds an initial “compat-tests” harness to lock in the Python CLI’s observable behavior (exit codes + selected stdout substrings) as groundwork for a future Rust implementation parity runner.
Changes:
- Add
compat-tests/pytest runner that discovers case directories and executespython -m mergify_cli. - Add seed compat fixtures for basic, zero-state CLI behaviors (
--help,--version, unknown command,stack --help). - Wire the harness into developer tooling (
poe compat-test) and CI, and addmergify_cli/__main__.pyto supportpython -m mergify_cli.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Adds Ruff per-file ignores for the new compat test tree. |
| poe.toml | Adds a compat-test Poe task to run the compat test suite. |
| mergify_cli/main.py | Enables module execution via python -m mergify_cli. |
| compat-tests/test_compat.py | Implements the compat case discovery + subprocess runner and assertions. |
| compat-tests/README.md | Documents how to add/run compat cases and the purpose of the harness. |
| compat-tests/cases/** | Adds initial compat test fixtures (args/expected exit/stdout substring). |
| .github/workflows/ci.yaml | Adds a dedicated CI job to run compat tests on every PR and gates CI on it. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Adds the infrastructure that Phase 1+ of the Rust port will use to verify the two implementations produce identical observable behavior. During Phase 0 it runs Python against itself, which already catches output drift and locks in the exit-code contract. ## What's here - ``compat-tests/`` at the repo root, kept separate from ``mergify_cli/tests/`` so it can grow into a cross-implementation runner without polluting the unit-test tree. - ``compat-tests/test_compat.py`` — pytest-discovered runner that iterates ``compat-tests/cases/<case-name>/`` directories and invokes ``python -m mergify_cli`` with the fixture's args. Asserts the exit code matches ``expected_exit`` and the optional ``stdout_contains`` substring appears in stdout. - Four seed fixtures exercising zero-state paths: ``--help``, ``--version``, unknown subcommand (exit 2), and ``stack --help``. - ``mergify_cli/__main__.py`` so ``python -m mergify_cli`` works. This is what the Rust port's Python shim will invoke once the port lands. - ``poe compat-test`` task and a dedicated CI job so the harness runs on every PR. ## What this doesn't do yet Fixtures needing mocked HTTP or prepared git state aren't included — they need additional harness support (env/stdin wiring, maybe a fixtures-as-conftest layer). Those land incrementally as the Rust port progresses; each new ported command gets at least one compat fixture covering its observable contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Change-Id: I011cf8321173c94fb46e8e6421b66534dc862813
jd
force-pushed
the
devs/jd/worktree-rust-port/scaffold-cross-impl-compat-test-harness--011cf832
branch
from
6703ebf to
6d6d0c0
Compare
This was referenced Apr 21, 2026
JulianMaurin
approved these changes
Apr 21, 2026
kozlek
approved these changes
Apr 21, 2026
Contributor
Merge Queue Status
This pull request spent 10 seconds in the queue, including 2 seconds running CI. Required conditions to merge
|
mergify
Bot
deleted the
devs/jd/worktree-rust-port/scaffold-cross-impl-compat-test-harness--011cf832
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the infrastructure that Phase 1+ of the Rust port will use to
verify the two implementations produce identical observable
behavior. During Phase 0 it runs Python against itself, which
already catches output drift and locks in the exit-code contract.
What's here
compat-tests/at the repo root, kept separate frommergify_cli/tests/so it can grow into a cross-implementationrunner without polluting the unit-test tree.
compat-tests/test_compat.py— pytest-discovered runner thatiterates
compat-tests/cases/<case-name>/directories andinvokes
python -m mergify_cliwith the fixture's args. Assertsthe exit code matches
expected_exitand the optionalstdout_containssubstring appears in stdout.--help,--version, unknown subcommand (exit 2), andstack --help.mergify_cli/__main__.pysopython -m mergify_cliworks.This is what the Rust port's Python shim will invoke once the
port lands.
poe compat-testtask and a dedicated CI job so the harnessruns on every PR.
What this doesn't do yet
Fixtures needing mocked HTTP or prepared git state aren't included
— they need additional harness support (env/stdin wiring, maybe a
fixtures-as-conftest layer). Those land incrementally as the Rust
port progresses; each new ported command gets at least one compat
fixture covering its observable contract.
Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com