Skip to content

test(executor-e2e): cover create-github-issue and set-github-issue-type - #1805

Merged
jamesadevine merged 3 commits into
mainfrom
test/executor-e2e-github-issue-scenarios
Aug 4, 2026
Merged

test(executor-e2e): cover create-github-issue and set-github-issue-type#1805
jamesadevine merged 3 commits into
mainfrom
test/executor-e2e-github-issue-scenarios

Conversation

@jamesadevine

@jamesadevine jamesadevine commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #1804. Base is fix/stage3-tool-config-staged-key; review that one first. The create-github-issue-label-denied scenario depends on the fix in #1804 to pass. ## What create-github-issue and set-github-issue-type shipped with zero runtime coverage — their only proof was a wiremock unit test. The smoke-suite rework removed smoke-failure-reporter, which was the last thing exercising create-github-issue end to end, so coverage went from thin to none. This also reverses the stale exclusion line in tests/executor-e2e/README.md ("Excluded … the GitHub-only create-github-issue"), which predates these becoming first-class public safe outputs. ## Scenarios | id | Tool | What it proves | | --- | --- | --- | | create-github-issue | create-github-issue | title-prefix applied; body + <!-- ado-aw --> footer; static and allowed agent labels merged | | create-github-issue-label-denied | create-github-issue | default-deny allowed-labels rejection via expectedFailure | | set-github-issue-type | set-github-issue-type | named type set on an existing issue | | set-github-issue-type-clear | set-github-issue-type | the documented issue_type: "" clear | | create-github-issue-temporary-id-handoff | set-github-issue-type (+ prior create-github-issue) | the same-run temporary_id handoff | ## Harness change worth reviewing: priorEntries The temporary-id registry (ExecutionContext::resolved_github_issues) is an in-process Arc<Mutex<HashMap<_>>> that is never persisted, so the handoff is only observable inside a single ado-aw execute process. Every existing scenario runs exactly one entry per invocation. New optional Scenario.priorEntries stages extra NDJSON lines ahead of the primary entry in the same run — which is also precisely how production works (a SafeOutputs job runs one execute over the whole safe_outputs.ndjson, in file order). Supporting changes: renderSourceMarkdown emits one safe-outputs key per tool; assert() gains a records argument; the runner reports a failed prior entry as its own execute-phase failure so a broken prerequisite can't masquerade as an assertion failure. ## No fourth GitHub client GitHub REST plumbing is moved out of the harness's own failure reporter into a shared github-client.ts. The repo has three GitHub clients and they are legitimately separate (shipped Rust executor / shipped App-token bundle / test harness); this keeps the count at three rather than letting scenarios add a fourth ad-hoc fetch wrapper. ## Close, don't delete GitHub has no delete-issue endpoint, so these are the only scenarios that cannot tear down completely — cleanup() closes as not_planned. Mitigations: titles embed the standard ado-aw-det-<buildId>-<id> marker, and cleanup does not depend solely on state set in assert() — when the executor filed an issue but the record came back non-succeeded (so assert() never ran), cleanup falls back to an exact-title search on that marker. Documented in code and in the README. ## Skips, not failures Missing preconditions skip: no token, no scratch repo, a token that authenticates but can't write issues (with the harness's auth diagnosis attached), or no org issue types. Issue types are an organisation-level construct with no user-account equivalent, so the named-type scenarios will skip on a user-owned scratch repo such as jamesadevine/ado-aw-issues. The handoff stays runnable there by falling back to the documented clear operation. There is deliberately no default repo: when neither EXECUTOR_E2E_SCENARIO_ISSUE_REPO nor EXECUTOR_E2E_ISSUE_REPO is set, the scenarios skip rather than filing scratch issues onto githubnext/ado-aw. ## Mutation-checked A test that passes either way is worse than none, so the handoff assertion was deliberately broken to confirm it goes red: - mismatched result.number vs the number create-github-issue actually filed - mismatched result.target_repo - a wrong echoed temporary_id - a fabricated result GitHub doesn't corroborate - inverted NDJSON ordering (verified against the real runner/execute-cli path) create-github-issue-label-denied matches only labels not in allowed-labels. The alternative message (no allowed-labels configured) is exactly what the executor emits when the config was dropped — accepting both would let it pass either way, which is the failure mode #1804 describes. ## Found a real bug This coverage immediately surfaced the Stage 3 config-drop defect fixed in #1804. That is the point of the exercise, and it is recorded in the README as the motivating example. ## Split-approval: already guarded

I initially flagged this as an unguarded gap and was wrong. require-approval on only one of the two tools would split Stage 3 into two execute processes and break the handoff — but validate_github_issue_outputs_config in src/compile/common.rs already rejects that configuration, keyed on the effective setting so a section-level default and a per-tool override are both handled.

Verified by compiling all three shapes (either direction, plus a global default with a per-tool opt-out); each is refused. The README note is corrected accordingly.

Testing - npm run typecheck ✅ - npx vitest run src/executor-e2e — 67 passed (was 29) ✅ - npm run build:executor-e2e ✅ - Full npx vitest run — 890 passed, no regressions ✅ - Multi-key front matter, NDJSON ordering, record names and the label-rejection message were all verified against the real ado-aw execute binary, not just mocks. Live proof comes from ADO definition 2550, whose PR path filter already covers scripts/ado-script/src/executor-e2e/** and tests/executor-e2e/**.

jamesadevine and others added 2 commits August 3, 2026 21:38
`ExecutionContext::get_tool_config` stripped the compiler-injected
`require-approval` key but not `staged`, which Stage 3 also injects into
every tool config (`main.rs` for the `--source` path,
`compile/custom_tools.rs` for the `--resolved-config` path production
actually uses).

`CreateGithubIssueConfig` and `SetGithubIssueTypeConfig` are the only
safe-output configs declared `deny_unknown_fields`, so deserialization
failed — and the error was swallowed by `.ok().unwrap_or_default()`,
silently replacing the operator config with `Default::default()`.

Observable effects:

* `target-repo` ignored, so Stage 3 failed outright on non-GitHub-backed
  ADO builds with "target-repo is required when the Azure DevOps pipeline
  source is not GitHub"
* `title-prefix` never applied; static `labels`/`assignees` dropped
* `allowed-labels` emptied, so default-deny rejected *every* agent label
* `require-temporary-id` never enforced; `max` budget override ignored
* `set-github-issue-type.allowed` never gated anything — an empty list is
  default-allow, so this one failed OPEN

Strip `staged` alongside `require-approval`, and log a warning instead of
silently defaulting so a future config-shape mismatch is visible rather
than presenting as a mysterious runtime failure.

The existing wiremock tests could not catch this: they build an
`ExecutionContext` directly with a `tool_configs` map that has no
`staged` key, i.e. a shape that never occurs in production. The added
regression tests assert an operator config survives both injected keys.

Found while adding deterministic executor-e2e coverage for these tools.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cefea537-5177-4dfd-8849-bcc2caff2845
`create-github-issue` and `set-github-issue-type` shipped with zero
runtime coverage: their only proof was a wiremock unit test, and the
smoke-suite rework removed `smoke-failure-reporter`, the last thing
exercising `create-github-issue` end to end.

Adds five deterministic Stage 3 scenarios:

* `create-github-issue` — title-prefix application, body + `<!-- ado-aw -->`
  footer, and static/agent label merging
* `create-github-issue-label-denied` — default-deny `allowed-labels`
  rejection via `expectedFailure`
* `set-github-issue-type` — named type on an existing issue
* `set-github-issue-type-clear` — the documented `issue_type: ""` clear
* `create-github-issue-temporary-id-handoff` — the same-run `temporary_id`
  handoff, the highest-value case and the one with the weakest prior proof

Harness changes:

* new optional `Scenario.priorEntries` hook, so one `ado-aw execute` can
  process multiple NDJSON lines. The temporary-id registry
  (`ExecutionContext::resolved_github_issues`) is an in-process
  `Arc<Mutex<HashMap<_>>>`, so the handoff is only observable inside a
  single executor process — which is also exactly how production runs it.
  The runner reports a failed prior entry as its own execute-phase
  failure so a broken prerequisite can't look like an assertion failure.
* `renderSourceMarkdown` now emits one `safe-outputs` key per tool
* `assert()` gains a `records` argument exposing every executed record
* GitHub REST plumbing extracted from the harness's failure reporter into
  a shared `github-client.ts`, so scenarios reuse it instead of adding a
  fourth GitHub client to the repo

Cleanup closes rather than deletes, because GitHub has no delete-issue
API. Titles embed the `ado-aw-det-*` marker, and cleanup falls back to a
title search when a run fails before `assert()` populated the number, so
an issue created by a non-succeeded record is still closed.

Every missing precondition skips rather than fails: no token, no scratch
repo, a token that can't write issues, or no org issue types. Issue types
are organisation-level with no user-account equivalent, so the named-type
scenarios skip on a user-owned scratch repo; the handoff stays runnable by
falling back to the clear operation.

The handoff assertion is mutation-checked — mismatched issue number,
mismatched repository, a wrong echoed temporary id, and a fabricated
result that GitHub does not corroborate all fail the test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cefea537-5177-4dfd-8849-bcc2caff2845
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

An earlier draft claimed there was no compile-time guard against putting
`require-approval` on only one of the two GitHub issue tools. That is
wrong: `validate_github_issue_outputs_config` in `src/compile/common.rs`
rejects it, keyed on the *effective* setting so a section-level default
and a per-tool override are both handled.

Verified by compiling all three shapes (either direction, plus a global
default with a per-tool opt-out) — each is refused.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: cefea537-5177-4dfd-8849-bcc2caff2845
Base automatically changed from fix/stage3-tool-config-staged-key to main August 3, 2026 21:04
@jamesadevine

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

🚀 review-typescript has started processing this pull request comment

@jamesadevine
jamesadevine merged commit 8a7824b into main Aug 4, 2026
64 of 69 checks passed
@jamesadevine
jamesadevine deleted the test/executor-e2e-github-issue-scenarios branch August 4, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant