Harden permission-reject E2E tests across all SDKs (#1194)#1317
Conversation
The existing "reject permission" E2E test in every SDK asserted only that the target file was unchanged after a model-driven edit attempt. That is a false-positive-prone check: it passes even when the agent freezes, when the permission decision is silently dropped, or when the wrong discriminator is sent. Specifically, it would not have caught the .NET `PermissionDecision` empty-JSON regression reported in #1194 (since fixed in codegen). Strengthen each SDK's reject test to additionally assert that the CLI emits a `tool.execution_complete` event whose error message contains "user rejected" (case-insensitive). The CLI emits a kind-specific message for the reject decision ("The user rejected this tool call.") vs. the distinct message for user-not-available, so this asserts that the specific `reject` discriminator round-tripped end-to-end - exactly the property that was broken in #1194. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Test-only hardening across all SDK languages to ensure the "reject permission" E2E tests catch regressions like #1194, where a permission decision could be silently dropped due to broken discriminator serialization. Each test now asserts that a tool.execution_complete event surfaces a "user rejected" error message, in addition to the existing assertion that the target file was unmodified.
Changes:
- Add event subscription in each SDK's reject test to detect
tool.execution_completeevents with a user-rejection error. - Add a Rust helper
is_user_rejected_tool_completionmirroring the existingis_permission_denied_tool_completion. - Comment each change with a reference to issue #1194.
Show a summary per file
| File | Description |
|---|---|
| dotnet/test/E2E/PermissionE2ETests.cs | Subscribe to events and assert user-rejected tool completion. |
| go/internal/e2e/permissions_e2e_test.go | Add mutex-guarded event listener asserting rejection error. |
| nodejs/test/e2e/permissions.e2e.test.ts | Add event listener and expectation for rejected tool call. |
| python/e2e/test_permissions_e2e.py | Add event listener via pattern match asserting rejection error. |
| rust/tests/e2e/permissions.rs | Subscribe to events and add is_user_rejected_tool_completion helper. |
Copilot's findings
- Files reviewed: 5/5 changed files
- Comments generated: 0
Cross-SDK Consistency Review ✅This PR updates all five SDK implementations (Node.js, Python, Go, .NET, Rust) with the same regression check, and the approach is consistent across the board. A few notes: Consistent across all SDKs:
One pre-existing asymmetry (out of scope, noted for awareness): The Python msg = (
error
if isinstance(error, str)
else (getattr(error, "message", None) if error is not None else None)
)This dual-type handling ( Overall, this PR is a high-quality, well-coordinated consistency fix. The new assertion pattern is a meaningful improvement over the previous file-unchanged-only check, and it's applied symmetrically across all languages.
|
Issue #1194 reported that the .NET SDK serialized
PermissionDecisionas empty JSON because it instantiated the polymorphic base type. The user-visible bug was already fixed at the codegen layer onmain(the[JsonIgnore]was removed from the generated base discriminator property), so no runtime code change is needed. However, none of the existing per-SDK E2E permission tests would have caught the regression, which is what this PR addresses.Why the existing tests missed it
Every SDK had a "reject permission" E2E test that only asserted the target file was unchanged after a model-driven edit attempt. That is a false-positive-prone check: it passes even when the agent freezes, when the permission decision is silently dropped, or when the wrong discriminator is sent on the wire. It would not have caught the .NET serialization bug.
What changed
Each SDK's reject test now additionally asserts that the CLI emits a
tool.execution_completeevent whose error message contains"user rejected"(case-insensitive). The CLI emits a kind-specific message for therejectdecision ("The user rejected this tool call.") that is distinct from the message used foruser-not-available, so this asserts the specificrejectdiscriminator round-tripped end-to-end - exactly the property that was broken in #1194.The pattern (subscribe to events before sending the prompt, then wait for the matching
tool.execution_complete) mirrors what each SDK's "user not available" test already does, so the new code reuses an established idiom in each language.Files touched (test-only):
dotnet/test/E2E/PermissionE2ETests.csnodejs/test/e2e/permissions.e2e.test.tspython/e2e/test_permissions_e2e.pygo/internal/e2e/permissions_e2e_test.gorust/tests/e2e/permissions.rs(also adds anis_user_rejected_tool_completionhelper next to the existingis_permission_denied_tool_completionhelper)Each test carries a comment referencing #1194 so future readers see why the event-level assertion exists.
Validation
Ran the targeted reject test in each SDK locally on Windows; all 5 pass:
dotnet test --filter "Should_Deny_Permission_When_Handler_Returns_Denied"npx vitest run test/e2e/permissions.e2e.test.ts -t "should deny permission when handler returns denied"python -m pytest e2e/test_permissions_e2e.py::TestPermissions::test_should_deny_permission_when_handler_returns_deniedgo test ./internal/e2e -run "TestPermissionsE2E/deny_permission$"cargo test --features test-support --test e2e should_deny_permission_when_handler_returns_deniedOut of scope
The .NET/Python/Go/Rust public
PermissionRequestResultsurface still only exposes 4 basic kinds (Approved/Rejected/UserNotAvailable/NoResult); only Node exposes the full polymorphic union with extra fields (session-scope approval rules, location keys, permanent-approval domains, reject feedback). That's a separate enhancement and not addressed here.