test: turn the #135 mutation audit into a repeatable test-adequacy gate by DevilTea · Pull Request #138 · DevilTea/valchecker

DevilTea · 2026-07-31T08:54:47Z

Draft. Opened to preserve and expose the work, not as a merge candidate — the owner intends to
rebuild the verification mechanism and rewrite a large number of steps against the new step
unit, so the parts of this that survive that decision are what should eventually land.

Closes #135. Its remainder is consolidated into a single follow-up issue rather than left
spread across #134/#135/#136.

What is finished

#135's concrete remainder, each item with executable evidence rather than an argument:

The two cross-step tests that did not discriminate what they claimed. intersection's
findConflictingLeftBranch falls back to rightBranch - 1, and both tests expected exactly
that value, so each passed under three separate attribution mutations. The fixtures now put
two path-less branches between the contributor and the conflict, and the accessor test is
renamed to what it actually distinguishes — locating the source walks own property
descriptors and declines an accessor-valued segment rather than invoking it, so attribution
falls back. It is now the only test that kills a mutation of the fallback value.
record's missing family-level coverage. It appeared in no structural.*.test.ts at
all. Internal fatality under collectAllIssues (collection on is what separates it from
ordinary short-circuiting), first-vs-collect-all over both the open entry loop and the finite
member loop, and synchronous children not inspected as thenables.
intersection.ts's unreachable merge tail, collapsed to one different_values conflict
with the invariant written down. 2058 tests pass with the branches replaced by a throw, and
ten adversarial pairs all report their conflict from discoverCompatibility. The guard is
kept rather than deleted so a future change weakening the invariant degrades into a reported
conflict instead of silently discarding a branch's value.
map/set live re-yield, decided by the owner as unsupported and documented. The divergence
was reproduced first: an identity child absorbs the re-yielded entry into its buffer while a
transforming child reports a duplicate.

#136's leftovers: the compare job's before/after SHA table (the shas now travel as measure
job outputs), and benchmarks/README.md rewritten around cells with every number recomputed
from the catalog — 245 cells over six groups, warm/success 124 of them, an eighteen-cell
canary — and the inputs reduced to the ones the workflow accepts.

The mutation system, which is what #135 asked for beyond "finish adding tests":

StrykerJS 9.6.1 + Vitest. pnpm mutation (whole tree, ~26 min), pnpm mutation:changed
(the diff's files plus the production files beside changed tests). It replaces an
edit-source/run/restore loop, which is the point: Stryker mutates a sandbox copy and never
touches the working tree, where the old shape once let a live mutation be committed by a
broad git add.
The contract is not a score. thresholds.break is null. The rule is that new and
unclassified survivors are zero: killed by a test, or recorded with a classification, the
invariant behind it, and how that was checked. Triage has six outcomes, four recordable;
TEST_GAP is closed rather than written down, and UNKNOWN is recordable but fails the gate,
so --write can never turn a red gate green.
Two homes, both checked for rot. A structural pattern whose equivalence follows from one
invariant is suppressed beside the code; anything isolated goes in mutation-survivors.json.
A killed ledger entry fails, a missing file fails, and a directive that no longer ignores
anything fails, so an exemption cannot outlive the construct it argued about.
CI: scoped gate on pull requests (skipped on drafts, so this PR does not run it), whole tree
weekly, reports retained as artifacts.

core.ts triage, measured: 74 surviving mutants down to 1 unexplained + 2 unconfirmed,
375 killed, 28 in-source directives all registering and none stale. Eleven test gaps closed,
each proved by watching the assertion fail under the mutation and pass without it — both async
continuation loops restarting their promise chain at the wrong index; all four tiers of the
deferred message placeholder, which ~execute hands to plugin authors; the scopes a child
collects before tuple's rest region replaces its path; the shared prototype's descriptor
flags; the plugin-declared default operation mode in all three registrars; the guard that keeps
a schema's mode climbing; a plugin declared async; the TypeError text behind an empty
failure collection; createIssue omitting the context key rather than assigning undefined;
and resolving a core:message_exception through a handler that did not throw for it.

What the exercise found about its own tooling

Three corrections, all written into .claude/skills/valchecker-dev/references/mutation.md
rather than left to be rediscovered:

NoCoverage is a claim, not a measurement. Stryker never executes those mutants. The
empty-capabilities array in createValchecker came back NoCoverage while a test in
core.test.ts kills it when applied by hand. They are reported as unconfirmed, with the
command that settles them.
A // Stryker disable next-line only attaches where the comment is a leading comment of
the mutated node. Inside a ternary, or before an else if, it silently does nothing — and
it covers one line, so a four-line expression needs block form. Three of mine were inert; the
suppression rot check is what caught them, and two more were caught by the reason rule for
saying "see above" instead of an argument.
An output digest is evidence about the inputs in it. It reported the
?? RUNTIME_OPERATION_MODE_MAYBE_ASYNC → && mutant as changing nothing, true of everything
it measured and false of the program, because no built-in step declares async.

One assertion was written and then deleted rather than kept: an initial-schema operation-mode
check that passed under its own mutation because it read the outer chain's mode. A test that
looks like coverage without discriminating anything is the defect this work exists to remove,
so ~operationMode in createInitialSchema is the one unexplained survivor left in core.

What remains, measured

Scope	Detected	Survived	NoCoverage
`core.ts`	378	7 (6 in the ledger, 1 unexplained)	2
`intersection`, `map`, `tuple`, `record`, `set`	1437	224	19
the other 120 files	2861	345	35

626 undetected mutants remain, and 129 of them are proven not to be behavioural changes
at all: operation-mode string literals that tsc rejects
(Argument of type '""' is not assignable to parameter of type 'OperationMode | undefined'),
admitted only because stryker.config.mjs sets disableTypeChecks. Adopting
@stryker-mutator/typescript-checker would reclassify them as CompileError, which is
detected — one config change instead of 129 ledger entries. Not done here, because it is a
dependency addition and the harness may be rebuilt anyway.

Also not done from #135: the single length/size read assertions for eight steps, and the
format-validator grammars (isJwt 19 survivors, isEmoji 14, isIp 13). The default-message
gaps it listed as 22 appear largely closed — only five sentence-shaped literals survive the
whole sweep. isEmoji's ZWJ sentence and the map/set metadata throws stay deliberately
deferred, which #135 judged correctly.

Supply-chain note

pnpm-workspace.yaml excludes semver@6.3.1 from the trust policy. It is reached only through
@babel/core, which the Stryker instrumenter pulls in, and pnpm compares trust evidence by
publish date across the whole package — so a 6.x patch published after a provenance-signed 7.x
release reads as a downgrade. The 6.x line predates npm provenance; nothing was withdrawn.
Pinned to the exact version so a genuinely new untrusted release still fails.

Verification

pnpm verify exits 0: 10 of 10 quality gates, 149 test files / 2372 tests, per-file coverage
policy, api-surface match, documentation examples compiled, docs site built.

Not run: pnpm typeperf, and the bundle-size and performance-impact workflows. The
Mutation workflow's pull-request job is skipped on drafts by design, and would be red here
anyway — 626 survivors are not classified, which is the honest state and the reason this is a
draft.

🤖 Generated with Claude Code

…criminate Three findings from #135's audit, each closed with evidence rather than an argument. Two cross-step tests claimed to identify a non-adjacent conflict source and did not. `intersection`'s `findConflictingLeftBranch` falls back to `rightBranch - 1` when no earlier branch carries the conflicting path, and both tests expected exactly that value, so each passed under three separate attribution mutations. The fixtures now separate the contributing branch from the conflicting one by two branches that do not carry the path, and the accessor test is renamed to what it actually distinguishes: locating the source walks own property descriptors and declines an accessor-valued segment rather than invoking it, so attribution falls back. It is now the only test that kills a mutation of the fallback value. `record` had no family-level coverage at all. It now has the contracts its siblings already had there, for both key domains: internal fatality under `collectAllIssues` (collection on is what separates it from ordinary short-circuiting), first-vs-collect-all over the open entry loop and the finite member loop, and synchronous children that are not inspected as thenables. `intersection`'s `mergeValues` no longer duplicates the conflict classification that `discoverCompatibility` performs. That pass walks the same pairs to completion before any merge and rejects everything that is neither `Object.is`-equal nor two plain objects with one prototype, so the branches below the `Object.is` check could not execute: 2058 tests pass with them replaced by a throw, and ten adversarial pairs all report their conflict from `discoverCompatibility`. They collapse to a single `different_values` conflict with the invariant written down; the guard is kept rather than deleted so a future change weakening the invariant degrades into a reported conflict instead of silently discarding the right branch's value. `map()` and `set()` document that mutating the input during validation is unsupported. The traversal is live, but the identity buffer and the materialized path disagree about a re-yielded entry, measured both ways. Agreeing would cost an allocation on the success path or a full copy per execution, so the input is documented as needing to hold still, with the snapshot named as the upgrade path. Every new assertion was proved by watching it fail under the mutation and pass without it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two items #136 left behind. `benchmarks/README.md` still called the cross-library scenarios the impact gate's unit. Phase 2 replaced them with each step's own `stepBench()` cells, so the numbers the page quoted were stale as well as the prose: they are recomputed from the catalog — 245 cells over six groups, `warm/success` 124 of them, an eighteen-cell canary, one cell-run at 2.33s from run 30547023911. The dispatch inputs are reduced to the ones the workflow accepts: `adapters` is gone because the gate measures one library against itself, and `fail_on_regression` because the two-stage screen-and-confirm design replaced the escape hatch it existed for. A cell's step is now the directory it lives in, so the paragraph about trusting `steps` declarations applies to the scenarios and to Performance Comparison, not to this selection. The scoping evidence measured on the scenario suite is kept and labelled as such, since it rests on per-cell process isolation, which the change to cells did not touch. The compare job's summary carries its identity table again. It was lost when measurement and comparison were split into separate jobs, because `compare` resolves no parameters of its own; the refs and the shas they resolved to now travel as `measure` job outputs, so a reader never has to open the measuring job to learn which two builds produced the numbers below. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Closes the part of #135 that turns a successful one-off audit into a repeatable system. The audit established that mutation answers a question coverage cannot: reading missed defects it found, and several tests that looked meaningful did not discriminate the behaviour their titles claimed. StrykerJS drives the Vitest suite. `pnpm mutation` mutates every production file (about an hour, the weekly sweep); `pnpm mutation:changed` mutates the files in the diff plus the production files beside any changed test (the pull-request scope). It replaces an edit-source, run-tests, restore-source loop, and that is the point rather than an implementation detail: Stryker mutates a sandbox copy and never touches the working tree, where the old shape let a live mutation be captured into a commit by a broad `git add` over a shared checkout — producing a "defect" and a "fix" for code that was never broken. The contract is deliberately not a mutation score. `thresholds.break` is null and no percentage decides anything, because a percentage rewards killing equivalent mutants and the only way to kill one of those is to assert an implementation detail. The rule is that new and unclassified survivors are zero. Triage has six outcomes and four are recordable: TEST_GAP is closed by a test rather than written down, EQUIVALENT, UNREACHABLE, PRODUCT_DECISION and TOOL_ARTIFACT carry an argument, and UNKNOWN is recorded so it is visible and fails the gate. Every survivor is triaged individually and assumed real until shown otherwise — about four fifths of those triaged in #135 were. A classification lives in one of two places. Several mutants whose equivalence follows from one structural invariant are suppressed beside the code with a documented `// Stryker disable`; anything isolated goes in `mutation-survivors.json` with its operator, location, reason and evidence. A reason must give the invariant rather than restate the classification, and both mechanisms are checked for rot: a ledger entry whose mutant is now killed fails, an entry naming a missing file fails, and a directive that no longer ignores any mutant fails, so an exemption cannot outlive the construct it argued about. Entries for files a run did not mutate are skipped rather than confirmed. `--write` records new survivors as UNKNOWN, so it cannot turn a red gate green. `scripts/mutation-survivors.test.ts` pins the part that fails silently: the hand-rolled sweep read Vitest's collection-time failure as "no test failed" and reported three killed mutants per slice as survivors, so every status is named in an assertion. The first triage results are included. In `core.ts`, both async continuation loops started their promise chain at `i + 1`; mutating that to `i - 1` re-ran steps that had already run and nothing noticed, which two tests now kill. The three step registrars take the plugin's declared default operation mode when a call omits one, which no built-in distinguished because every one of them either passes a mode or is declared maybe-async. The arity and mode specializations of `createFinalizedPipeExecutor`, the `then(undefined)` loop bounds, and the write-elision guards on `~operationMode` are suppressed with the invariant that makes each equivalent. `pnpm-workspace.yaml` excludes `semver@6.3.1` from the trust policy: it is reached only through `@babel/core`, and pnpm compares trust evidence by publish date across the whole package, so a 6.x patch published after a provenance-signed 7.x release reads as a downgrade. The 6.x line predates npm provenance; nothing was withdrawn. Pinned to the exact version so a genuinely new untrusted release still fails. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…tation found Batch one of the #135 triage, taken in contract-risk order: core first, because it is the substrate under every step. Four test gaps closed, each proved by watching the new assertion fail under the mutation and pass without it. The deferred-message placeholder had no test at all. While resolution is deferred the issue still carries a message, and `~execute` hands that placeholder straight to a plugin author — only the public `execute()` replaces it. Its rule is the tier order restricted to string handlers, and all four tiers could be rewritten unnoticed. The scopes a child issue has already collected did not survive a path replacement in any test. `tuple`'s rest region is the only production caller of `replaceIssuePath`, and nothing nested a scoped structure inside one, so dropping the accumulated context messages and keeping only the new scope was invisible — an inner structure's message is nearer than the tuple's and must still win. The shared-prototype layout was unasserted although it is an architectural contract: registered methods are non-enumerable prototype properties and the fixed schema fields are own enumerable ones, which is what makes `Object.keys` and `for...in` over a schema return its data rather than the whole registered API. All three descriptor flags are pinned together, with the observable consequence asserted separately from the descriptor causing it. Five equivalences suppressed in place, each with the invariant that makes it hold and each verified by comparing executed output across nine failure shapes — single and multi-issue, custom, global and enclosing messages, nested and array — rather than by reasoning: `finalizeFailureResult`'s single-issue arm duplicates what its loop does, its two null guards are dead under the caller's `hasIssueDraft` precondition, `i === 0` reuses a lookup, and `hasIssueDraft` itself only decides how much work happens because finalizing an issue without draft metadata is the identity. `core.ts` is down from 74 undetected mutants to about 39. The gate itself gained a correction this batch found. `NoCoverage` is no longer treated as undetected: Stryker never executes those mutants and reports them from the `perTest` coverage map alone, and that map is wrong here — the empty-capabilities array in `createValchecker` came back `NoCoverage` while a test in `core.test.ts` kills it when the mutation is applied by hand. They are now reported as unconfirmed, with the command that settles them, so the ledger cannot fill up with entries for mutants that are already dead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…valences Batch 1b of the #135 triage. Two test gaps closed, each proved by watching the assertion fail under the mutation. A schema's operation mode only ever climbs, and nothing said so. Every existing case registers its step against a lower current mode, where raising and assigning agree, so replacing the guard with an unconditional write was invisible in all three registrars — a synchronous step after an asynchronous one would have reported the chain synchronous again. A plugin declared `async` was equally invisible. The runtime modes are ordered numbers and the declaration is read with `?? MAYBE_ASYNC`, so folding that into `&&` is harmless for every mode the library actually declares: sync is 0, and maybe-async is the fallback itself. Only an `async` plugin distinguishes reading the declaration from defaulting, and no built-in is one. Worth recording how this was found, because the method nearly missed it: the output digest said the mutant changed nothing, which was true of everything it measured and false of the program. A digest is evidence about the inputs in it. Nine equivalences suppressed in place, each with its invariant, verified by comparing executed output rather than by reasoning — thirteen end-to-end path and scope shapes for the first group, an 85-case matrix of global and step handler shapes in root, nested, raw, thrown and asynchronous positions for the second: - the public executor's sync and async forms, and the error wrapper's sync form, are mode specializations whose general form re-decides `isPromiseLike` at run time; - `prependIssuePath`'s nullish path guard is invariantly true — all thirteen call sites pass an array — and its empty-path branch is spread elision, since `[...path, ...[]]` is `[...path]`; - attaching draft metadata built from an absent scope resolves back to the issue's own message, and copying an absent metadata value leaves the issue reading as undrafted; - widening what `hasDynamicMessageForCode` counts as dynamic only defers to the resolver that would answer the same, which is why deferring is always the safe direction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three `// Stryker disable next-line` directives added in the last batch silently did nothing. A directive attaches only where the comment is a leading comment of the mutated node, and two placements look right without being: inside a ternary, before the branch meant to be covered, and before an `else if`, where the comment reads as trailing the preceding block. There is no error for either — the mutant returns as `Survived` with the directive sitting above it. The suppression rot check is what found them, which is the first time it has earned its keep: it reported all three as ignoring nothing. Two further directives were caught by the reason rule for saying "see above" and "as above." rather than an argument, and both now state their invariant. The six mutants the inert directives covered move to `mutation-survivors.json`, which opens the ledger. Each carries the invariant and the thirteen-shape output comparison it was checked against, and each says why it is recorded rather than suppressed in place — a wider block directive would have covered neighbouring mutants nobody has argued about, which is the trade the ledger exists to avoid. The limitation is written into the mutation reference so the next person does not rediscover it. `core.ts` now reports 12 unexplained survivors and 2 unconfirmed, from 74. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…nces Batch 1d, closing out `core.ts`. Four test gaps closed, each proved by watching the assertion fail under the mutation: - the `TypeError` text behind an empty failure collection, which is the only thing telling a plugin author which invariant broke and which `expect.any(TypeError)` accepts any wording of, including none; - `createIssue` omitting the `context` key rather than assigning it `undefined`. Every other assertion here uses `toEqual` or `toMatchObject`, and both treat an undefined property as equal to a missing one, so always assigning it looked correct everywhere; - resolving a `core:message_exception` through a global handler that did not throw for it. The existing case has the handler throw for every code including that one, so the fallback message survives either way and resolving it is indistinguishable there; - the placeholder a deferred internal issue carries. `??` and `&&` agree whenever the static path committed, so only a deferring global separates falling back to the placeholder from discarding it. Four equivalences suppressed with their invariants: the two `Symbol.for` descriptions are compared only against themselves through one module constant; the `'maybe-async'` default parameter resolves through `toRuntimeOperationMode`'s catch-all, which maps anything that is neither `'sync'` nor `'async'` to the same runtime mode; and defining the capabilities symbol with `undefined` is indistinguishable from not defining it, because the one reader tests `!= null` and the method enumeration skips that symbol by name. `hasDynamicMessageForCode`'s directive became block form. It was `next-line` over a four-line expression, so the two trailing conjuncts came back as survivors with a directive apparently sitting above them — the same attachment trap recorded last commit, in its second guise. One assertion was written and then removed rather than kept: an initial-schema operation mode check that passed under its mutation, because it read the outer chain's mode instead of the composed schema's. A test that looks like coverage without discriminating anything is the defect this whole exercise is about, so `~operationMode` in `createInitialSchema` stays untriaged instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The `'maybe-async'` default parameter was classified equivalent on a runtime argument — `toRuntimeOperationMode` maps anything but `'sync'`/`'async'` to maybe-async, which is true. But the parameter is typed `OperationMode`, so every replacement the runner produces there is a type error `pnpm typecheck` rejects. It is a tool artifact rather than an equivalent program, and the reason now says so. Found while measuring the long tail, where 129 survivors turned out to be the same class: operation-mode string literals that `tsc` refuses. They are admitted only because `stryker.config.mjs` sets `disableTypeChecks`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

DevilTea · 2026-07-31T08:56:14Z

Cross-references: closes #135, remainder consolidated into #139.

#136 needed no action — it is a merged pull request rather than an open issue, so there was
nothing to close. Its two outstanding items are handled: the compare job's before/after SHA
table and benchmarks/README.md's description of the impact unit are both in this branch. The
third, optimising map/set's buffered path so both accepted-regressions.mjs entries can be
deleted, is carried to #139 and was not attempted.

DevilTea and others added 8 commits July 31, 2026 14:56

This was referenced Jul 31, 2026

test: finish the mutation-adequacy remainder, or rebuild it — consolidated from #134/#135/#136/#137 #139

Open

test: finish the step and core audit stopped part-way in #134 #135

Closed

Navigation Menu

test: turn the #135 mutation audit into a repeatable test-adequacy gate - #138

test: turn the #135 mutation audit into a repeatable test-adequacy gate#138
DevilTea wants to merge 8 commits into
mainfrom
test/mutation-adequacy-system

DevilTea commented Jul 31, 2026

Uh oh!

DevilTea commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

DevilTea commented Jul 31, 2026

What is finished

What the exercise found about its own tooling

What remains, measured

Supply-chain note

Verification

Uh oh!

DevilTea commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant