Skip to content

fix(dogfood): batch 5 — the last three audit findings, and a producer that could fabricate its own verdict - #2458

Merged
noahgift merged 5 commits into
mainfrom
integration/dogfood-audit-batch7
Aug 13, 2026
Merged

fix(dogfood): batch 5 — the last three audit findings, and a producer that could fabricate its own verdict#2458
noahgift merged 5 commits into
mainfrom
integration/dogfood-audit-batch7

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

fix(dogfood): batch 5 — the last three audit findings, and a producer that could fabricate its own verdict

Closes out #2377 and #2375. Four branches, two of which an adversarial pass sent
back before they could merge.

#2377-3 — three lints documented a producer the binary did not have

attn-parity-lint, audio-inspect-lint and embed-viz-lint each name a
producer command in their help text. apr kernel parity, apr dataset audio-inspect and apr debug embed-viz did not exist, so the only documented
way to produce the observation they consume was unreachable: their gates had
never run on real data and could not. All three are implemented, wired into the
clap tree and the dispatch arm, with round-trip falsifiers — run the producer,
feed its output to the lint, assert exit 0 — plus the negative, that a corrupted
producer body makes the lint exit non-zero, so the round trip cannot pass
vacuously.

The first attempt shipped the defect this audit exists to eliminate.
Replacing the emitted metrics in kernel_parity.rs with the literals
max_abs_diff: 0.0, cosine_sim: 1.0 — a producer fabricating perfect parity —
left ALL 16 tests green, including the one named
the_parity_metrics_are_not_vacuous, which never went through run() at all:
it re-measured the values itself in the test body. Nothing asserted that the
EMITTED numbers were the MEASURED ones. Two falsifiers now bind emitted JSON to
measurement, and the 0.0/1.0 mutation turns them red.

And it emitted garbage on the format its own lint targets. locate_embedding
took shape[0] as vocab for every format. GGUF reports GGML ne order, so
token_embd.weight on Qwen3.5-0.8B-Q4_K_M is [1024, 248320] — the producer
wrote 1024 rows for a 248320-token vocabulary and embed-viz-lint exited 5 on
its own producer's output. The axis rule is now per-format and measured, not
assumed: reading the GGUF payload as [vocab][hidden] matches the same model's
SafeTensors row at cosine 0.999, transposed at 0.014. The tests missed it because
the only fixture was an APR file — the one layout where the axes coincide.

Note the trap the contract now documents: token_str is resolved by row index,
so it reads correctly no matter which axis produced the coordinates. No assertion
on token_str can detect this. The tests assert row count against the vocabulary
and coordinates against a hidden-length slice.

#2375 — every streamed chat reply was empty, and /v1/metrics measured nothing

Also remediated after review. A falsifier named
v1_metrics_model_name_is_derived_not_a_constant PASSED on a constant
("default"); the derived branch was executed by no test, and it only went red on
revert because the previous constant differed. A test comment and commit message
claimed 0.63.0-era provenance for a code path that did not exist in v0.63.0. And
the temperature fix special-cased only == 0.0, so negative, NaN and +inf still
reached apply_temperature and returned HTTP 500 — the same class as #2391,
where NaN <= 0.0 is false and a guard that looks like validation admits it.

#2310 — wasm32 could not compile

(seed * 6364136223846793005 + ...) in the SGD epoch shuffle: the literal does
not fit usize on a 32-bit target, so aprender-core failed to build for wasm32
entirely.

pmat complexity — the files that taxed every commit

The pre-commit hook rejected any commit TOUCHING validate.rs,
dispatch_analysis.rs, prometheus_classifier.rs or react_trace_classifier.rs;
three commits this week landed with --no-verify carrying before/after tables
proving they had not moved the metric.

One claim corrected rather than merged

The producer branch reported "PCA exit 124 @300S -> exit 0 @7s". Independent
re-verification measured the DEFAULT --projection pca on that model at exit
124, wall 300.33s, no CSV — with an unbounded rerun still going at 2198s. The 7s
was the RANDOM projection. Fixing the axes made PCA tractable, not fast: at fixed
hidden=1024 the cost is linear in ROWS at ~6.3ms each (5,000 rows 82.6s; 20,000
rows 176.9s), so a 248,320-token vocabulary does not finish in 300s.

The book now carries that scaling table, and FALSIFY-LINT-PRODUCER-008's scope
says explicitly that RUNTIME is discharged nowhere — every test in the contract
uses --projection random. Stated rather than gated on purpose: a falsifier for
a 26-30 minute run is a 26-30 minute CI test. An unstated performance claim in a
book chapter is what this audit exists to stop.

Closes #2377
Closes #2375
Closes #2310

Refs #2373


Verification

  • cargo test -p apr-cli -p aprender-serve -p aprender-core --lib36,801 passed, 0 failed
    (7,064 + 14,099 + 15,638; these are the only crates this diff touches)
  • CI's 18-command integration chain + all 9 guard scripts — 27/27, TOTAL_FAILS=0
  • scripts/check_assertions_exclude.sh — 319 sites, delta 0
  • cargo fmt --all -- --check — clean
  • contract count 1771 matches the README claims row (FALSIFY-README-007 green)

A full cargo test --workspace --lib was started twice and killed externally
both times, at 48,161 and ~0 tests in with 0 failures in each. Rather than
report a number I did not see complete, the figures above are the targeted run
that did finish. CI runs the full workspace on this PR regardless.

noahgift and others added 3 commits August 13, 2026 16:42
…asm32 compile error AND a live 64-bit panic (#2310)

#2310 reported four "literal out of range for usize" errors when aprender-core is
built for wasm32-unknown-unknown. Reproduced verbatim at HEAD with the reporter's
own command:

    RUSTFLAGS='--cfg getrandom_backend="custom"' \
      cargo check --locked -p aprender-core --no-default-features \
      --target wasm32-unknown-unknown

    error: literal out of range for `usize`
       --> crates/aprender-core/src/classification/mod.rs:377:33
        |
    377 |                 let j = (seed * 6364136223846793005 + i * 1442695040888963407) % (i + 1);
        |                                 ^^^^^^^^^^^^^^^^^^^
        |
        = note: the literal `6364136223846793005` does not fit into the type `usize` whose range is `0..=4294967295`
        = note: `#[deny(overflowing_literals)]` on by default
    ... (x4: the two MMIX constants, in fit_stochastic and again in fit_minibatch)
    error: could not compile `aprender-core` (lib) due to 4 previous errors

ROOT CAUSE, and it is worse than the report. The Fisher-Yates partner index for the
per-epoch sample shuffle was written as a bare usize expression:

    let j = (seed * 6364136223846793005 + i * 1442695040888963407) % (i + 1);

Both MMIX LCG constants exceed u32::MAX, so on a 32-bit target they cannot even be
typed. But the products also overflow u64 — seed * MUL from seed == 3, i * INC from
i == 13 — so on x86_64 this aborts under overflow checking too. FitMode::Stochastic
and FitMode::MiniBatch had ZERO test coverage anywhere in the tree (nothing outside
this file ever constructed either variant), which is why a defect that panics with
default settings on the developers' own architecture shipped in v0.60.0 and had to
be reported from outside. #2310 is not a portability nit; it is the 32-bit shadow of
a live 64-bit crash.

Every other LCG site in the workspace (28 of them, aprender-data, aprender-serve,
apr-cli, aprender-core/online/, ...) already uses u64 wrapping_mul/wrapping_add.
These two were the only raw-usize copies.

FIX. Extract the expression into one shuffle_partner(seed, i) helper computed in
explicit wrapping u64, and call it from both SGD modes. wrapping_* reproduces the
prior 64-bit RELEASE-mode result bit-for-bit, so no already-trained model's epoch
order shifts; what changes is that the debug build no longer aborts and the 32-bit
build now compiles at all.

FALSIFIER — crates/aprender-core/src/classification/tests_sgd_portable_shuffle.rs,
5 tests, wired into aprender-core --lib (so CI's workspace-test runs them).
MUTATION-VERIFIED: with shuffle_partner's body reverted to the pre-fix usize
expression, all 5 fail on x86_64. Verbatim:

    running 5 tests

    thread 'classification::tests_sgd_portable_shuffle::test_epoch_shuffle_is_a_permutation' (2354541) panicked at crates/aprender-core/src/classification/mod.rs:118:35:
    attempt to multiply with overflow
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

    thread 'classification::tests_sgd_portable_shuffle::test_shuffle_partner_matches_64bit_wrapping_reference' (2354543) panicked at crates/aprender-core/src/classification/mod.rs:118:6:
    attempt to multiply with overflow

    thread 'classification::tests_sgd_portable_shuffle::test_shuffle_partner_never_exceeds_i' (2354544) panicked at crates/aprender-core/src/classification/mod.rs:118:35:
    attempt to multiply with overflow
    thread 'classification::tests_sgd_portable_shuffle::test_stochastic_fit_survives_overflowing_epoch_and_index' (2354545) panicked at crates/aprender-core/src/classification/mod.rs:118:35:
    attempt to multiply with overflow
    thread 'classification::tests_sgd_portable_shuffle::test_minibatch_fit_survives_overflowing_epoch_and_index' (2354542) panicked at crates/aprender-core/src/classification/mod.rs:118:35:
    attempt to multiply with overflow

    failures:
        classification::tests_sgd_portable_shuffle::test_epoch_shuffle_is_a_permutation
        classification::tests_sgd_portable_shuffle::test_minibatch_fit_survives_overflowing_epoch_and_index
        classification::tests_sgd_portable_shuffle::test_shuffle_partner_matches_64bit_wrapping_reference
        classification::tests_sgd_portable_shuffle::test_shuffle_partner_never_exceeds_i
        classification::tests_sgd_portable_shuffle::test_stochastic_fit_survives_overflowing_epoch_and_index

    test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 14018 filtered out; finished in 0.00s

Restored: test result: ok. 5 passed; 0 failed. Whole crate: 14021 passed, 0 failed.

REGRESSION GUARD — scripts/check_wasm32_core_builds.sh, also mutation-verified.
It runs the reporter's exact command (the getrandom_backend cfg is a real
precondition: without it getrandom 0.3 refuses the target and the build dies before
reaching our code, so its absence would produce a failure that is not ours). With
the fix reverted the guard exits 1 and prints:

    FAIL: aprender-core does not compile for wasm32-unknown-unknown (exit 101).
    error: literal out of range for `usize`
       --> crates/aprender-core/src/classification/mod.rs:123:13
        = note: the literal `6364136223846793005` does not fit into the type `usize` whose range is `0..=4294967295`

The guard fails CLOSED (no rustup, no target, no toolchain => exit 1, never a silent
pass) and carries --self-test, which compiles the #2310 literal as a usize for
wasm32 and requires rustc to reject it. That self-test is itself mutation-verified:
changing DEFECT_LITERAL to 5 makes it report

    SELF-TEST FAIL: rustc ACCEPTED 5 as a usize on wasm32-unknown-unknown.
                    This guard can no longer detect the #2310 defect class.

so the guard cannot go green by being toothless if a toolchain ever demotes
overflowing_literals.

CONTRACT — contracts/apr-stochastic-lr-v1.yaml gains FALSIFY-SGD-004 plus two
proof obligations (partner index in [0, i]; wrapping u64 never usize). Its grep is
anchored on a NON-ZERO pass count, "^test result: ok\. [1-9][0-9]* passed", because
the unanchored form matches "test result: ok. 0 passed". pv validate: 0 errors,
0 warnings. No new YAML file, so the README count row (1770) is unchanged.

Gates: cargo fmt --all --check clean; cargo clippy -p aprender-core --lib
-D warnings clean; cargo test -p aprender-core --lib 14021 passed 0 failed;
readme_contract 15 passed; check_assertions_exclude delta 0; check_pass_grep_anchored
OK; bashrs lint 0 errors (3 warnings, vs 14 on the existing guards).

NOT DONE, stated plainly: the wasm32 guard is NOT wired into .github/workflows/ —
CI workflow edits require sign-off, and adding a wasm32 job changes runner
provisioning. It is reachable as "make check-wasm32" or
"bash scripts/check_wasm32_core_builds.sh". The 64-bit half of the same defect IS
gated in CI today via the 5 lib tests, so the arithmetic cannot silently regress;
only the pure 32-bit-compile aspect depends on the manual script.

ALSO FOUND, not fixed here: FALSIFY-SGD-002 and FALSIFY-SGD-003 in the same
contract are vacuous. They run "cargo test ... -- test_stochastic_imbalanced" and
"-- test_minibatch_equals_batch"; neither test exists anywhere in the workspace, so
cargo prints "test result: ok. 0 passed; ... filtered out" and their unanchored
grep "ok" matches it. Both have passed since GH-428 while asserting nothing.
Anchoring them would correctly turn them RED, which is a separate ticket.

Closes #2310

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…so the suite failed ~1 run in 25

`cargo test -p apr-cli --lib` failed on clean origin/main before any of my
changes:

  ---- commands::publish::tests::test_publish_dry_run_json_stdout_parses_as_json stdout ----
  thread '...' panicked at crates/apr-cli/src/commands/publish_tests.rs:840:5:
  assertion `left == right` failed
    left: Number(0)
   right: 16

  test result: FAILED. 7003 passed; 1 failed; 12 ignored; 0 measured; 0 filtered out

ROOT CAUSE. `dry_run_plan_fixture` built its artifact at one fixed path,
`$TMPDIR/apr_publish_json_fixture/model.safetensors`, and BOTH
test_publish_dry_run_json_stdout_parses_as_json and
test_publish_dry_run_human_mode_is_still_human called it. The harness runs them
on parallel threads. `fs::write` truncates to 0 before writing the 16 bytes, so
one test could stat the file inside the other's truncation window and read
size_bytes 0. Hence `left: Number(0)` against the expected 16 — the number is
not arbitrary, it is `b"not-a-real-model".len()`.

Fix: `slug` parameter, one fixture directory per test. Disjoint paths, so the
two tests can no longer contend — the fix is by construction, not by widening a
window.

REPRODUCTION. It did not reproduce in 60 isolated runs of the two tests, nor in
25 unloaded full-suite runs; the window is too narrow. It DOES reproduce under
the condition the original failure occurred in — a full suite competing with
other work on the box (at the time, a 492-file `pmat analyze complexity` scan).
Running the built test binary 25x with one spinner per core (48):

  run 4: FAIL
  thread 'commands::publish::tests::test_publish_dry_run_json_stdout_parses_as_json'
    panicked at crates/apr-cli/src/commands/publish_tests.rs:840:5:
    left: Number(0)
   right: 16
  RESULT: 1/25 full-suite runs failed under 48-core load

Same test, same line, same values as the origin/main failure. After the fix, the
identical probe reports 0/25 under the same 48-core load.

Honest read on the evidence: 0/25 post-fix is weak on its own against a ~1/25
base rate — roughly a coin flip's worth of sampling power. The load-bearing
argument is structural (the shared path is gone, so the race has no surface),
and the sampling is corroboration rather than proof.

Not marked #[ignore] — per the repo's andon rule, a flake gets fixed, not muted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… that could fabricate its own verdict

Closes out #2377 and #2375. Four branches, two of which an adversarial pass sent
back before they could merge.

## #2377-3 — three lints documented a producer the binary did not have

`attn-parity-lint`, `audio-inspect-lint` and `embed-viz-lint` each name a
producer command in their help text. `apr kernel parity`, `apr dataset
audio-inspect` and `apr debug embed-viz` did not exist, so the only documented
way to produce the observation they consume was unreachable: their gates had
never run on real data and could not. All three are implemented, wired into the
clap tree and the dispatch arm, with round-trip falsifiers — run the producer,
feed its output to the lint, assert exit 0 — plus the negative, that a corrupted
producer body makes the lint exit non-zero, so the round trip cannot pass
vacuously.

**The first attempt shipped the defect this audit exists to eliminate.**
Replacing the emitted metrics in `kernel_parity.rs` with the literals
`max_abs_diff: 0.0, cosine_sim: 1.0` — a producer fabricating perfect parity —
left ALL 16 tests green, including the one named
`the_parity_metrics_are_not_vacuous`, which never went through `run()` at all:
it re-measured the values itself in the test body. Nothing asserted that the
EMITTED numbers were the MEASURED ones. Two falsifiers now bind emitted JSON to
measurement, and the 0.0/1.0 mutation turns them red.

**And it emitted garbage on the format its own lint targets.** `locate_embedding`
took `shape[0]` as vocab for every format. GGUF reports GGML `ne` order, so
`token_embd.weight` on Qwen3.5-0.8B-Q4_K_M is `[1024, 248320]` — the producer
wrote 1024 rows for a 248320-token vocabulary and `embed-viz-lint` exited 5 on
its own producer's output. The axis rule is now per-format and measured, not
assumed: reading the GGUF payload as `[vocab][hidden]` matches the same model's
SafeTensors row at cosine 0.999, transposed at 0.014. The tests missed it because
the only fixture was an APR file — the one layout where the axes coincide.

Note the trap the contract now documents: `token_str` is resolved by row index,
so it reads correctly no matter which axis produced the coordinates. No assertion
on `token_str` can detect this. The tests assert row count against the vocabulary
and coordinates against a hidden-length slice.

## #2375 — every streamed chat reply was empty, and /v1/metrics measured nothing

Also remediated after review. A falsifier named
`v1_metrics_model_name_is_derived_not_a_constant` PASSED on a constant
("default"); the derived branch was executed by no test, and it only went red on
revert because the previous constant differed. A test comment and commit message
claimed 0.63.0-era provenance for a code path that did not exist in v0.63.0. And
the temperature fix special-cased only `== 0.0`, so negative, NaN and +inf still
reached `apply_temperature` and returned HTTP 500 — the same class as #2391,
where `NaN <= 0.0` is false and a guard that looks like validation admits it.

## #2310 — wasm32 could not compile

`(seed * 6364136223846793005 + ...)` in the SGD epoch shuffle: the literal does
not fit `usize` on a 32-bit target, so aprender-core failed to build for wasm32
entirely.

## pmat complexity — the files that taxed every commit

The pre-commit hook rejected any commit TOUCHING validate.rs,
dispatch_analysis.rs, prometheus_classifier.rs or react_trace_classifier.rs;
three commits this week landed with --no-verify carrying before/after tables
proving they had not moved the metric.

## One claim corrected rather than merged

The producer branch reported "PCA exit 124 @300S -> exit 0 @7s". Independent
re-verification measured the DEFAULT `--projection pca` on that model at exit
124, wall 300.33s, no CSV — with an unbounded rerun still going at 2198s. The 7s
was the RANDOM projection. Fixing the axes made PCA tractable, not fast: at fixed
hidden=1024 the cost is linear in ROWS at ~6.3ms each (5,000 rows 82.6s; 20,000
rows 176.9s), so a 248,320-token vocabulary does not finish in 300s.

The book now carries that scaling table, and FALSIFY-LINT-PRODUCER-008's scope
says explicitly that RUNTIME is discharged nowhere — every test in the contract
uses `--projection random`. Stated rather than gated on purpose: a falsifier for
a 26-30 minute run is a 26-30 minute CI test. An unstated performance claim in a
book chapter is what this audit exists to stop.

Closes #2377
Closes #2375
Closes #2310

Refs #2373
@noahgift
noahgift enabled auto-merge August 13, 2026 15:10
CI failed this batch with

    FAIL: book/src/cli/beat-run.md does not exist (apr beat-run has no chapter)

`apr beat-run` shipped in #1995. `book/src/cli/beat-run.md` has never existed in
git history. FALSIFY-BOOK-CLI-PARITY-001 has therefore been failing-in-waiting
for months, and nothing noticed, because .github/workflows/book.yml filters on

    paths:
      - "book/**"
      - ".github/workflows/book.yml"

A parity gate asserts that two things agree. Watching only ONE of them means a
command can be added with no chapter and the gate never runs to say so; it took
an unrelated batch that happened to touch book/ to wake it. That is the same
class as the four gates fixed in batch 4 — a check that cannot fail when the
thing it guards actually moves.

Fixes, in order of importance:

1. book.yml now also triggers on the CLI command tree — commands_enum.rs,
   extended_commands.rs, tool_commands.rs — so adding a subcommand runs the gate
   that checks subcommands.
2. book/src/cli/beat-run.md, written from beat_run.rs rather than from the help
   text: the two modes (report vs judge), the exit-code table, and why an
   UNJUDGEABLE contract exits non-zero — a beat that cannot decide is not a beat
   that passed, which is the whole point of the runner.
3. book/src/lib/datasets.md and book/src/lib/pipeline.md. Fixing (1) revealed
   that check_book_lib_parity.sh, the very next step in the same job, was
   failing for the identical reason on aprender::datasets and aprender::pipeline
   — both missing on main too, both hidden behind the same path filter. Fixing
   only the CLI half would have handed the next batch a red build.

The datasets chapter names what is NOT implemented (load_digits,
load_california_housing) rather than implying a completeness the module does not
have.

Verified: check_book_cli_parity 106/106, check_book_lib_parity 71/71, plus
check_book_example_block, check_book_lib_example_block and check_book_linkcheck
all rc=0 — the last of which proves the new cross-references resolve.

Refs #2373
Exit 127 — command not found — the FIRST time this step ever ran.

The `pmat comply gate` step guards `pmat` with `command -v` and a comment saying
it "tolerates absence (pmat may not be available on every runner)", and then one
line EARLIER invokes a bare `pv` with no guard at all, in a step whose entire
purpose is running pv. Nothing caught it because every previous run of this
workflow died at the parity gate above, so execution never reached the line.

That is the third instance of the same class in this one job, all surfaced by
fixing the first: the CLI parity gate that only ran when the book changed, the
lib parity gate failing identically behind it, and now a step that could not run
the tool it exists to run.

`pv` is IN-TREE — crates/aprender-contracts-cli, `[[bin]] name = "pv"` — so the
fix is to build it from the tree rather than depend on an installed copy:

    cargo run --quiet -p aprender-contracts-cli --bin pv -- \
      validate contracts/apr-book-completeness-v1.yaml

That is the dogfood rule in CLAUDE.md ("pv is THE dogfooded contract CLI"), and
it has the property an installed binary does not: the validator cannot drift from
the contracts it validates, because both come from the commit under test.

Verified locally: rc=0, "0 error(s), 0 warning(s) / Contract is valid."

Refs #2373
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant