fix(dogfood): batch 5 — the last three audit findings, and a producer that could fabricate its own verdict - #2458
Merged
Merged
Conversation
…asm32 compile error AND a live 64-bit panic (#2310) #2310 reported four "literal out of range for usize" errors when aprender-core is built for wasm32-unknown-unknown. Reproduced verbatim at HEAD with the reporter's own command: RUSTFLAGS='--cfg getrandom_backend="custom"' \ cargo check --locked -p aprender-core --no-default-features \ --target wasm32-unknown-unknown error: literal out of range for `usize` --> crates/aprender-core/src/classification/mod.rs:377:33 | 377 | let j = (seed * 6364136223846793005 + i * 1442695040888963407) % (i + 1); | ^^^^^^^^^^^^^^^^^^^ | = note: the literal `6364136223846793005` does not fit into the type `usize` whose range is `0..=4294967295` = note: `#[deny(overflowing_literals)]` on by default ... (x4: the two MMIX constants, in fit_stochastic and again in fit_minibatch) error: could not compile `aprender-core` (lib) due to 4 previous errors ROOT CAUSE, and it is worse than the report. The Fisher-Yates partner index for the per-epoch sample shuffle was written as a bare usize expression: let j = (seed * 6364136223846793005 + i * 1442695040888963407) % (i + 1); Both MMIX LCG constants exceed u32::MAX, so on a 32-bit target they cannot even be typed. But the products also overflow u64 — seed * MUL from seed == 3, i * INC from i == 13 — so on x86_64 this aborts under overflow checking too. FitMode::Stochastic and FitMode::MiniBatch had ZERO test coverage anywhere in the tree (nothing outside this file ever constructed either variant), which is why a defect that panics with default settings on the developers' own architecture shipped in v0.60.0 and had to be reported from outside. #2310 is not a portability nit; it is the 32-bit shadow of a live 64-bit crash. Every other LCG site in the workspace (28 of them, aprender-data, aprender-serve, apr-cli, aprender-core/online/, ...) already uses u64 wrapping_mul/wrapping_add. These two were the only raw-usize copies. FIX. Extract the expression into one shuffle_partner(seed, i) helper computed in explicit wrapping u64, and call it from both SGD modes. wrapping_* reproduces the prior 64-bit RELEASE-mode result bit-for-bit, so no already-trained model's epoch order shifts; what changes is that the debug build no longer aborts and the 32-bit build now compiles at all. FALSIFIER — crates/aprender-core/src/classification/tests_sgd_portable_shuffle.rs, 5 tests, wired into aprender-core --lib (so CI's workspace-test runs them). MUTATION-VERIFIED: with shuffle_partner's body reverted to the pre-fix usize expression, all 5 fail on x86_64. Verbatim: running 5 tests thread 'classification::tests_sgd_portable_shuffle::test_epoch_shuffle_is_a_permutation' (2354541) panicked at crates/aprender-core/src/classification/mod.rs:118:35: attempt to multiply with overflow note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'classification::tests_sgd_portable_shuffle::test_shuffle_partner_matches_64bit_wrapping_reference' (2354543) panicked at crates/aprender-core/src/classification/mod.rs:118:6: attempt to multiply with overflow thread 'classification::tests_sgd_portable_shuffle::test_shuffle_partner_never_exceeds_i' (2354544) panicked at crates/aprender-core/src/classification/mod.rs:118:35: attempt to multiply with overflow thread 'classification::tests_sgd_portable_shuffle::test_stochastic_fit_survives_overflowing_epoch_and_index' (2354545) panicked at crates/aprender-core/src/classification/mod.rs:118:35: attempt to multiply with overflow thread 'classification::tests_sgd_portable_shuffle::test_minibatch_fit_survives_overflowing_epoch_and_index' (2354542) panicked at crates/aprender-core/src/classification/mod.rs:118:35: attempt to multiply with overflow failures: classification::tests_sgd_portable_shuffle::test_epoch_shuffle_is_a_permutation classification::tests_sgd_portable_shuffle::test_minibatch_fit_survives_overflowing_epoch_and_index classification::tests_sgd_portable_shuffle::test_shuffle_partner_matches_64bit_wrapping_reference classification::tests_sgd_portable_shuffle::test_shuffle_partner_never_exceeds_i classification::tests_sgd_portable_shuffle::test_stochastic_fit_survives_overflowing_epoch_and_index test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 14018 filtered out; finished in 0.00s Restored: test result: ok. 5 passed; 0 failed. Whole crate: 14021 passed, 0 failed. REGRESSION GUARD — scripts/check_wasm32_core_builds.sh, also mutation-verified. It runs the reporter's exact command (the getrandom_backend cfg is a real precondition: without it getrandom 0.3 refuses the target and the build dies before reaching our code, so its absence would produce a failure that is not ours). With the fix reverted the guard exits 1 and prints: FAIL: aprender-core does not compile for wasm32-unknown-unknown (exit 101). error: literal out of range for `usize` --> crates/aprender-core/src/classification/mod.rs:123:13 = note: the literal `6364136223846793005` does not fit into the type `usize` whose range is `0..=4294967295` The guard fails CLOSED (no rustup, no target, no toolchain => exit 1, never a silent pass) and carries --self-test, which compiles the #2310 literal as a usize for wasm32 and requires rustc to reject it. That self-test is itself mutation-verified: changing DEFECT_LITERAL to 5 makes it report SELF-TEST FAIL: rustc ACCEPTED 5 as a usize on wasm32-unknown-unknown. This guard can no longer detect the #2310 defect class. so the guard cannot go green by being toothless if a toolchain ever demotes overflowing_literals. CONTRACT — contracts/apr-stochastic-lr-v1.yaml gains FALSIFY-SGD-004 plus two proof obligations (partner index in [0, i]; wrapping u64 never usize). Its grep is anchored on a NON-ZERO pass count, "^test result: ok\. [1-9][0-9]* passed", because the unanchored form matches "test result: ok. 0 passed". pv validate: 0 errors, 0 warnings. No new YAML file, so the README count row (1770) is unchanged. Gates: cargo fmt --all --check clean; cargo clippy -p aprender-core --lib -D warnings clean; cargo test -p aprender-core --lib 14021 passed 0 failed; readme_contract 15 passed; check_assertions_exclude delta 0; check_pass_grep_anchored OK; bashrs lint 0 errors (3 warnings, vs 14 on the existing guards). NOT DONE, stated plainly: the wasm32 guard is NOT wired into .github/workflows/ — CI workflow edits require sign-off, and adding a wasm32 job changes runner provisioning. It is reachable as "make check-wasm32" or "bash scripts/check_wasm32_core_builds.sh". The 64-bit half of the same defect IS gated in CI today via the 5 lib tests, so the arithmetic cannot silently regress; only the pure 32-bit-compile aspect depends on the manual script. ALSO FOUND, not fixed here: FALSIFY-SGD-002 and FALSIFY-SGD-003 in the same contract are vacuous. They run "cargo test ... -- test_stochastic_imbalanced" and "-- test_minibatch_equals_batch"; neither test exists anywhere in the workspace, so cargo prints "test result: ok. 0 passed; ... filtered out" and their unanchored grep "ok" matches it. Both have passed since GH-428 while asserting nothing. Anchoring them would correctly turn them RED, which is a separate ticket. Closes #2310 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…so the suite failed ~1 run in 25
`cargo test -p apr-cli --lib` failed on clean origin/main before any of my
changes:
---- commands::publish::tests::test_publish_dry_run_json_stdout_parses_as_json stdout ----
thread '...' panicked at crates/apr-cli/src/commands/publish_tests.rs:840:5:
assertion `left == right` failed
left: Number(0)
right: 16
test result: FAILED. 7003 passed; 1 failed; 12 ignored; 0 measured; 0 filtered out
ROOT CAUSE. `dry_run_plan_fixture` built its artifact at one fixed path,
`$TMPDIR/apr_publish_json_fixture/model.safetensors`, and BOTH
test_publish_dry_run_json_stdout_parses_as_json and
test_publish_dry_run_human_mode_is_still_human called it. The harness runs them
on parallel threads. `fs::write` truncates to 0 before writing the 16 bytes, so
one test could stat the file inside the other's truncation window and read
size_bytes 0. Hence `left: Number(0)` against the expected 16 — the number is
not arbitrary, it is `b"not-a-real-model".len()`.
Fix: `slug` parameter, one fixture directory per test. Disjoint paths, so the
two tests can no longer contend — the fix is by construction, not by widening a
window.
REPRODUCTION. It did not reproduce in 60 isolated runs of the two tests, nor in
25 unloaded full-suite runs; the window is too narrow. It DOES reproduce under
the condition the original failure occurred in — a full suite competing with
other work on the box (at the time, a 492-file `pmat analyze complexity` scan).
Running the built test binary 25x with one spinner per core (48):
run 4: FAIL
thread 'commands::publish::tests::test_publish_dry_run_json_stdout_parses_as_json'
panicked at crates/apr-cli/src/commands/publish_tests.rs:840:5:
left: Number(0)
right: 16
RESULT: 1/25 full-suite runs failed under 48-core load
Same test, same line, same values as the origin/main failure. After the fix, the
identical probe reports 0/25 under the same 48-core load.
Honest read on the evidence: 0/25 post-fix is weak on its own against a ~1/25
base rate — roughly a coin flip's worth of sampling power. The load-bearing
argument is structural (the shared path is gone, so the race has no surface),
and the sampling is corroboration rather than proof.
Not marked #[ignore] — per the repo's andon rule, a flake gets fixed, not muted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… that could fabricate its own verdict Closes out #2377 and #2375. Four branches, two of which an adversarial pass sent back before they could merge. ## #2377-3 — three lints documented a producer the binary did not have `attn-parity-lint`, `audio-inspect-lint` and `embed-viz-lint` each name a producer command in their help text. `apr kernel parity`, `apr dataset audio-inspect` and `apr debug embed-viz` did not exist, so the only documented way to produce the observation they consume was unreachable: their gates had never run on real data and could not. All three are implemented, wired into the clap tree and the dispatch arm, with round-trip falsifiers — run the producer, feed its output to the lint, assert exit 0 — plus the negative, that a corrupted producer body makes the lint exit non-zero, so the round trip cannot pass vacuously. **The first attempt shipped the defect this audit exists to eliminate.** Replacing the emitted metrics in `kernel_parity.rs` with the literals `max_abs_diff: 0.0, cosine_sim: 1.0` — a producer fabricating perfect parity — left ALL 16 tests green, including the one named `the_parity_metrics_are_not_vacuous`, which never went through `run()` at all: it re-measured the values itself in the test body. Nothing asserted that the EMITTED numbers were the MEASURED ones. Two falsifiers now bind emitted JSON to measurement, and the 0.0/1.0 mutation turns them red. **And it emitted garbage on the format its own lint targets.** `locate_embedding` took `shape[0]` as vocab for every format. GGUF reports GGML `ne` order, so `token_embd.weight` on Qwen3.5-0.8B-Q4_K_M is `[1024, 248320]` — the producer wrote 1024 rows for a 248320-token vocabulary and `embed-viz-lint` exited 5 on its own producer's output. The axis rule is now per-format and measured, not assumed: reading the GGUF payload as `[vocab][hidden]` matches the same model's SafeTensors row at cosine 0.999, transposed at 0.014. The tests missed it because the only fixture was an APR file — the one layout where the axes coincide. Note the trap the contract now documents: `token_str` is resolved by row index, so it reads correctly no matter which axis produced the coordinates. No assertion on `token_str` can detect this. The tests assert row count against the vocabulary and coordinates against a hidden-length slice. ## #2375 — every streamed chat reply was empty, and /v1/metrics measured nothing Also remediated after review. A falsifier named `v1_metrics_model_name_is_derived_not_a_constant` PASSED on a constant ("default"); the derived branch was executed by no test, and it only went red on revert because the previous constant differed. A test comment and commit message claimed 0.63.0-era provenance for a code path that did not exist in v0.63.0. And the temperature fix special-cased only `== 0.0`, so negative, NaN and +inf still reached `apply_temperature` and returned HTTP 500 — the same class as #2391, where `NaN <= 0.0` is false and a guard that looks like validation admits it. ## #2310 — wasm32 could not compile `(seed * 6364136223846793005 + ...)` in the SGD epoch shuffle: the literal does not fit `usize` on a 32-bit target, so aprender-core failed to build for wasm32 entirely. ## pmat complexity — the files that taxed every commit The pre-commit hook rejected any commit TOUCHING validate.rs, dispatch_analysis.rs, prometheus_classifier.rs or react_trace_classifier.rs; three commits this week landed with --no-verify carrying before/after tables proving they had not moved the metric. ## One claim corrected rather than merged The producer branch reported "PCA exit 124 @300S -> exit 0 @7s". Independent re-verification measured the DEFAULT `--projection pca` on that model at exit 124, wall 300.33s, no CSV — with an unbounded rerun still going at 2198s. The 7s was the RANDOM projection. Fixing the axes made PCA tractable, not fast: at fixed hidden=1024 the cost is linear in ROWS at ~6.3ms each (5,000 rows 82.6s; 20,000 rows 176.9s), so a 248,320-token vocabulary does not finish in 300s. The book now carries that scaling table, and FALSIFY-LINT-PRODUCER-008's scope says explicitly that RUNTIME is discharged nowhere — every test in the contract uses `--projection random`. Stated rather than gated on purpose: a falsifier for a 26-30 minute run is a 26-30 minute CI test. An unstated performance claim in a book chapter is what this audit exists to stop. Closes #2377 Closes #2375 Closes #2310 Refs #2373
noahgift
enabled auto-merge
August 13, 2026 15:10
CI failed this batch with
FAIL: book/src/cli/beat-run.md does not exist (apr beat-run has no chapter)
`apr beat-run` shipped in #1995. `book/src/cli/beat-run.md` has never existed in
git history. FALSIFY-BOOK-CLI-PARITY-001 has therefore been failing-in-waiting
for months, and nothing noticed, because .github/workflows/book.yml filters on
paths:
- "book/**"
- ".github/workflows/book.yml"
A parity gate asserts that two things agree. Watching only ONE of them means a
command can be added with no chapter and the gate never runs to say so; it took
an unrelated batch that happened to touch book/ to wake it. That is the same
class as the four gates fixed in batch 4 — a check that cannot fail when the
thing it guards actually moves.
Fixes, in order of importance:
1. book.yml now also triggers on the CLI command tree — commands_enum.rs,
extended_commands.rs, tool_commands.rs — so adding a subcommand runs the gate
that checks subcommands.
2. book/src/cli/beat-run.md, written from beat_run.rs rather than from the help
text: the two modes (report vs judge), the exit-code table, and why an
UNJUDGEABLE contract exits non-zero — a beat that cannot decide is not a beat
that passed, which is the whole point of the runner.
3. book/src/lib/datasets.md and book/src/lib/pipeline.md. Fixing (1) revealed
that check_book_lib_parity.sh, the very next step in the same job, was
failing for the identical reason on aprender::datasets and aprender::pipeline
— both missing on main too, both hidden behind the same path filter. Fixing
only the CLI half would have handed the next batch a red build.
The datasets chapter names what is NOT implemented (load_digits,
load_california_housing) rather than implying a completeness the module does not
have.
Verified: check_book_cli_parity 106/106, check_book_lib_parity 71/71, plus
check_book_example_block, check_book_lib_example_block and check_book_linkcheck
all rc=0 — the last of which proves the new cross-references resolve.
Refs #2373
Exit 127 — command not found — the FIRST time this step ever ran.
The `pmat comply gate` step guards `pmat` with `command -v` and a comment saying
it "tolerates absence (pmat may not be available on every runner)", and then one
line EARLIER invokes a bare `pv` with no guard at all, in a step whose entire
purpose is running pv. Nothing caught it because every previous run of this
workflow died at the parity gate above, so execution never reached the line.
That is the third instance of the same class in this one job, all surfaced by
fixing the first: the CLI parity gate that only ran when the book changed, the
lib parity gate failing identically behind it, and now a step that could not run
the tool it exists to run.
`pv` is IN-TREE — crates/aprender-contracts-cli, `[[bin]] name = "pv"` — so the
fix is to build it from the tree rather than depend on an installed copy:
cargo run --quiet -p aprender-contracts-cli --bin pv -- \
validate contracts/apr-book-completeness-v1.yaml
That is the dogfood rule in CLAUDE.md ("pv is THE dogfooded contract CLI"), and
it has the property an installed binary does not: the validator cannot drift from
the contracts it validates, because both come from the commit under test.
Verified locally: rc=0, "0 error(s), 0 warning(s) / Contract is valid."
Refs #2373
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(dogfood): batch 5 — the last three audit findings, and a producer that could fabricate its own verdict
Closes out #2377 and #2375. Four branches, two of which an adversarial pass sent
back before they could merge.
#2377-3 — three lints documented a producer the binary did not have
attn-parity-lint,audio-inspect-lintandembed-viz-linteach name aproducer command in their help text.
apr kernel parity,apr dataset audio-inspectandapr debug embed-vizdid not exist, so the only documentedway to produce the observation they consume was unreachable: their gates had
never run on real data and could not. All three are implemented, wired into the
clap tree and the dispatch arm, with round-trip falsifiers — run the producer,
feed its output to the lint, assert exit 0 — plus the negative, that a corrupted
producer body makes the lint exit non-zero, so the round trip cannot pass
vacuously.
The first attempt shipped the defect this audit exists to eliminate.
Replacing the emitted metrics in
kernel_parity.rswith the literalsmax_abs_diff: 0.0, cosine_sim: 1.0— a producer fabricating perfect parity —left ALL 16 tests green, including the one named
the_parity_metrics_are_not_vacuous, which never went throughrun()at all:it re-measured the values itself in the test body. Nothing asserted that the
EMITTED numbers were the MEASURED ones. Two falsifiers now bind emitted JSON to
measurement, and the 0.0/1.0 mutation turns them red.
And it emitted garbage on the format its own lint targets.
locate_embeddingtook
shape[0]as vocab for every format. GGUF reports GGMLneorder, sotoken_embd.weighton Qwen3.5-0.8B-Q4_K_M is[1024, 248320]— the producerwrote 1024 rows for a 248320-token vocabulary and
embed-viz-lintexited 5 onits own producer's output. The axis rule is now per-format and measured, not
assumed: reading the GGUF payload as
[vocab][hidden]matches the same model'sSafeTensors row at cosine 0.999, transposed at 0.014. The tests missed it because
the only fixture was an APR file — the one layout where the axes coincide.
Note the trap the contract now documents:
token_stris resolved by row index,so it reads correctly no matter which axis produced the coordinates. No assertion
on
token_strcan detect this. The tests assert row count against the vocabularyand coordinates against a hidden-length slice.
#2375 — every streamed chat reply was empty, and /v1/metrics measured nothing
Also remediated after review. A falsifier named
v1_metrics_model_name_is_derived_not_a_constantPASSED on a constant("default"); the derived branch was executed by no test, and it only went red on
revert because the previous constant differed. A test comment and commit message
claimed 0.63.0-era provenance for a code path that did not exist in v0.63.0. And
the temperature fix special-cased only
== 0.0, so negative, NaN and +inf stillreached
apply_temperatureand returned HTTP 500 — the same class as #2391,where
NaN <= 0.0is false and a guard that looks like validation admits it.#2310 — wasm32 could not compile
(seed * 6364136223846793005 + ...)in the SGD epoch shuffle: the literal doesnot fit
usizeon a 32-bit target, so aprender-core failed to build for wasm32entirely.
pmat complexity — the files that taxed every commit
The pre-commit hook rejected any commit TOUCHING validate.rs,
dispatch_analysis.rs, prometheus_classifier.rs or react_trace_classifier.rs;
three commits this week landed with --no-verify carrying before/after tables
proving they had not moved the metric.
One claim corrected rather than merged
The producer branch reported "PCA exit 124 @300S -> exit 0 @7s". Independent
re-verification measured the DEFAULT
--projection pcaon that model at exit124, wall 300.33s, no CSV — with an unbounded rerun still going at 2198s. The 7s
was the RANDOM projection. Fixing the axes made PCA tractable, not fast: at fixed
hidden=1024 the cost is linear in ROWS at ~6.3ms each (5,000 rows 82.6s; 20,000
rows 176.9s), so a 248,320-token vocabulary does not finish in 300s.
The book now carries that scaling table, and FALSIFY-LINT-PRODUCER-008's scope
says explicitly that RUNTIME is discharged nowhere — every test in the contract
uses
--projection random. Stated rather than gated on purpose: a falsifier fora 26-30 minute run is a 26-30 minute CI test. An unstated performance claim in a
book chapter is what this audit exists to stop.
Closes #2377
Closes #2375
Closes #2310
Refs #2373
Verification
cargo test -p apr-cli -p aprender-serve -p aprender-core --lib— 36,801 passed, 0 failed(7,064 + 14,099 + 15,638; these are the only crates this diff touches)
scripts/check_assertions_exclude.sh— 319 sites, delta 0cargo fmt --all -- --check— cleanA full
cargo test --workspace --libwas started twice and killed externallyboth times, at 48,161 and ~0 tests in with 0 failures in each. Rather than
report a number I did not see complete, the figures above are the targeted run
that did finish. CI runs the full workspace on this PR regardless.