Doc example drift - #314
Merged
Merged
Conversation
All three `=== "Python"` tabs in docs/ implement _run() by printing rather than returning, so each exits 1 with a specification_error under the 2.0.0 contract. The leaked print still reaches the terminal, so the documented output appears above the error — which is why three broken examples shipped in 2.0.0 and survived a release. Root cause is scope: the 2.0.0 design enumerated affected docs by hand and scoped quick-start.md to "update the Java example", since Java's void _run() could not compile. index.md and examples.md were never listed. Adds #181 (the doc errors) and #182 (nothing executes doc examples, and ci.yml skips CI for docs-only PRs), with roadmap entries and a design for fixture-backed doc examples verified by a behavior tier plus a byte-identity check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven tasks. Tasks 1-3 land a fixture and behaviour test per page against the broken example first, confirm it fails for the documented reason, then fix page and fixture together. Task 4 adds the identity check, Task 5 the prose fix and CI wiring, Task 6 red-proofs both halves independently. Also refines the design's CI section: docs-tests.yml alone would not fire for a src/ change that breaks a working example, which is the 2.0.0 scenario itself, so ci.yml gets a docs job too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…nd fixtures Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
plcc-scan and plcc-parse exit 0 against a spec with a broken _run(): they stop at --through=scan and --through=parse and never build the interpreter. Only plcc-rep reaches semantics, so the rep test's exit-status assertion is the sole detector for this class of bug. Verified by running the broken quick-start spec through all three commands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Python tab printed from _run() instead of returning, so it exited 1 with a specification_error under the 2.0.0 contract. The leaked print put the documented output on stdout anyway, which is why it went unnoticed. Adds bin/test/docs.bash and tests/bats/docs/, which assert exit status as well as output — the output comparison alone passes for the broken example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hon tab Same pre-2.0.0 print idiom as the quick start: _run() printed instead of returning, so the example exited 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_run() printed an int instead of returning a string. Rewriting print(x) to return x is not sufficient here — eval() returns an int, which is also a specification_error. The fix wraps it in str(), mirroring the Java tab's String.valueOf(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bats tier proves the fixtures work; this proves the pages still contain what the fixtures ran. Drift in either direction now fails with a unified diff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The design mandates documenting both convention deviations (tests/docs/ outside src/, and its deliberate double-collection). Task 5's CONTRIBUTING text omitted them, which would leave the docstring in tests/docs/example_block_test.py pointing at a CONTRIBUTING that documents only the rule it breaks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
semantic.md described the pre-2.0.0 behaviour ("the default
implementation prints...") and never stated that _run returns a string,
despite being the language guide's conceptual home for it.
Two CI jobs, because a doc example breaks from two directions: a
docs-only PR (which ci.yml skips) and a src/ change (which the doc-path
trigger misses).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e instances
A reviewer caught this in Task 5's net-new semantic.md text; verified against
the running CLI. A stray print from _run() IS echoed under
--verbose-format=json — rep.py prints any non-JSON line unconditionally,
before verbose_format is consulted. The real failure mode is stream
corruption, not silent omission.
The identical wrong claim is already shipped in languages/{python,javascript,
java}.md and cli/guide/language-extensions.md, so Step 1b corrects all four.
The docs tier cannot catch a wrong prose claim; noted as a design limitation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A stray print from _run() is not silently dropped under
--verbose-format=json: rep.py echoes any non-JSON line unconditionally,
before verbose_format is consulted. The real failure mode is stream
corruption (a non-JSON line among the JSON records), not omission.
Fixes the net-new sentence in semantic.md (Task 5) plus the identical
pre-existing claim already shipped in languages/{python,javascript,java}.md
and cli/guide/language-extensions.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Task 5 added a fifth sub-script to functional.bash without updating tests/bats/commands/test-scripts-path-filter.bats, which stubs each sub-script — so the no-argument test now dies on a missing docs.bash. A plan defect: the plan changed functional.bash without checking its coverage. Task 8 also adds the per-tier routing test the new arm lacked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
functional.bash gained a fifth sub-script and a fifth routing arm, but its stub tree still stubbed only four, so the no-argument test died on a missing docs.bash (exit 127). Adds docs to the stub tree, corrects the test's name and assertions from four to five, and adds the per-tier routing test the new arm was missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rict gate mkdocs build --strict fails on three nav entries left over from #113's diagram-command rename. Pre-existing on main and unrelated to this branch's work, but it blocks Task 7's verification, and the reason it survived is this branch's own subject: nothing in CI runs mkdocs build --strict. Adds Task 9 to the plan: delete the orphans and add the gate to docs-tests.yml, which already triggers on mkdocs.yml and docs/**. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mkdocs.yml still listed three command pages that #113's rename removed. The renamed pages were already in the nav, so the entries were pure orphans putting three dead links in the published site. It survived because nothing in CI runs `mkdocs build --strict`: docs.yml publishes via mike (no --strict) and only after merge, and ci.yml skips docs-only PRs. Adds the gate to docs-tests.yml, which already triggers on mkdocs.yml and docs/**, so a broken link fails the PR that introduces it. Closes #183. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
bin/issues/new.bash incremented .next-id.txt when issue 183 was created, but the file was left out of that commit. Without it a fresh checkout would hand out 183 a second time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Issue 183 was filed and fixed on this branch by Task 9, so it closes with 181 and 182. Also corrects the expected check.bash line: seven issues are open before the step and .next-id.txt advanced to 184. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…roadmap Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eteness The identity check pinned only spec.plcc to its page tab, so the expected-* files were pinned to nothing: a src/ change that altered plcc-scan's output format could be absorbed by updating the fixtures, leaving every page showing output the tool no longer produces with every tier green. MANIFEST now declares each fixture's spec tab and its documented outputs, and extract_output_block() pulls the first unindented ```text fence out of the named section. Unindented is the discriminator: a tab's fence is indented four spaces, so an output lookup can never return a specification. Two completeness gates close the hand-enumeration gap that MANIFEST otherwise reintroduces: - Every runnable specification in docs/ (every %%%-bearing fence) must be covered by MANIFEST or named in UNCOVERED with a reason, so a new example cannot land with zero coverage. - Every expected-* file in a fixture must be pinned to a heading, or declared unpinned with the reason the page cannot show it. read_text() calls now pass encoding="utf-8"; docs/language-guide/index.md holds non-ASCII, which a POSIX-locale runner would fail on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three places stated that an output-only comparison passes for a broken Python example. It does not: plcc-rep writes its specification error to stdout, not stderr (src/plcc/cmd/output.py implements print_user_error as a bare print), so the broken example's output is the documented text plus two error lines and exact equality fails. The exit-status assertion is cheap defense in depth, not the sole detector. Corrected in CONTRIBUTING.md, the design's Part 3, and the plan's Task 1 Step 4, Task 5 Step 8, and Task 6 Step 2. Also: - "Every runnable specification in docs/ has a fixture" was false — six of nine %%%-bearing pages are uncovered. CONTRIBUTING now names the six covered fixtures and the four uncovered quick-reference specs, and points at the UNCOVERED allowlist that keeps the gap honest. - The design claimed the tier is wired into bin/test/all.bash. It is not, and should not be: all.bash calls functional.bash, which chains it. - The design claimed neither check requires remembering which docs a change affects. That was false for new examples until the completeness guard landed; it now describes the guard instead. - CONTRIBUTING's narrow-and-rerun paragraph omitted bin/test/docs.bash, which accepts an optional path like the other three. The plan's quoted commit message in Task 1 Step 8 keeps the false claim, because it accurately quotes a commit that was already made. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Build docs strictly step ran `pdm run mkdocs build --strict` against mkdocs.yml, which loads the kroki plugin. kroki POSTs the three plantuml fences to kroki.io and reports a fetch failure through mkdocs' plugin logger, which --strict promotes to a build abort — so a kroki outage or rate-limit would turn every docs PR red for a reason unrelated to the change. A required check that fails at random trains people to ignore it. mkdocs-strict.yml INHERITs mkdocs.yml with kroki dropped. What this gate checks is nav and link integrity, which is what #183 was about and needs no network; the deployed site still builds from mkdocs.yml with kroki. bin/docs/build.bash runs the strict build, so the check has a local entry point — #183 existed precisely because nobody ran it locally. The workflow step now calls the script, which also drops its hidden dependency on the previous step having run `pdm install`. Verified: exit 0 today, and exit 1 with the expected warning when an orphan nav entry is reintroduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…berate bats' default `run` merges stderr into stdout, so the "$output" comparisons assert the command emits nothing beyond the documented text on either stream. That is stricter than the page's claim and worth keeping, but a future benign stderr line would fail with a message about "the documented output" and point the reader at the page rather than at the new stderr text. Say so in each file's header. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Steps to Reproduce quoted "The following pages exist in the docs directory, but are not included in the 'nav' configuration:". mkdocs never emitted that here — it is the message for orphan *files*, and deleting nav entries could not have fixed it. Replaced with the three warnings mkdocs actually prints, captured by reintroducing the orphan entries and rerunning the strict build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The paths filter listed mkdocs.yml and bin/test/docs.bash but not mkdocs-strict.yml or bin/docs/build.bash, so a PR editing either of the strict gate's own inputs — including one that broke them — skipped the workflow that consumes them. 'docs/**' does not match 'bin/docs/'. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seven observations from the #181/#182/#183 reviews were judged non-blocking and left unfixed, but were recorded only in a git-ignored scratch ledger that this workflow deletes when it finishes. They would not have survived. Captured here with a roadmap entry so they live in the repo, per the issue conventions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a “docs examples” testing tier to prevent runnable documentation specs and their promised outputs from drifting from what CI actually runs, while also fixing stale _run() examples that still used the pre-2.0.0 “print” behavior.
Changes:
- Fix documented
_run()examples to return a string (not print), matching the runtime contract. - Add fixtures + bats tests to execute doc examples via the real CLI and assert exact output + exit status.
- Add a pytest identity check to keep docs’ fenced blocks and output blocks byte-identical to the fixtures, and wire this tier into local runners + CI; add a strict MkDocs build gate.
Reviewed changes
Copilot reviewed 52 out of 56 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/fixtures/docs/quick-start-python/spec.plcc | Fixture spec for quick-start Python tab (runnable copy of docs block). |
| tests/fixtures/docs/quick-start-python/input | Fixture input for quick-start Python example. |
| tests/fixtures/docs/quick-start-python/expected-scan | Pinned expected scan output for quick-start Python. |
| tests/fixtures/docs/quick-start-python/expected-rep | Pinned expected runtime output for quick-start Python. |
| tests/fixtures/docs/quick-start-python/expected-parse | Pinned expected parse output for quick-start Python. |
| tests/fixtures/docs/quick-start-java/spec.plcc | Fixture spec for quick-start Java tab. |
| tests/fixtures/docs/quick-start-java/input | Fixture input for quick-start Java example. |
| tests/fixtures/docs/quick-start-java/expected-scan | Pinned expected scan output for quick-start Java. |
| tests/fixtures/docs/quick-start-java/expected-rep | Pinned expected runtime output for quick-start Java. |
| tests/fixtures/docs/quick-start-java/expected-parse | Pinned expected parse output for quick-start Java. |
| tests/fixtures/docs/lang-guide-index-python/spec.plcc | Fixture spec for language-guide overview Python tab. |
| tests/fixtures/docs/lang-guide-index-python/input | Fixture input for language-guide overview Python example. |
| tests/fixtures/docs/lang-guide-index-python/expected-rep | Pinned expected runtime output for language-guide overview Python. |
| tests/fixtures/docs/lang-guide-index-java/spec.plcc | Fixture spec for language-guide overview Java tab. |
| tests/fixtures/docs/lang-guide-index-java/input | Fixture input for language-guide overview Java example. |
| tests/fixtures/docs/lang-guide-index-java/expected-rep | Pinned expected runtime output for language-guide overview Java. |
| tests/fixtures/docs/lang-guide-examples-python/spec.plcc | Fixture spec for language-guide examples (subtraction) Python tab. |
| tests/fixtures/docs/lang-guide-examples-python/input | Fixture input (“samples”) for subtraction example. |
| tests/fixtures/docs/lang-guide-examples-python/expected-scan | Pinned expected scan output for subtraction example (Python). |
| tests/fixtures/docs/lang-guide-examples-python/expected-rep | Pinned expected runtime output for subtraction example (Python). |
| tests/fixtures/docs/lang-guide-examples-python/expected-parse | Pinned expected parse output for subtraction example (Python). |
| tests/fixtures/docs/lang-guide-examples-java/spec.plcc | Fixture spec for subtraction example (Java tab). |
| tests/fixtures/docs/lang-guide-examples-java/input | Fixture input (“samples”) for subtraction example (Java). |
| tests/fixtures/docs/lang-guide-examples-java/expected-scan | Pinned expected scan output for subtraction example (Java). |
| tests/fixtures/docs/lang-guide-examples-java/expected-rep | Pinned expected runtime output for subtraction example (Java). |
| tests/fixtures/docs/lang-guide-examples-java/expected-parse | Pinned expected parse output for subtraction example (Java). |
| tests/docs/example_block_test.py | Pytest identity + coverage/allowlist checks for runnable specs and documented outputs. |
| tests/bats/docs/quick-start.bats | Bats behavior tests for quick-start examples (scan/parse/rep). |
| tests/bats/docs/language-guide-index.bats | Bats behavior tests for language-guide overview examples. |
| tests/bats/docs/language-guide-examples.bats | Bats behavior tests for subtraction examples (scan/parse/rep). |
| tests/bats/commands/test-scripts-path-filter.bats | Updates routing tests to include the new docs tier. |
| mkdocs.yml | Removes orphan nav entries that break strict builds. |
| mkdocs-strict.yml | Adds a “strict” MkDocs config inheriting from mkdocs.yml but dropping kroki for network-free strict gating. |
| docs/quick-start.md | Fixes Python _run() example to return a string. |
| docs/language-guide/semantic.md | Updates semantic guide to state the _run() return contract and stdout caveats accurately. |
| docs/language-guide/languages/python.md | Clarifies stdout leakage behavior across verbose formats. |
| docs/language-guide/languages/javascript.md | Clarifies stdout leakage behavior across verbose formats. |
| docs/language-guide/languages/java.md | Clarifies stdout leakage behavior across verbose formats. |
| docs/language-guide/index.md | Fixes Python _run() example to return a string. |
| docs/language-guide/examples.md | Fixes Python _run() example to return a string (via str(...)). |
| docs/cli/guide/language-extensions.md | Clarifies stdout leakage behavior in the CLI extension contract. |
| dev-docs/specs/2026-08-03-doc-example-drift-design.md | Design doc for the new docs example drift prevention approach. |
| dev-docs/roadmap.md | Roadmap entry for follow-ups stemming from the docs tier work. |
| dev-docs/plans/2026-08-03-181-182-doc-example-drift.md | Implementation plan capturing the full approach and rationale. |
| dev-docs/issues/done/183-docs-orphan-plantuml-nav-entries.md | Issue record documenting the orphan MkDocs nav entries and strict gate fix. |
| dev-docs/issues/done/182-test-doc-example-drift.md | Issue record documenting why doc examples drifted and the testing response. |
| dev-docs/issues/done/181-docs-run-contract-stale-python-examples.md | Issue record documenting the stale _run() examples and their failure mode. |
| dev-docs/issues/184-docs-tier-followups.md | Captures deferred follow-ups and diagnostics improvements for the docs tier. |
| dev-docs/issues/.next-id.txt | Advances next issue ID after creating new issue(s). |
| CONTRIBUTING.md | Documents the docs tier, strict docs build command, and the fixture/identity convention. |
| bin/test/functional.bash | Wires the docs tier into aggregate test running + path routing. |
| bin/test/docs.bash | Adds a dedicated docs tier runner (pytest identity + bats behavior). |
| bin/docs/build.bash | Adds a strict, network-free docs build gate using mkdocs-strict.yml. |
| .gitignore | Ignores .superpowers/ scratch directory. |
| .github/workflows/docs-tests.yml | Adds docs-only PR CI workflow running docs tier + strict docs build. |
| .github/workflows/ci.yml | Adds docs tier job to main CI to catch src-caused doc drift. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+159
to
+163
| complete and copy-and-run, so all able to drift unnoticed. The `UNCOVERED` | ||
| allowlist in `tests/docs/example_block_test.py` names every uncovered | ||
| specification with the reason, and a test fails if a page grows a runnable | ||
| specification that is in neither the `MANIFEST` nor that allowlist. New examples | ||
| therefore cannot land silently unguarded. |
…ption
The allowlist skipped every %%% fence on a listed page, so a new runnable
specification added to an already-exempt page — a second one in
language-guide/languages/python.md, say — landed with zero coverage and no
failure. CONTRIBUTING claimed the opposite ("New examples therefore cannot
land silently unguarded"), which was the same overstatement this branch
exists to remove.
UNCOVERED now maps each page to how many such fences it holds, so an
exemption covers the fences that existed when it was written rather than
the page forever. Verified by adding a fence to an allowlisted page: the
guard fails with "UNCOVERED says 4, found 5 at [26, 94, 113, 151, 215]".
Corrects the CONTRIBUTING and design wording to match, and drops the item
from #184 since it is no longer a follow-up.
Raised by a Copilot review of the PR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The authors of this PR...