feat(scripts): gate the de-slopped surfaces against em-dash regression - #3344
feat(scripts): gate the de-slopped surfaces against em-dash regression#3344kyle-sexton wants to merge 3 commits into
Conversation
The de-slop campaign purges prose surface by surface, and each landed shard is paid for by hand: a mechanical em-dash split changes meaning often enough that every shard so far has needed a rationale-withheld reviewer to catch clauses the automated passes waved through. Nothing then stopped the next contributor from reintroducing one, because no lane enforced the policy. A one-time purge with no gate is a purge that silently rots. Add scripts/check-purged-em-dashes.sh, which enforces zero em dashes on the paths declared in scripts/em-dash-purged-paths.txt, and wire it into the ci hygiene job so its outcome reaches the ci-status aggregate. Enforcement is scoped to an allowlist rather than the corpus. 29,649 em-dash prose lines across 1,074 tracked markdown files remain, so a repo-wide check could only have been merged switched off. Listing what is already clean inverts that: adoption breaks nothing, because every listed path passes the moment it is added, and enforcement grows with the campaign instead of waiting for it. This is the same allowlist-not-blocklist argument, and the same fail-safe direction, that scripts/docs-only-paths.txt already makes. The tracked .claude/ai-slop.json is untouched. Re-enabling rule-em-dash corpus-wide there is a separate decision the campaign has gated on the purge finishing, so the gate instead copies that config into a throwaway layer for its own detector run and removes only rule-em-dash from disabled_rules. Every exclusion the tracked file carries, including the vendor, catalog and eval-fixture paths that hold em dashes as data, keeps applying unchanged. The detector is driven rather than reimplemented: a bare grep would fire inside code fences, inline code spans and ignore-marked regions, which legitimately carry the character. Because that detector exits 0 on every audit path, the gate believes a run only when the rule-em-dash summary is present, reports disabled=0, and accounts for exactly the files handed to it; any other shape is exit 2 rather than a false green. A stale allowlist entry matching no tracked file fails for the same reason. Refs #2891 Co-authored-by: Kyle Sexton <kyle-sexton@users.noreply.github.com>
…right The gate shipped mode 100644 while the workflow invokes it as a bare path, `scripts/check-purged-em-dashes.sh`, exactly as it invokes every sibling scanner. Every one of those siblings is tracked 100755. The hygiene job would therefore have failed on contact with a permission error, and the repository's own unconditional exec-bit scanner would have reported the same defect a second time. Both the gate and its test script are now tracked executable. Add plugins/playwright to the allowlist. Its README and its skills were purged in #3320 and contain no em dash anywhere, so the entry meets the stricter standard the allowlist header declares for a seed entry: the whole list stays auditable by expanding it with --list and grepping the named files for the character. The plugin was clean when the list was seeded and was simply not carried over. Restructure the spaced double hyphen in the workflow comment into a sentence break. The gate's own failure message tells a contributor not to reach for a hyphen when an em dash comes out, and a comment introducing that gate should not model the substitution it warns against. Refs #2891 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gate A fresh-context reviewer audited the gate with the authoring rationale withheld and returned two defects that every automated pass had waved through. Both are fixed here, each with a regression case that fails against the previous code. --list ran the whole detector. The mode changed only what the expansion loop printed; control then fell through to the detector invocation and the check verdict, so a listing cost the same minutes as a full check and returned the check's exit code. That contradicted the allowlist header, which sends a reader to --list precisely to audit the declaration WITHOUT running the detector. It now stops after expansion, still failing on a stale entry, and the real allowlist lists in under a minute instead of nine. The coverage assertion double-counted a declined file. The gate proved the detector had accounted for exactly the files handed to it by adding the two totals in the detector's summary, but those totals overlap: a file the detector opens and then declines on an in-file marker is counted under BOTH "files scanned" and "files declined", while a file excluded by a config glob is never opened and appears only under the latter. An allowlisted path carrying an ai-slop-ignore-file marker therefore failed the run at exit 2, naming a coverage problem that did not exist, for using an exemption the allowlist explicitly promises. Unique files handled is now the scanned count plus the excluded-glob declines alone, read from the decline rows the detector already labels with a cause. Two smaller corrections come with them. A declared path that the tracked detector config excludes is now named in the output instead of being folded into the clean count: such a path sits inside the allowlist and outside enforcement at once, which is the one way this gate can report green over a surface it reads nothing on. And the clean verdict now reports files scanned rather than files declared, so the number means what it says. The workflow comment called an ungated docs-only diff a fail-closed hole when the hazard is the opposite, and the two enumerations of the unconditional whole-repo scanners had not been told about the fourth one. Refs #2891 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 4m 45s —— View job Code review complete
This lane covers correctness/maintainability only ( FindingsPosted one inline finding on Coverage-assertion gap: Other things checked, no issues found
Not independently verified: this environment blocked execution of |
|
Claude finished @kyle-sexton's task in 2m 7s —— View job Security review
No security issues found in this PR. Scope reviewed: the new CI gate ( What I checked for and didn't find:
Nothing here touches authn/authz, secrets, or code execution paths beyond a read-only lint gate over tracked markdown, and the design is consistent with the same-shaped |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1b64959cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| findings="$(grep '^Finding: rule=ai-slop/audit/rule-em-dash ' "$OUT")" | ||
| if [[ -z "$findings" ]]; then | ||
| printf 'check-purged-em-dashes: %s declared paths, %s files scanned, no em dashes.\n' "${#GLOBS[@]}" "$scanned" | ||
| exit 0 |
There was a problem hiding this comment.
When an allowlisted path also matches the tracked config's excluded_paths, the preceding branch reports that the file was not checked but execution continues here; because an excluded file produces no finding, the script prints “no em dashes” and exits 0. This lets CI remain green over an explicitly unenforced purged surface, so excluded > 0 must produce a nonzero verdict rather than falling through to the clean path.
Useful? React with 👍 / 👎.
| if ! jq '.disabled_rules |= ((. // []) | map(select(. != "rule-em-dash")))' \ | ||
| "$SLOP_CONFIG" >"$TMP/root/.claude/ai-slop.json"; then |
There was a problem hiding this comment.
When the tracked config adds an allowlisted file to em_dash_allowed_paths, copying that property unchanged makes the detector suppress em-dash findings for the file. Unlike excluded_paths, the file still contributes to the scanned count and the rule summary still says disabled=0, so an actual em dash passes this gate silently; the derived config must remove these exemptions or reject any overlap with the purged allowlist.
Useful? React with 👍 / 👎.
|
Last security-reviewed head: |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if [[ "$summary" != *" disabled=0"* ]]; then | ||
| echo "check-purged-em-dashes: rule-em-dash was disabled for this run ($summary); the config layer did not take effect" >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| # The detector's two counters OVERLAP, and reading them as disjoint is how this | ||
| # assertion misfires. "files scanned" counts every file the detector opened, | ||
| # which includes one it opened and then declined on an in-file marker; that file | ||
| # is counted a second time under "files declined". Only a file excluded by a | ||
| # config glob is never opened, so only that cause is missing from the scanned | ||
| # count. Unique files handled is therefore the scanned count plus the | ||
| # excluded-glob declines alone, and the detector names the cause on every | ||
| # decline row, so the gate reads those rather than inferring them. | ||
| scanned="$(sed -n 's/^Summary total: [0-9]* findings across \([0-9]*\) files scanned ([0-9]* files declined)$/\1/p' "$OUT")" | ||
| if [[ -z "$scanned" ]]; then | ||
| echo "check-purged-em-dashes: detector emitted no total summary; cannot confirm coverage" >&2 | ||
| exit 2 | ||
| fi | ||
| excluded="$(grep -c '^Declined: file=.* cause=excluded-glob$' "$OUT" || true)" | ||
| handled=$((scanned + excluded)) | ||
| if ((handled != EXPECTED)); then | ||
| echo "check-purged-em-dashes: detector handled $handled files but $EXPECTED were declared; coverage is not what the allowlist claims" >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| # --- Verdict ---------------------------------------------------------------- | ||
|
|
||
| # A declared path that the tracked config excludes is reported rather than | ||
| # folded into the clean count. Such a path is inside the allowlist and outside | ||
| # enforcement at the same time, which is the one way this gate can be green over | ||
| # a surface it is checking nothing on, and a silent count would hide it. | ||
| if ((excluded > 0)); then | ||
| echo "check-purged-em-dashes: $excluded declared file(s) are excluded by $SLOP_CONFIG and were NOT checked:" >&2 | ||
| sed -n 's/^Declined: file=\([^ ]*\) cause=excluded-glob$/ \1/p' "$OUT" >&2 | ||
| echo "check-purged-em-dashes: remove them from $ALLOWLIST or from the config's excluded_paths; declaring a path the detector never reads enforces nothing" >&2 | ||
| fi |
There was a problem hiding this comment.
The coverage assertion (lines 213-236) proves the detector opened every declared file and reports the excluded_paths blind spot explicitly (244-248, added in a1b6495 for exactly this class of problem), but it doesn't cover the sibling exemption mechanism the tracked config carries for this same rule: em_dash_allowed_paths.
In detect.sh (unchanged by this PR), a file matching em_dash_allowed_paths is skipped for rule-em-dash specifically (plugins/ai-slop/skills/audit/scripts/detect.sh:514-518) — but unlike the excluded-glob case, this decline is not logged as a per-file Declined: row. It only nudges the rule's aggregate declined= count in the Summary rule=... line, which this gate never reads. So:
scannedstill counts the file (it was opened).excludeddoesn't count it (it's not anexcluded-globdecline).handled == EXPECTEDstill holds.- No findings are possible for that file, and nothing here says why.
If .claude/ai-slop.json ever gains an em_dash_allowed_paths entry that overlaps a path in scripts/em-dash-purged-paths.txt (today's tracked config has none, so this is currently unreachable), the gate would report that surface clean while silently not checking it for the one rule it exists to enforce — the exact "green over a surface it is checking nothing on" failure mode the excluded_paths warning above was written to catch, just not extended to this path.
Consider asserting disabled= and the em-dash-allowed decline count together (the rule summary line already carries declined=N; a nonzero delta beyond what excluded accounts for would indicate an em-dash-allowed-glob decline) and warning on it the same way, or at minimum a code comment noting the gap is accepted rather than overlooked.
|
Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
1 participant
No linked issue
Summary
The de-slop campaign (#2891) has rewritten prose surface by surface for a week, and each shard is paid for by hand: a mechanical em-dash split changes meaning often enough that every shard so far has needed a rationale-withheld reviewer to catch clauses the automated passes waved through. Nothing then stopped the next contributor from reintroducing one, because no lane enforced the policy. This adds the ratchet, so that what the campaign has already cleaned stays clean.
scripts/check-purged-em-dashes.shenforces zero em dashes on the paths declared inscripts/em-dash-purged-paths.txt, and runs unconditionally in the hygiene job.Enforcement is scoped to an allowlist rather than the corpus, and that is the design rather than a concession. Roughly 29,600 em-dash prose lines across 1,074 tracked markdown files remain, so a repo-wide check could only have been merged switched off. Declaring what is already clean inverts the problem: adoption breaks nothing, because every listed path passes the moment it is added, and enforcement grows with the campaign instead of waiting for it. It is the same positive-list shape, and the same fail-safe direction, as
scripts/docs-only-paths.txt.The allowlist currently covers ten plugins. Sixty-nine have had instruction surfaces rewritten by campaign shards, so the gate defends a fraction of what has been cleaned; the backfill and what blocks it are tracked in #3342.
Fix
scripts/check-purged-em-dashes.shdrivesplugins/ai-slop/skills/audit/scripts/detect.shrather than grepping, so fenced code blocks, inline code spans, and ignore-marked regions keep carrying the character as data. The tracked.claude/ai-slop.jsonis not modified; the gate copies it into a throwaway layer for its own run and removes onlyrule-em-dashfromdisabled_rules, so every exclusion that file carries, including the vendor, catalog, and eval-fixture paths that hold em dashes deliberately, keeps applying.HOMEis emptied so a contributor's user-global config cannot move the verdict.rule-em-dashsummary is present, reportsdisabled=0, and accounts for exactly the files handed to it. Any other shape is exit 2 rather than a false green, and a stale allowlist entry matching no tracked file fails for the same reason.scripts/em-dash-purged-paths.txtseeds the declaration. Every entry meets a stricter standard than the gate enforces, containing no em dash anywhere rather than none in prose, so the whole list can be audited by expanding it with--listand grepping the named files..github/workflows/ci.ymlruns the self-test first and ungated, so a broken gate cannot mask a regression, then the gate itself, whose outcome reaches theci-statusaggregate throughaggregate-hygiene-results.sh. The step is deliberately not gated onrun_full: it reads markdown prose, which is exactly what a docs-only diff changes.Verification
An independent fresh-context reviewer audited the change with the authoring rationale withheld and returned two defects, both fixed in
a1b64959cwith regression cases that fail against the previous code:--listran the whole detector rather than stopping at expansion, contradicting the allowlist header's own audit instructions; and the coverage assertion added two overlapping detector totals, so an allowlisted file carrying anai-slop-ignore-filemarker failed the run at exit 2 for using an exemption the allowlist promises.plugins/naming/README.md): exit 1, naming file and line. Reverted: exit 0.scripts/check-purged-em-dashes.test.sh:PASS=17 FAIL=0. The suite drives the real detector rather than a stub, plants a violation and requires exit 1, and separately plants em dashes in a code fence and an ignore-marked line and requires silence.scripts/check-purged-em-dashes.sh: 21 declared paths, 30 files scanned, exit 0.scripts/check-purged-em-dashes.sh --list: 55 seconds, down from roughly nine minutes.git ls-tree HEADconfirms both scripts tracked100755; the workflow invokes the gate as a bare path.shellcheck -x,shfmt -d -i 2, andactionlintclean.check-changelog-parity.sh --check-bump origin/main,check-silent-skips.sh,check-discriminating-test-skips.sh,check-orphaned-fixtures.sh, andcheck-docs-only-gate.sh --checkall pass. No plugin version changes, so no changelog entry is due.Related
Refs #2891. This shard does not close it. The campaign's second checkbox, carrying the purge through the remaining corpus, is nowhere near done, and its fourth, tightening
.claude/ai-slop.jsononce the purge completes, is explicitly gated on that. A gate over already-cleaned paths is the ratchet the campaign was missing, not its completion.