context-budget 0.6.10 — post-use audit: additivity claim, discarded caveats, hook scope, and a measured deny counterexample

[kyle-sexton]

Plugin: context-budget@melodic-software v0.6.10 · Measured against: Claude Code 2.1.241,
C:\Users\KyleSexton\.local\bin\claude.exe · Mode: sdk (exact), headless · Platform:
Windows 11, cwd $HOME (nonrepo state key) · Agent SDK 0.3.241

Audited after a real end-to-end run: /context-budget:setup check + apply, then
/context-budget:audit --full-sweep (27 tools + additivity), then an applied lever with a
before/after ledger row. Three components in scope: the audit skill, the setup skill, and the
settings-write-ask PreToolUse hook.

Severities are the auditors' own, uncalibrated (no human calibration was applied).

---

HIGH

H1 — --verify-additivity cannot detect non-additivity on a full sweep

combinedSaved for the observed run was 24,197 — exactly 10854 + 13343, the entire attributed
payload. systemBucketSaving (measure.mjs:610-614) sums both bucket deltas, and a combined deny
drives both to zero, so on any full sweep the check degenerates to sumOfParts === baselineTotal.
It reports additive: false whether or not anything is wrong.

Reproduced: denying only the 11 prefix savers leaves 13 live tools including ToolSearch (worth
+15,824 prefix when denied) and the SDK still omits the bucket, which the engine records as a
measured 0.

Fix: add a saturation guard — additive: null with a reason when either bucket is a synthesized
zero, plus an epsilon. Report additivity per bucket; the data is already on every perTool row.

H2 — SKILL.md's "deltas add" is false as written

SKILL.md:18 claims A/B differencing "is compositional (deltas add), so a basket of trims can be
priced from its members", unconditionally. Non-additivity is real, established from three clean
non-saturated readings: baseline 10,557; denying the 10 prefix savers other than PowerShell →
2,399 (−8,158, shortfall exactly zero); PowerShell solo → −3,355. PowerShell stays live in the
10-deny run, so its marginal contribution is bounded by the 2,399 remaining while its solo value is
3,355 — at least 956 tokens double-counted, with no synthesized zero anywhere in that argument.

Deferred-side additivity, by contrast, is exact: 12 deferred savers summed to 13,343 (the bucket, to
the token), and a 3-tool basket predicted 4,608 and measured 4,608.

The skill-frontmatter-subtraction hypothesis is dead: compareSnapshots requires both sigMatch
and skillTokensMatch (measure.mjs:551-558, 588), so a constant subtraction cancels in a delta;
independently confirmed 1:1 — a 297-token skill moved Skills +297 and System tools −297.

Fix: qualify the claim per bucket in both homes — SKILL.md:18 and
levers.json → deny-bare-tool.categoryBasis (SSOT duplication). Ship with H1 or it is cosmetic.

H3 — measure.mjs:723 discards every deny run's caveats

attribute returns caveats: baseline.caveats, so caveats raised by any deny run — including the
synthesized-zero disclosure promised at engine.md:76-79 — never reach the record. This is what
made H1 shippable: the instrument's own disclosure channel is severed.

Fix: merge deny-run caveats into the attribution record. Smallest diff, highest value in this
report — do this one first regardless of the rest.

H4 — measured counterexample to deny-bare-tool.mechanism (possibly UPSTREAM)

With a bare-name permissions.deny entry for EndConversation in force, the tool remained in
the interactive session's deferred registry, ToolSearch select:EndConversation returned its full
schema, and the call succeeded (the session ended). Verified verbatim against the permissions page
(WebFetch + curl, 498,122 bytes, 2026-08-24): "Bare-name deny rules like Bash remove the tool
from Claude's context before this evaluation begins", and the options table's "The Bash tool
definition is removed from the request. Claude does not see the tool and cannot attempt it." No
carve-out for any tool appears there.

The same mechanism demonstrably works on this machine for other deferred tools: 27 → 24 live,
deferred bucket −4,608, exact. So this is tool-specific, not general breakage.

The row's recheckTrigger is structurally blind to it. It reads "a measured bare-name deny stops
moving the System tools buckets", but EndConversation never enters either bucket headless — the
trigger could never fire for this case. That is a defect in the trigger's design, not a missing row.

⚠️ Likely upstream. The plugin cites the docs accurately; the docs and the binary disagree.
Please do not "fix" a correct citation. Consider escalating to Anthropic.

Fix: add EndConversation to interactive-only-tools.json; record the deny-no-op as a measured
counterexample; widen deny-bare-tool.recheckTrigger to cover tools appearing in neither attributed
bucket.

H5 — apply's scope guidance is refuted on both branches

setup's apply says to "pass the scope claude plugin list reports for this plugin … or the write
lands at a scope that does not load." Both branches fail: project-scope pluginConfigs is ignored
outright per docs, or the write goes to user settings regardless and the -s/cwd ceremony is
inert. It also omits that -s governs enabledPlugins placement, so following it can install the
plugin at a new scope and create a stray <cwd>/.claude/settings.json.

Observed trigger: with cwd == $HOME, claude plugin list reported this plugin twice (user
and project) because project scope resolves to <cwd>/.claude/settings.json — byte-identical to
the user path. One file, two rows. The auditing session initially acted on that and gave the operator
wrong advice.

Fix: replace with doc-grounded facts — default user; -s governs enablement placement;
plugin list can show one file under two labels. Resolve the disjunction before finalizing wording.

H6 — check step 1 cannot detect the failure its own rationale describes

Step 1 argues the hook registers in exec form ("command": "node"), so a bare node must resolve
in the hook's environment, and that an unresolvable node means the checkpoint silently never
fires while the option still reads true. But the prescribed probe is command -v node, which under
fnm returns a per-call ephemeral multishell path (AppData\Local\fnm_multishells\<pid>_<ts>\node
— three probes in one session returned PIDs 64452, 64196, 33540). That is neither the hook's
environment nor the persisted PATH. The check PASSes and stops.

The Windows caveat that would catch this ("confirm the hook's environment resolves the same node")
is filed under the Missing node bullet — reachable only on FAIL. The failure it describes is a
PASS-path failure.

Fix: on Windows, resolve against persisted Machine+User PATH, report in-process resolution
separately, flag PID-bearing shim paths as ephemeral, FAIL when only an ephemeral hit exists.
Ambitious version: a check --deep that fires the PreToolUse path against a throwaway
settings-shaped path and confirms the ask actually arrives.

---

MEDIUM

M1 — hook matcher cannot see shell writes, but the claim says "any settings-surface write"

Matcher is Write|Edit|MultiEdit|NotebookEdit. A PowerShell Set-Content or a Bash heredoc to
~/.claude/settings.json bypasses the checkpoint entirely, so SKILL.md:224's "any settings-surface
write" and the header's "the ordinary auto-mode path cannot rewrite settings silently" are both
overclaimed. The bypass enumeration names two exotic escapes (PermissionRequest, disableAllHooks)
and omits the reachable one.

Found independently by two auditors, in separate packets. Corroborated in-session: an inline
node -e write to a settings-composing file was stopped by the auto-mode classifier, while the
Edit path — the one the matcher covers — went through.

Fix: extend the matcher to Bash|PowerShell, or keep it and reword the claim. Do not ship
the current claim with the current matcher. (Extending has real false-positive cost; honest scoping
is likely the better fix.)

M2 — the hook does not cover render-into-settings sources (chezmoi class)

The regex (settings-write-ask.mjs:45-46) matches only .claude/settings.json,
.claude/settings.local.json, and managed-settings.json. Four Edit calls landed on
~/.local/share/chezmoi/.chezmoidata/claude.json — which provably composes the deployed
~/.claude/settings.json (chezmoi source-path → dot_claude/modify_settings.json, which consumes
it) — with no ask.

Reranked up on evidence: .claude is a protected directory and .mcp.json/.claude.json are
protected files, so on most paths this hook matches it largely duplicates harness protection. The
in-cwd non-protected file (this class) is the one write path with no harness review at all.
managed-settings.json is on neither protected list and is the hook's only genuinely non-redundant
coverage.

Fix: at minimum document the scope boundary. A general solution is hard (arbitrary templating),
but the class is real and recognizable.

M3 — disable-artifact is offerable but structurally unmeasurable

The row is removes-weight / recommendable-on-fit, yet its own measurement field prescribes
attribute --tools Artifact, and Artifact is in interactive-only-tools.json — never a headless
candidate. Both halves of the route are dead inside the skill's declared scope. Applied live here,
the Skills delta was 0 because the artifact-* skills never enter a headless listing.

knownUncovered protects tools from exactly this misreading; there is no lever-side
equivalent. report.md's "Zeros are findings" then requires publishing the zero — so the contract
actively causes a working lever to be reported as inert.

Fix: add a lever-side unmeasurable-in-this-session-kind marker plus a report.md rule
rendering such levers in their own group rather than as zeros.

M4 — knownUncoveredRecord attributes operator-caused absence to product structure

It computes listing.tools − candidates (measure.mjs:634-641), so AskUserQuestion,
EnterPlanMode, and ExitPlanMode — absent because the operator bare-name-denied them — are
reported under "interactive-only — structurally unreachable."

Fix: subtract known deny entries before attributing absence to interactive-only status.

M5 — works-but-saves-nothing-here asserts a consumer-specific fact statically

skillOverrides / disable-bundled-skills are stored with that category, conditional on cap
saturation. Measured here the cap was slack (13/13 included, 0 collapsed, 2,111 tokens against a
~30,000 budget at skillListingBudgetFraction: 0.03), so the condition resolves the opposite way and
the levers would genuinely remove weight. Separately the saturation signal those rows name is absent
from the record and unit-mismatched (tokens vs characters).

Fix: split into stored mechanism + runtime-resolved saving; note the token-vs-character unit
change wherever the listing budget appears.

M6 — meta.purpose claims completeness that verify-catalogue cannot check

The catalogue claims to hold "every known operator-controllable switch", but verify-catalogue is
one-directional: it confirms stored tokens exist in the binary and cannot see existing-but-unstored
switches. Binary scan of 2.1.241 found CLAUDE_CODE_DISABLE_CRON and CLAUDE_CODE_ENABLE_DESIGN_SYNC
with no rows. (RemoteTrigger / PushNotification genuinely have no native switch — same scan — so
deny is correct for those.)

Fix: add --find-unstored to verify-catalogue; ENV_TOKEN_RE is already the right regex.

M7 — SDK data is fetched and thrown away

The SDK returns per-skill tokens (summing exactly to the Skills bucket) and slashCommands;
measure.mjs:396 discards both. Fix: keep skillFrontmatter[].tokens + pluginName; record
slashCommands in the snapshot.

M8 — a headless artifact is attributed to bypassPermissions

SKILL.md stamps the hook's block behavior as measured at v2.1.232 under bypassPermissions. The
measurement is not wrong; the attribution is imprecise. Re-verified this audit at 2.1.241 across
three headless probes — default, auto, and bypassPermissions — all fired and blocked, no file
created. Fix: re-stamp to 2.1.241 and scope the claim to what was actually exercised.

M9 — custom-output-style detection has no built-in-style branch

Detection reads "keep-coding-instructions in the style's frontmatter", but a built-in style has no
frontmatter file. Confirmed live: outputStyle: "Concise" set, ~/.claude/output-styles/ absent.

M10 — apply prints a write-causing command without the never-run marker

The network-only npm install carries an explicit "operator's to run" marker; the
claude plugin install … --config command — which writes pluginConfigs — does not. An agent
executing apply can therefore cause exactly the write the skill's own contract forbids.

---

LOW

• L1 — nonrepo cwd additionally drops project settings, .mcp.json, and CLAUDE.md discovery;
  nothing states the reported total is a floor. State it in report.md §1.
• L2 — --full-sweep (argument-hint) vs --tools from-baseline (engine flag); disclosed at
  SKILL.md:101, cosmetic.
• L3 — the "uninstall drops the entire stored pluginConfigs entry" claim carries no version
  stamp while the --config claim two sentences earlier does — inconsistent evidence discipline
  inside one paragraph.
• L4 — step 4's template renders "Report the effective true": the sentence has no noun.
  One-word fix, not a malformed template.
• L5 — the printed npm remediation is POSIX shell handed to a Windows operator to run themselves.
• L6 — the skill names an SSOT then restates all three prerequisites inline anyway (drift
  surface).

---

OPEN QUESTION — not a finding, and no remediation depends on it

During this session, four settings-composing Edit calls and one probe write produced no ask,
in an interactive auto-mode session. In fresh headless sessions the hook fires and blocks in all
three permission modes tested. Two hypotheses remain, and nothing runnable from here separates them:

• H-1a — post-/reload-plugins --force session state. That reload reported 1 hook reloaded and
  1 error during load (unattributed; /plugin is operator-only). Of the two enabled plugins only
  context-budget ships a hook, so 1 hook can only be this one. The anomalous counter is
  0 skills against 4.
• H-1b — an interactive dispatch gap. hooks.md L1732 documents that a hook's "ask" forces a
  prompt in auto mode, and records a prior bug of this exact shape (pre-2.1.211, Bash only, not
  Write). If this reproduces, it is an upstream Claude Code defect, not a plugin one.

An auto-mode-only gap (H-1c) was tested and disconfirmed: headless auto mode fires normally.

Closing test: a fresh interactive session in auto mode writing to any
managed-settings.json outside .claude/. Manual mode no longer discriminates.

---

What worked (recorded so it is not regressed)

The engine refused a number for Skill and published the subtraction-artifact reason instead;
preserved CronDelete/CronList zeros rather than dropping them; caveats[] honestly empty;
verify-catalogue returned absent: [] across all 19 rows; the dual-ledger prefixDelta /
deferredDelta split held throughout and is what made the ToolSearch finding legible (deny it and
13,343 tokens migrate into the window, +15,824 prefix). setup apply was genuinely idempotent
and wrote nothing. The disableWorkflows lever was correctly detected as already applied. The ledger
round-trip predicted 4,608 and measured 4,608, exact.

---

Evidence packets (sealed, with per-file digests) at
~/.claude/plugins/data/plugin-quality-melodic-software/evidence/6cd8f3c9-.../<target>/20260824T203113Z/
— audit-notes.md per target, plus evidence*.md and contract.md.