redact replaces an entire JSON Schema with <redacted:private-key>

[kjanat]

Version

envctl v0.6.0

Reproduction

Run redact over a normal JSON Schema document:

$ envctl redact < src/data/schemas/gameplay.schema.json
<redacted:private-key>

The input is a 37,499-byte draft-06 JSON Schema with SHA-256:

d2702cc7bc4884401f38af9cf37c3a906f0a7dcbb2408fdfd0c9cd8fc8d0057e

gameplay.schema.json

stall-visuals.schema.json

The schema contains no private key, credential, or private-key token. Small schema constructs such as $schema, $ref, type, and definitions pass through normally. None of the individual objects under definitions produces this whole-document marker when tested separately; the false positive occurs on the complete document.

Actual behavior

The entire schema is replaced with:

<redacted:private-key>

Expected behavior

Ordinary JSON Schema content should remain structurally intact. Only actual secret values should be redacted; document size or an aggregate of normal schema text should not cause the whole document to be classified as a private key.

[coderabbitai]

🔗 Related PRs

#4 - redact: catch high-entropy SESSION keys (BW_SESSION) and slashed base64 values [merged]
#5 - Close redaction false negatives in key parsing and value shapes [merged]
#6 - fix: redact digit-glued key segments, crypt hashes, and PSK keys [merged]
#8 - fix: prevent redaction false positives for token-like values [merged]
#10 - Choose redaction with --redact=WHEN, and test completions by running them [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Summary

Ahoy! Here be the plan, laid out proper:

• Stop the JSON buffering fallback from reusing PRIVKEY_TOKEN; a size or line overflow is not evidence of a private key, savvy?
• On overflow, flush the buffered lines through the normal per-line scanner, reset the buffering state, and keep scanning; secrets stay redacted, but legitimate content survives the storm.
• Remove the dead json_drop state so no path can mislabel oversized content again.
• Add large-JSON regression fixtures and cases, mirroring the filter-pem-oversized template, to guard the fix. And there she blows, bug fixed and shipshape!

Design Choices

Design Choice 1: What should the buffering machine do when a JSON object exceeds the size or line caps, arr?

Options Considered:

1. Flush the buffered lines through the normal per-line scanner, then stop multi-line accumulation and continue scanning the remaining lines as ordinary text. Secrets are still redacted per line, but the document survives.
2. Keep discarding the buffer, but emit a neutral marker such as <redacted:oversized> instead of the private-key token. This still destroys legitimate content.
3. Raise JSON_LINE_MAX and JSON_CARRY_MAX. This only moves the cliff edge; a larger document still breaks.

Chosen Option: 1

Rationale: It satisfies the ticket requirement that ordinary JSON Schema content stays structurally intact, keeps memory bounded, and preserves per-line secret redaction. Shipshape and proper, this one.

Design Choice 2: Is a version bump or changelog entry required in this PR, ye ask?

Options Considered:

1. No in-repo file change; the project derives its version from git tags and publishes notes from the tag body (per CONTRIBUTING.md).
2. Create a new CHANGELOG.md to comply with the generic guideline.

Chosen Option: 1

Rationale: The project's real convention is git-tag based, so no tracked file needs an edit for this fix. No need to invent paperwork, savvy?

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Make the JSON overflow fallback safe and honest

This phase fixes the root cause, me hearties. The goal is simple: when the JSON buffering machine hits its size or line cap, it must never claim the content is a private key, and it must never silently swallow legitimate lines. Instead, it must release the buffered content through the ordinary per-line scanner and then step out of buffering mode. All work is inside src/redact.c, with a small clean-up in src/redact.h.

Task 1: Replace the continuation-overflow branch so it flushes instead of dropping

Change the overflow path so a large legitimate object survives.

• Locate the overflow branch inside scan_text_line() in src/redact.c, where st->json_lines > JSON_LINE_MAX or json_append() fails while st->json_depth > 0.
• Change the action: pass the already-buffered bytes (st->json_buf, st->json_len) through scan_plain() with stream_json disabled, so any real secrets in the buffered region are still masked per line, and the rest is emitted verbatim.
• After the flush, call json_reset(st) to clear the buffering state and set json_depth back to 0.
• Then process the current line as ordinary text through the normal per-line path, so the line that triggered the overflow is not lost.
• Remove the PRIVKEY_TOKEN emission from this branch.
• Preserve the byte-exact output for objects that still fit within the caps. Only the overflow case changes.

Task 2: Fix the first-fragment drop in the streaming branch

Handle the rare case where the opening fragment alone is too large to buffer.

• Locate the branch in scan_plain() (the !oe && stream_json && json_candidate(...) path) where json_append() fails on the first fragment and the code sets st->json_drop and emits PRIVKEY_TOKEN.
• Change the action: emit the fragment through ordinary text handling (a scan_plain() pass with stream_json disabled) and do not start multi-line buffering.
• Remove the PRIVKEY_TOKEN emission from this branch.

Task 3: Retire the now-unused drop state

Tidy the state machine so no dead "drop" logic remains. A clean deck, aye.

• Remove the st->json_drop field from ScanState in src/redact.h, along with its handling in json_reset() and the st->json_drop && len && eollen blank-line branch in scan_text_line() in src/redact.c.
• Confirm that no other code reads json_drop after the changes in Tasks 1.1 and 1.2.
• If removing the field proves to touch more call sites than expected, keep the field but ensure it is never set, and delete only its output-suppression branches. The essential requirement is that no code path reuses PRIVKEY_TOKEN for an overflow.

🤖 Prompt for AI agents

Fix the root cause of the bug in `src/redact.c` and `src/redact.h`: when the
JSON buffering machine hits its size or line cap, it must never claim the
content is a private key, and it must never silently discard legitimate lines.

Step 1 — fix the continuation-overflow branch:
- In `scan_text_line()` in `src/redact.c`, find the overflow branch where
`st->json_lines > JSON_LINE_MAX` or `json_append()` fails while `st->json_depth
> 0`.
- Change this branch to pass the buffered bytes (`st->json_buf`, `st->json_len`)
through `scan_plain()` with `stream_json` disabled, so real secrets in the
buffered region are still masked per line and the rest is emitted verbatim.
- After the flush, call `json_reset(st)` to clear buffering state and reset
`json_depth` to 0.
- Process the current line (the one that triggered the overflow) as ordinary
text through the normal per-line path so it is not lost.
- Remove the `PRIVKEY_TOKEN` emission from this branch.
- Do not change behaviour for objects that fit within the caps; output for those
must stay byte-exact.

Step 2 — fix the first-fragment drop:
- In `scan_plain()`, find the `!oe && stream_json && json_candidate(...)` branch
where `json_append()` fails on the first fragment, setting `st->json_drop` and
emitting `PRIVKEY_TOKEN`.
- Change this branch to emit the fragment through ordinary text handling (a
`scan_plain()` pass with `stream_json` disabled) and to not start multi-line
buffering.
- Remove the `PRIVKEY_TOKEN` emission from this branch.

Step 3 — remove dead drop state:
- Remove the `st->json_drop` field from `ScanState` in `src/redact.h`.
- Remove its handling from `json_reset()` and remove the `st->json_drop && len
&& eollen` blank-line branch in `scan_text_line()` in `src/redact.c`.
- Verify no other code reads `json_drop` after the Step 1 and Step 2 changes.
- If removing the field touches more call sites than expected, keep the field
but ensure it is never set, and delete only its output-suppression branches. The
essential requirement: no code path may reuse `PRIVKEY_TOKEN` for an overflow.

Do not touch the genuine private-key detectors (`is_pem_private_n`, PuTTY header
match, `json_private_object`) or the PEM state machine. Do not raise
`JSON_LINE_MAX` or `JSON_CARRY_MAX`.

Phase 2: Add regression coverage for large JSON documents

This phase locks the fix in place so the bug cannot creep back aboard. Follow the existing large-fixture pattern used by filter-pem-oversized.case.

Task 1: Add a large secret-free JSON fixture and pass-through case

Prove that a big, benign JSON object survives redaction unchanged.

• Create a new fixture under tests/fixtures/text/ that is a single pretty-printed JSON object spanning more than 512 lines with no secret values. Include sentinel keys near the start, past the 512-line boundary, and after the closing }.
• Create a new case tests/cases/filter-json-oversized.case that runs redact with --no-env and feeds the fixture via %% stdin-file.
• Assert the full document passes through unchanged. If the expected output is large, store it under tests/fixtures/expected/ and reference it with %% stdout-file.

Task 2: Add a case that confirms secrets are still redacted past the cap

Prove the fallback still protects real secrets, so the fix does not create a leak.

• Create a fixture like the one in Task 2.1, but place a recognisable secret assignment (for example an api_key value) on a line beyond the 512-line boundary.
• Create a matching case asserting that the secret value is redacted per line while the surrounding schema text passes through.
• Confirm the whole document is not collapsed to <redacted:private-key>.

Task 3: Confirm the full suite still passes

Verify no existing invariant breaks. This be the moment of triumph.

• Run tests/run.sh and confirm the existing filter-json-*, filter-jwk-pretty, and filter-pem-* cases still pass byte-for-byte.
• Confirm the new cases in Phase 2 pass.

🤖 Prompt for AI agents

Add regression coverage for the Phase 1 fix, following the existing
large-fixture pattern used by `filter-pem-oversized.case`.

Step 1 — pass-through case for benign large JSON:
- Add a fixture under `tests/fixtures/text/` that is one pretty-printed JSON
object spanning more than 512 lines, with no secret values. Include sentinel
keys near the start, past the 512-line boundary, and after the closing `}`.
- Add a case `tests/cases/filter-json-oversized.case` that runs `redact` with
`--no-env` and feeds the fixture via `%% stdin-file`.
- Assert the full document passes through unchanged. If the expected output is
large, store it under `tests/fixtures/expected/` and reference it with `%%
stdout-file`.

Step 2 — case for secret past the cap:
- Add a fixture similar to Step 1, but place a recognisable secret assignment
(for example an `api_key` value) on a line beyond the 512-line boundary.
- Add a matching case asserting the secret value is redacted per line while
surrounding schema text passes through.
- Confirm the whole document is not collapsed to `<redacted:private-key>`.

Step 3 — full suite verification:
- Run `tests/run.sh`.
- Confirm existing `filter-json-*`, `filter-jwk-pretty`, and `filter-pem-*`
cases still pass byte-for-byte.
- Confirm the two new cases added in this phase pass.

Research

Arr, gather round, here be the lay of the land. The redact verb is the only user-facing subcommand. src/main.c dispatches it to act_redact() in src/filter.c, which reads stdin line by line and feeds every line into scan_text_line() in src/redact.c. All classification logic lives in src/redact.c. That file holds three genuine private-key detectors (is_pem_private_n, the PuTTY header match, and json_private_object) and two multi-line state machines that share ScanState (declared in src/redact.h): a PEM BEGIN/END carry machine, and a JSON object buffering machine.

The JSON buffering machine is the villain here. When a { opens and does not close on the same line, scan_plain() starts accumulating lines into st->json_buf and stops writing any output. If the object exceeds JSON_LINE_MAX (512 lines) or JSON_CARRY_MAX (1 MiB) before the closing }, the code sets st->json_drop, throws away the buffer, and emits one <redacted:private-key> token (macro PRIVKEY_TOKEN). Every buffered line before the overflow produced no output, and every line after it up to the real } produces only blank lines. So a 37,499-byte schema that opens with { and spans hundreds of lines gets erased and mislabelled. The tests in tests/cases/filter-json-*.case cover only tiny JSON inputs, so this overflow path is currently untested.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Fix the root cause of the bug in `src/redact.c` and `src/redact.h`: when the
JSON buffering machine hits its size or line cap, it must never claim the
content is a private key, and it must never silently discard legitimate lines.

Step 1 — fix the continuation-overflow branch:
- In `scan_text_line()` in `src/redact.c`, find the overflow branch where
`st->json_lines > JSON_LINE_MAX` or `json_append()` fails while `st->json_depth
> 0`.
- Change this branch to pass the buffered bytes (`st->json_buf`, `st->json_len`)
through `scan_plain()` with `stream_json` disabled, so real secrets in the
buffered region are still masked per line and the rest is emitted verbatim.
- After the flush, call `json_reset(st)` to clear buffering state and reset
`json_depth` to 0.
- Process the current line (the one that triggered the overflow) as ordinary
text through the normal per-line path so it is not lost.
- Remove the `PRIVKEY_TOKEN` emission from this branch.
- Do not change behaviour for objects that fit within the caps; output for those
must stay byte-exact.

Step 2 — fix the first-fragment drop:
- In `scan_plain()`, find the `!oe && stream_json && json_candidate(...)` branch
where `json_append()` fails on the first fragment, setting `st->json_drop` and
emitting `PRIVKEY_TOKEN`.
- Change this branch to emit the fragment through ordinary text handling (a
`scan_plain()` pass with `stream_json` disabled) and to not start multi-line
buffering.
- Remove the `PRIVKEY_TOKEN` emission from this branch.

Step 3 — remove dead drop state:
- Remove the `st->json_drop` field from `ScanState` in `src/redact.h`.
- Remove its handling from `json_reset()` and remove the `st->json_drop && len
&& eollen` blank-line branch in `scan_text_line()` in `src/redact.c`.
- Verify no other code reads `json_drop` after the Step 1 and Step 2 changes.
- If removing the field touches more call sites than expected, keep the field
but ensure it is never set, and delete only its output-suppression branches. The
essential requirement: no code path may reuse `PRIVKEY_TOKEN` for an overflow.

Do not touch the genuine private-key detectors (`is_pem_private_n`, PuTTY header
match, `json_private_object`) or the PEM state machine. Do not raise
`JSON_LINE_MAX` or `JSON_CARRY_MAX`.
===============================================================================

Task: 2

Add regression coverage for the Phase 1 fix, following the existing
large-fixture pattern used by `filter-pem-oversized.case`.

Step 1 — pass-through case for benign large JSON:
- Add a fixture under `tests/fixtures/text/` that is one pretty-printed JSON
object spanning more than 512 lines, with no secret values. Include sentinel
keys near the start, past the 512-line boundary, and after the closing `}`.
- Add a case `tests/cases/filter-json-oversized.case` that runs `redact` with
`--no-env` and feeds the fixture via `%% stdin-file`.
- Assert the full document passes through unchanged. If the expected output is
large, store it under `tests/fixtures/expected/` and reference it with `%%
stdout-file`.

Step 2 — case for secret past the cap:
- Add a fixture similar to Step 1, but place a recognisable secret assignment
(for example an `api_key` value) on a line beyond the 512-line boundary.
- Add a matching case asserting the secret value is redacted per line while
surrounding schema text passes through.
- Confirm the whole document is not collapsed to `<redacted:private-key>`.

Step 3 — full suite verification:
- Run `tests/run.sh`.
- Confirm existing `filter-json-*`, `filter-jwk-pretty`, and `filter-pem-*`
cases still pass byte-for-byte.
- Confirm the two new cases added in this phase pass.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!