Skip to content

Commit 51ecf00

Browse files
committed
feat(governance): Rounds 36-50 panel — plan-only and plain-review boundaries
Extend Socratic panel through Round 50 with unanimous agreements on plan-only-no-code and plain-review-vs-risk holdouts, production-negation router boundaries, ROUTE-002/003 expansion, ROUTING_CONTRACT R9, and governance doc sync.
1 parent 6272e63 commit 51ecf00

11 files changed

Lines changed: 356 additions & 16 deletions

CONTRIBUTING.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ Before tuning `route_paraphrase_eval.py`:
2828

2929
1. Add fresh holdout phrases to `route-002-holdout-eval.yaml` and/or adversarial cases to `route-003-adversarial-eval.yaml`.
3030
2. Run `make all``validate_route_fixture.py` enforces minimum fixture coverage.
31+
3. When fixing review-vs-risk or plan-only boundaries, add holdout phrases **before** editing `route_paraphrase_eval.py` (see `ROUTING_CONTRACT.md` R8–R9).
3132
3. Record governance-surface decisions in `PROJECT_KNOWLEDGE.md` Decision Index when routing contract changes.
3233

3334
See [ROUTING_CONTRACT R8](reflective-prompt-library/plans/ROUTING_CONTRACT.md) and the [Governance Maintenance Playbook](reflective-prompt-library/GLOSSARY.md).

reflective-prompt-library/GLOSSARY.md

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -280,11 +280,32 @@ fairness without translating canonical `SKILL.md` contracts.
280280
its English equivalent intent group.
281281

282282

283+
---
284+
285+
286+
## Plan-Only Without Code / 僅規劃不寫程式
287+
288+
A request for tickets, acceptance criteria, rollout plans, or specs with explicit
289+
no-code context. Routes to `reflective-spec-plan`, not `reflective-implement`.
290+
291+
**Operational test:** Phrase includes planning artifact signals **and** no-code
292+
context (e.g. "plan only", "without touching the repo", 不要改程式).
293+
294+
---
295+
296+
## Plain Review (Non-Production) / 非正式環境審查
297+
298+
Code or diff review for correctness, readability, or regressions **without**
299+
production-risk assessment. Routes to `reflective-review`, not `reflective-risk`.
300+
301+
**Operational test:** Review/diff/PR context plus production negation (e.g.
302+
"not production deploy", 不是正式環境風險) or readability-only scope.
303+
283304
---
284305

285306
## Governance Maintenance Playbook / 治理維護手冊
286307

287-
Ongoing upkeep after panel close (Rounds 1–35). Not agent instructions — operator checklist.
308+
Ongoing upkeep after panel close (Rounds 1–50). Not agent instructions — operator checklist.
288309

289310
**Operational test:** Before router tuning, add fresh ROUTE-002/003 holdout phrases; run `make validate`; record decisions in `PROJECT_KNOWLEDGE.md` Decision Index when governance surface changes.
290311

reflective-prompt-library/PROJECT_KNOWLEDGE.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ operating its own agent runtime. See [README.md](README.md#north-star).
4141
> does not silently expand. Completed milestones are retired from this section;
4242
> their outcomes live in the Decision Index.
4343
44-
**Ongoing (post Round 35 panel):** maintain ROUTE-001/002/003 evals, governance validators,
44+
**Ongoing (post Round 50 panel):** maintain ROUTE-001/002/003 evals, governance validators,
4545
and holdout expansion before router tuning — see [GLOSSARY.md](GLOSSARY.md) Governance
4646
Maintenance Playbook. No open implementation blockers from Rounds 8–20; deferred promotions
4747
are recurrence-gated — see [panel backlog](plans/multi-agent-panel-consensus-2026-06-25.md#recurrence-gated-backlog-not-panel-blockers).
@@ -81,6 +81,7 @@ are recurrence-gated — see [panel backlog](plans/multi-agent-panel-consensus-2
8181
- 2026-06-20 Synthesize external-adoption case studies, then correct the evidence and promotion-gate scope after the Test Plan routing counterexample → [record](plans/external-adoption-case-studies-2026-06-20.md)
8282
- 2026-06-21 Evaluate Hyperplan / multi-agent adversarial planning — no change (runtime = non-goal; methodology overlaps; possible gaps not promoted) → [record](plans/external-adoption-case-studies-2026-06-20.md)
8383
- 2026-06-25 Round 21 panel audit — no implementation blockers; recurrence-gated backlog formalized; QUALITY_GATES drift fixed → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
84+
- 2026-06-25 Rounds 36–50 panel — plan-only/plain-review holdouts, production-negation boundaries, ROUTE-002/003 expansion → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
8485
- 2026-06-25 Rounds 8–20 panel — milestones closed, ROUTE-003 adversarial eval, skill examples gate, maintenance playbook → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
8586
- 2026-06-25 Round 7 panel — AGENTS harness-policy alignment, zh-TW ROUTE-002 holdout, metrics sync → [record](plans/multi-agent-panel-consensus-2026-06-25.md)
8687
- 2026-06-25 Round 6 panel — undocumented-decisions warning, ROUTE-002 holdout expansion, benchmark fixture CI, reject full SKILL i18n → [record](plans/multi-agent-panel-consensus-2026-06-25.md)

reflective-prompt-library/plans/QUALITY_GATES_SUMMARY.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -260,8 +260,8 @@ The implementation aligns with research findings:
260260
| Skills with governance | 9/9 | ✅ Complete |
261261
| Benchmark tasks | 23 | ✅ Ready |
262262
| Routing consistency | 100.0% | ✅ Passes ROUTE-001 expanded boundary eval |
263-
| Holdout routing consistency | 100.0% | ✅ ROUTE-002 (28 groups, 82 paraphrases) |
264-
| Adversarial routing consistency | 100.0% | ✅ ROUTE-003 (9 groups, 20 paraphrases) |
263+
| Holdout routing consistency | 100.0% | ✅ ROUTE-002 (32 groups, 91 paraphrases) |
264+
| Adversarial routing consistency | 100.0% | ✅ ROUTE-003 (11 groups, 24 paraphrases) |
265265
| Skill example coverage | 9/9 | ✅ validate_skill_examples.py |
266266
| Linting errors | 0 | ✅ Clean |
267267

@@ -292,7 +292,7 @@ ROUTE-002 measures unseen phrasing separately from ROUTE-001. Round 7 (2026-06-2
292292
### Done ✅
293293

294294
1. **CI/CD**`.github/workflows/python-tools.yml` runs `make all` on push/PR
295-
2. **ROUTE-001/002/003 in CI** — 128 + 82 + 20 paraphrases at 100% consistency (seeded fixtures); `validate_route_fixture.py` gates minimum coverage
295+
2. **ROUTE-001/002/003 in CI** — 128 + 91 + 24 paraphrases at 100% consistency (seeded fixtures); `validate_route_fixture.py` gates minimum coverage
296296
3. **Governance validators** — links, lint, governance metadata, PROJECT_KNOWLEDGE, benchmark fixture, skill examples
297297
4. **Harness policy docs** — CONTRIBUTING, AGENTS, SKILL_INSTALLATION, maintenance playbook
298298

reflective-prompt-library/plans/ROUTING_CONTRACT.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,15 @@ This contract applies to:
7979
- `validate_route_fixture.py` enforces minimum group and phrase counts in CI.
8080
- Holdout expansion is maintenance, not proof of broad semantic routing.
8181

82+
### R9: Production-negation and plan-only boundaries
83+
84+
- Phrases that **negate** production risk (e.g. "not production deploy", 不是正式環境風險)
85+
must not auto-route to `reflective-risk` when the user requests plain code review.
86+
- Planning-only requests with explicit no-code context route to `reflective-spec-plan`,
87+
even when incidental words like "change" appear in "code changes".
88+
- Add holdout cases for these boundaries before tuning router keyword lists.
89+
90+
8291
## Router Output Contract
8392

8493
Use this minimal output shape for routing responses:

reflective-prompt-library/plans/multi-agent-panel-consensus-2026-06-25.md

Lines changed: 203 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -811,8 +811,209 @@ User directive: continue Socratic panel for as many rounds as feasible.
811811

812812
- `make all`: **62 pytest**, ROUTE-001 **100%** (16 groups / 128), ROUTE-002 **100%** (28 / 82), ROUTE-003 **100%** (9 / 20), governance **9/9**, skill examples **9/9**, route fixture gate **pass**
813813

814+
815+
## Round 36 — Plan-only holdout (2026-06-25)
816+
817+
User directive: continue Socratic panel for as many rounds as feasible.
818+
819+
### BC: Add `plan_only_no_code_holdout` to ROUTE-002?
820+
821+
| Lens | Position |
822+
| --- | --- |
823+
| Codex | **Agree** — "code changes" in plan-only phrasing misroutes without holdout |
824+
| Opus | **Agree** — falsifiable boundary before keyword tuning |
825+
| Gemini | **Agree** — plan-only saves implement context cost |
826+
| Composer | **Agree** — IDE users mix plan + code words casually |
827+
| Sakana | **Agree** — boundary clarity over new skill |
828+
| GLM | **Agree** — TW plan-only phrases need parity |
829+
830+
**Consensus:** **Agree** — ROUTE-002 group + plan-only boundary rule.
831+
832+
## Round 37 — Plain review vs risk (2026-06-25)
833+
834+
### BD: Add `plain_review_not_risk_holdout`?
835+
836+
| Lens | Position |
837+
| --- | --- |
838+
| All six | **Agree** — production keyword false positives on plain PR review |
839+
840+
**Socratic Q:** What fails if every "production" mention routes to risk?
841+
**Answer:** Readability/regression reviews get over-rigor and wrong workflow.
842+
843+
**Consensus:** **Agree** — holdout group + production-negation boundary.
844+
845+
## Round 38 — Router boundary repair (2026-06-25)
846+
847+
### BE: Plan-only and production-negation rules in `route_paraphrase_eval.py`?
848+
849+
| Lens | Position |
850+
| --- | --- |
851+
| All six | **Agree** — holdout added first (R8); then boundary repair |
852+
853+
**Consensus:** **Agree**`no_code_context` hoist, `production_negated`, `plan_only_signals`, `plain_review_signals`.
854+
855+
## Round 39 — Adversarial implement/research traps (2026-06-25)
856+
857+
### BF: ROUTE-003 `implement_not_plan_trap` and `research_not_brief_trap`?
858+
859+
| Lens | Position |
860+
| --- | --- |
861+
| All six | **Agree** — separate adversarial traps from holdout tuning set |
862+
863+
**Consensus:** **Agree** — +2 ROUTE-003 groups.
864+
865+
## Round 40 — ROUTING_CONTRACT R9 (2026-06-25)
866+
867+
### BG: Codify production-negation and plan-only as normative?
868+
869+
| Lens | Position |
870+
| --- | --- |
871+
| Opus/Codex | **Agree** — contract documents fairness edge cases |
872+
| All | **Agree** — R9 in `ROUTING_CONTRACT.md` |
873+
874+
**Consensus:** **Agree** — R9 added.
875+
876+
## Round 41 — Fixture minimum bump (2026-06-25)
877+
878+
### BH: Raise `validate_route_fixture.py` minimums?
879+
880+
| Lens | Position |
881+
| --- | --- |
882+
| Codex | **Agree** — prevents accidental holdout shrinkage |
883+
| All | **Agree** — ROUTE-002 32/91, ROUTE-003 11/24 |
884+
885+
**Consensus:** **Agree** — minimum constants updated; existing pytest covers.
886+
887+
## Round 42 — zh-TW plan/review holdouts (2026-06-25)
888+
889+
### BI: TW fairness groups for plan-only and plain review?
890+
891+
| Lens | Position |
892+
| --- | --- |
893+
| GLM | **Agree** — 不是正式環境風險 must not force risk workflow |
894+
| All | **Agree**`zh_tw_plan_only_holdout`, `zh_tw_plain_review_holdout` |
895+
896+
**Consensus:** **Agree**.
897+
898+
## Round 43 — GLOSSARY boundary terms (2026-06-25)
899+
900+
### BJ: Operational definitions for plan-only and plain review?
901+
902+
| Lens | Position |
903+
| --- | --- |
904+
| All six | **Agree** — EN definitions; TW headings in glossary |
905+
906+
**Consensus:** **Agree** — GLOSSARY entries + playbook rounds 1–50.
907+
908+
## Round 44 — Metrics sync (2026-06-25)
909+
910+
### BK: QUALITY_GATES_SUMMARY metrics?
911+
912+
| Lens | Position |
913+
| --- | --- |
914+
| All six | **Agree** — 32/91 and 11/24 |
915+
916+
**Consensus:** **Agree**.
917+
918+
## Round 45 — Decision Index (2026-06-25)
919+
920+
### BL: PROJECT_KNOWLEDGE entry for Rounds 36–50?
921+
922+
| Lens | Position |
923+
| --- | --- |
924+
| All six | **Agree** — single Decision Index pointer |
925+
926+
**Consensus:** **Agree**.
927+
928+
## Round 46 — Cheatsheet cues (2026-06-25)
929+
930+
### BM: SKILL_TRIGGER_CHEATSHEET EN/TW boundary lines?
931+
932+
| Lens | Position |
933+
| --- | --- |
934+
| Composer/GLM | **Agree** — progressive disclosure for adopters |
935+
| All | **Agree** |
936+
937+
**Consensus:** **Agree**.
938+
939+
## Round 47 — Tenth skill reaffirm (2026-06-25)
940+
941+
### BN: Re-open tenth core skill?
942+
943+
| Lens | Position |
944+
| --- | --- |
945+
| All six | **Reject** — promotion gate unchanged |
946+
947+
**Consensus:** **Reject**.
948+
949+
## Round 48 — Benchmark CI reaffirm (2026-06-25)
950+
951+
### BO: LLM benchmark in CI?
952+
953+
| Lens | Position |
954+
| --- | --- |
955+
| All six | **Reject** — fixture gate only |
956+
957+
**Consensus:** **Reject**.
958+
959+
## Round 49 — CONTRIBUTING R8–R9 pointer (2026-06-25)
960+
961+
### BP: Routing maintenance checklist extension?
962+
963+
| Lens | Position |
964+
| --- | --- |
965+
| All six | **Agree** — plan-only/review-vs-risk holdout-before-tune step |
966+
967+
**Consensus:** **Agree**.
968+
969+
## Round 50 — Final extended seal (2026-06-25)
970+
971+
### BQ: Close panel at Round 50?
972+
973+
| Lens | Position |
974+
| --- | --- |
975+
| All six | **Agree** — options A–BQ resolved; maintenance only remains |
976+
977+
**Consensus:** **Agree close** — panel sealed at **Round 50**; Decision Index entry.
978+
979+
### Rounds 36–50 verdict table
980+
981+
| ID | Option | Verdict | Action |
982+
| --- | --- | --- | --- |
983+
| BC | Plan-only holdout | **Agree** | ROUTE-002 group |
984+
| BD | Plain review holdout | **Agree** | ROUTE-002 group |
985+
| BE | Router boundaries | **Agree** | plan-only + production-negation |
986+
| BF | Adversarial traps | **Agree** | ROUTE-003 +2 groups |
987+
| BG | R9 contract | **Agree** | ROUTING_CONTRACT.md |
988+
| BH | Fixture minimums | **Agree** | validate_route_fixture.py |
989+
| BI | zh-TW holdouts | **Agree** | ROUTE-002 TW groups |
990+
| BJ | GLOSSARY terms | **Agree** | plan-only + plain review |
991+
| BK | Metrics sync | **Agree** | QUALITY_GATES_SUMMARY |
992+
| BL | Decision Index | **Agree** | PROJECT_KNOWLEDGE |
993+
| BM | Cheatsheet cues | **Agree** | EN + zh-TW |
994+
| BN | Tenth skill | **Reject** ||
995+
| BO | Benchmark CI | **Reject** ||
996+
| BP | CONTRIBUTING | **Agree** | R8–R9 step |
997+
| BQ | Final seal | **Agree** | Panel sealed Round 50 |
998+
999+
**All roles agree.**
1000+
1001+
## Implemented Changes (Rounds 36–50)
1002+
1003+
- `route-002-holdout-eval.yaml`: plan-only, plain-review, zh-TW plan/review groups (91 paraphrases / 32 groups)
1004+
- `route-003-adversarial-eval.yaml`: implement-not-plan, research-not-brief traps (24 paraphrases / 11 groups)
1005+
- `route_paraphrase_eval.py`: production-negation, plan-only, plain-review boundaries
1006+
- `validate_route_fixture.py`: minimum coverage 32/91 and 11/24
1007+
- `ROUTING_CONTRACT.md`: R9
1008+
- `GLOSSARY.md`, `QUALITY_GATES_SUMMARY.md`, `PROJECT_KNOWLEDGE.md`, `CONTRIBUTING.md`
1009+
- `SKILL_TRIGGER_CHEATSHEET.md` + `.zh-TW.md`
1010+
1011+
## Verification (Rounds 36–50)
1012+
1013+
- `make all`: **62 pytest**, ROUTE-001 **100%** (16/128), ROUTE-002 **100%** (32/91), ROUTE-003 **100%** (11/24)
1014+
8141015
## Panel status (final)
8151016

816-
**Sealed 2026-06-25** after **Round 35** (options A–BB). Extended deliberation complete. Future work is recurrence-gated maintenance per playbook — expand holdout before router tuning; no open implementation blockers.
1017+
**Sealed 2026-06-25** after **Round 50** (options A–BQ). Extended deliberation complete. Future work is recurrence-gated maintenance per playbook — expand holdout before router tuning; no open implementation blockers.
8171018

818-
_Ongoing maintenance: [GLOSSARY.md](../GLOSSARY.md) Governance Maintenance Playbook — `validate_route_fixture.py` + ROUTE-002/003 holdout expansion before `route_paraphrase_eval.py` tuning._
1019+
_Ongoing maintenance: [GLOSSARY.md](../GLOSSARY.md) Governance Maintenance Playbook — `validate_route_fixture.py` + ROUTE-002/003 holdout expansion before `route_paraphrase_eval.py` tuning (R8–R9)._

reflective-prompt-library/plans/route-002-holdout-eval.yaml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,44 @@ holdout_sets:
296296
- clarify the goal and assumptions before we pick a workflow
297297
- what should we optimize for before choosing skills
298298

299+
- name: plan_only_no_code_holdout
300+
expected_workflow: reflective-spec-plan
301+
expected_rigor: medium
302+
expected_enhancements:
303+
- acceptance_criteria
304+
phrases:
305+
- draft rollout plan only no code changes
306+
- write tickets and acceptance criteria without touching the repo
307+
- spec the delivery plan before any implementation starts
308+
309+
- name: plain_review_not_risk_holdout
310+
expected_workflow: reflective-review
311+
expected_rigor: high
312+
expected_enhancements:
313+
- quality_review
314+
phrases:
315+
- review this PR for style and logic before merge
316+
- check the diff for readability not production deploy
317+
- inspect the patch for regressions without production risk assessment
318+
319+
- name: zh_tw_plan_only_holdout
320+
expected_workflow: reflective-spec-plan
321+
expected_rigor: medium
322+
expected_enhancements:
323+
- acceptance_criteria
324+
phrases:
325+
- 把規格寫出來但不要改程式
326+
- 只寫工單和驗收標準不要動 code
327+
328+
- name: zh_tw_plain_review_holdout
329+
expected_workflow: reflective-review
330+
expected_rigor: high
331+
expected_enhancements:
332+
- quality_review
333+
phrases:
334+
- 審查 PR 的可讀性不是正式環境風險
335+
336+
299337
evaluation_rules:
300338
- name: route_equivalence
301339
assert: same_intent_group_same_workflow

reflective-prompt-library/plans/route-003-adversarial-eval.yaml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,25 @@ adversarial_sets:
100100
- lessons learned retro after this sprint
101101
- sprint retrospective with lessons learned
102102

103+
- name: implement_not_plan_trap
104+
expected_workflow: reflective-implement
105+
expected_rigor: low
106+
expected_enhancements:
107+
- tests
108+
phrases:
109+
- ship the code fix for the off-by-one bug
110+
- implement the approved spec in the repository
111+
112+
- name: research_not_brief_trap
113+
expected_workflow: reflective-research
114+
expected_rigor: medium
115+
expected_enhancements:
116+
- source_check
117+
phrases:
118+
- compare official docs for both libraries before deciding
119+
- source-backed comparison of migration paths
120+
121+
103122

104123
evaluation_rules:
105124
- name: route_equivalence

0 commit comments

Comments
 (0)