Skip to content

eval: Add offline readiness checks and opt-in arm bundles - #87

Draft
cgwalters-bot wants to merge 1 commit into
cgwalters-forge:mainfrom
cgwalters-bot:dispatch/implement/agent-run-43522626557
Draft

cgwalters-bot wants to merge 1 commit into
cgwalters-forge:mainfrom
cgwalters-bot:dispatch/implement/agent-run-43522626557

Conversation

@cgwalters-bot

Copy link
Copy Markdown
Collaborator

Executive summary

Add an offline allocation/enforcement evidence checker and complete opt-in evaluation bundles through the shared tier resolver, leaving normal routing unchanged. This makes the first readiness slice usable without inventing capacity or launching measurements.

Part of #52

The existing report records the pinned runner contract and a concrete external reservation/enforcement follow-up. The measured comparison remains blocked at 0/54 cells: authenticated capacity/reservations, hard child-inclusive limits, frozen tasks, actual role/model execution and cache evidence remain unproved. No inference or extra jobs were launched.

Verification: just check passes all 66 tests; the archived-summary test matches summarize-eval output. Initial diagnostic/number-spelling test failures and whitespace findings were fixed.

Review: one adversarial pass, no blockers; follow-ups are stricter artifact URL syntax and independent assertions of the full built-in agent set, plus a no-inference configuration-load test when available.

{"reason":"incomplete","done":"Offline readiness checker, shared-resolver arm bundles, tests and pinned contract report are implemented and reviewed.","left":"Authenticated capacity and reservation evidence, enforceable aggregate/per-cell limits, execution/cache proof, frozen tasks and 54 measured cells.","next":"Resolve the external accounting/enforcement follow-up documented in eval/report.md; independently validate evidence before any inference, then continue the protocol."}

Generated by dispatch · ◷

## Executive summary

Add an offline allocation/enforcement evidence checker and complete opt-in evaluation bundles through the shared tier resolver, leaving normal routing unchanged. This makes the first readiness slice usable without inventing capacity or launching measurements.

Part of cgwalters-forge#52

The existing [report](eval/report.md#current-runner-contract-and-external-prerequisites) records the pinned runner contract and a concrete external reservation/enforcement follow-up. The measured comparison remains blocked at 0/54 cells: authenticated capacity/reservations, hard child-inclusive limits, frozen tasks, actual role/model execution and cache evidence remain unproved. No inference or extra jobs were launched.

Verification: `just check` passes all 66 tests; the archived-summary test matches `summarize-eval` output. Initial diagnostic/number-spelling test failures and whitespace findings were fixed.

Review: one adversarial pass, no blockers; follow-ups are stricter artifact URL syntax and independent assertions of the full built-in agent set, plus a no-inference configuration-load test when available.

```checkpoint
{"reason":"incomplete","done":"Offline readiness checker, shared-resolver arm bundles, tests and pinned contract report are implemented and reviewed.","left":"Authenticated capacity and reservation evidence, enforceable aggregate/per-cell limits, execution/cache proof, frozen tasks and 54 measured cells.","next":"Resolve the external accounting/enforcement follow-up documented in eval/report.md; independently validate evidence before any inference, then continue the protocol."}
```

Assisted-by: https://github.com/cgwalters/cgwalters#llms
@cgwalters-bot

cgwalters-bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Continuation round 1 of 2 for 3b28d0c83364: run.

Generated-by: https://github.com/cgwalters/#llms

@cgwalters-bot

cgwalters-bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author
⏸️ [dispatch](https://github.com/cgwalters-forge/tracker/actions/runs/43758959261) is blocked: it handed back missing_data and no change. Its reason:

Authenticated allocation and enforcement evidence: No dated existing-pool capacity/reservation receipts or hard child-inclusive aggregate/per-cell limits are supplied; the checked-in protocol blocks inference without them.
context: PR #87 at 3b28d0c; eval/report.md#current-runner-contract-and-external-prerequisites documents the external follow-up. No VERDICT review or failing required check exists.
alternatives: Controller must resolve the documented agentic-job follow-up and supply authenticated evidence, or authorize a revised protocol in trusted task context. add_comment is unavailable in this run; no push requested.

@cgwalters-bot

cgwalters-bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Fresh continuation round 3, after 2, for 3b28d0c83364: run.

Generated-by: https://github.com/cgwalters/#llms

@cgwalters-bot cgwalters-bot added the needs-human The poll stopped here; a person decides what is next label Nov 14, 2026
@cgwalters-bot

Copy link
Copy Markdown
Collaborator Author

The poll stopped following this pull request: continuation round 3 is blocked: its run https://github.com/cgwalters-forge/tracker/actions/runs/43528708021 ended (success) and pushed no new head for 3b28d0c (its checkpoint says incomplete). It is labelled needs-human; remove the label to have the poll start again on it, with fresh fix rounds.

Generated-by: https://github.com/cgwalters/#llms

cgwalters-bot added a commit that referenced this pull request Nov 14, 2026
## Executive summary

Move Check's tool provisioning into ordinary pinned, checksum-verified, bounded job-local setup and remove the Rust installer and its CLI/Justfile interfaces. Keep offline eval and configuration behavior intact.

Part of #76

The refreshed caller audit is in README.md: no active external installer dependency was found in current tracker, agentic-job or agent-board default branches; deployed and other-branch callers remain unverified. No cross-repository changes or dispatch installation were made. Eval relocation and final Cargo/tracker package removal remain later slices; PR #87 and its human-approval hold are untouched. The audit is durable in README.md; this run cannot update the issue body's Findings link.

Verified: just check, reference/dependency audit, and controlled cold-cache network provisioning with all version/navigation smoke checks and checksum-mismatch rejection before extraction. GitHub-hosted CI itself has not run here; no paid evals or hosted role probes ran.

Review: one adversarial pass, no blockers; its download-size cap, test-only tempfile and stale constraint-comment findings were addressed afterward and rechecked; no follow-ups remain.

Assisted-by: https://github.com/cgwalters/cgwalters#llms
@cgwalters-bot cgwalters-bot removed the needs-human The poll stopped here; a person decides what is next label Nov 15, 2026
@cgwalters-bot

cgwalters-bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

Continuation round 1 of 2 for 3b28d0c83364: run.

Generated-by: https://github.com/cgwalters/#llms

@cgwalters-bot

cgwalters-bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

Fresh continuation round 3, after 2, for 3b28d0c83364: run.

Generated-by: https://github.com/cgwalters/#llms

@cgwalters-bot cgwalters-bot added the needs-human The poll stopped here; a person decides what is next label Nov 15, 2026
@cgwalters-bot

Copy link
Copy Markdown
Collaborator Author

The poll stopped following this pull request: continuation round 3 is blocked: its run https://github.com/cgwalters-forge/tracker/actions/runs/43758959261 ended (success) and pushed no new head for 3b28d0c (its checkpoint says incomplete). It is labelled needs-human; remove the label to have the poll start again on it, with fresh fix rounds.

Generated-by: https://github.com/cgwalters/#llms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-human The poll stopped here; a person decides what is next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant