fix(selfdeploy): plan-bound shapes, loopback http, owned-only CVM teardown, fail-loud CVM counting - #50
Closed
echobt wants to merge 38 commits into
Closed
fix(selfdeploy): plan-bound shapes, loopback http, owned-only CVM teardown, fail-loud CVM counting#50echobt wants to merge 38 commits into
echobt wants to merge 38 commits into
Conversation
Guest CVMs fetch the miner ZIP over a short-lived bearer grant instead of baking bytes into the image. Compose allowlists the delivery env names so the generator hash stays intentional. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Guest hashes download and executed bytes; schema-v2 envelopes carry guest_artifact_proof; host scoring rejects success results whose proof describes a different artifact than the submission (distinct reason codes). Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Static app_id pins only worked for the operator account. Discovery omits nonce and app_id so Phala mints the account-derived handle; live 422 showed nonce without app_id is illegal. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Reference miner loop and ZIP builder for submission path exercises against a controlled hotkey without on-chain registration. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Optional background task posts winner-take-all weights to master with a ledger so retries stay idempotent; wires eval artifact router on create_app. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Persist image digest allowlist and one-time attestation nonces so master can gate Lium/constation evidence without in-memory-only state. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Shared attestation payload, nonce issue/consume, digest allowlist checks, and constation poller/runner so pod evidence can be corroborated offline. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Wire nonce/allowlist/bundle endpoints through app proxy and CLI helpers so workers can complete the Lium attestation handshake end-to-end. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ingest and check constation bundles before accepting prism results; attestation routes and breakglass keep offline characterization intact. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Prod hotpatch refuses multi-GPU hosts that reject gpu_count=1 splits and exposes LiumClient.for_prism_training for the capacity scheduler path.
Bring prod hotpatch modules for master-owned training admission (LiumCapacityScheduler) and orphan pod cleanup, plus settings wiring helpers.
Prod hotpatch settings surface for fail-closed constation orchestration and optional master-owned Lium training admission (default off).
…ules Hotpatched allowlist_repository gains get_active_pin and identity payload mapping. Also restore image-resident custody/orchestrator/pod_binding/ attestation_keys/bundle_seal/routes pieces required by the prod main wiring but missing from this branch tip.
…ration Prod hotpatch adds ConstationPinSource stamping on Prism dispatch and optional LiumCapacityAdmission.tick during run_once.
…aster Prod hotpatch builds custody runtime, pre-forward hook, pin source, and try_build_lium_capacity_scheduler when constructing the orchestration driver.
Prod entrypoint hotpatch (and fix/master-embed-ac-env-file) restore the operator extension point dropped by env -i isolation so attestation and review switches from embed.env actually reach embedded challenges. Host-only Prism policy defaults are intentionally not copied.
…ateway-free analyzer Prod hotpatches: REPO_ROOT finds docker/review when installed in site-packages; host analyzer skips missing Base LLM gateway parking and completes AST+similarity under gateway-free attested product mode.
Prod hotpatch modules: safe pod boot env/install planning, and the OpenRouter dual-gate plagiarism adjudicator (wired when queue lands).
Prod hotpatch: PrismSettings plagiarism_llm_* / openrouter fields, and CheckpointIntakeService last_status observability with fail-closed publish.
Prod hotpatch no longer parks missing Base LLM gateway as llm_standby. Update the two contracts to expect first-pass allow and no requeue loop.
… delete Prod bucket-D shapes/lifecycle/phala fixes: stage disk bounds, eval tdx.xlarge default, disk-aware budget, and PhalaCloudClient.delete_cvm so validators can tear down without a phala binary.
…nstalls Prefer env/manifest and /app|/opt golden layouts over Path(__file__).parents[3] so eval/prepare no longer 503s when the package is installed under site-packages.
…d compose envs Land miner n_concurrent plan binding, optional hydration_digest, mid-run progress validators, and EVAL_PROGRESS/DSTACK_DOCKER allowed envs. Keep guest_artifact_proof, RA-TLS KR authority checks, dual residual review gate, and raw-weight-push settings from the repo side.
…handoff Provision disk_size without re-pinning app_id or reintroducing nonce. CLI gains HTTP teardown, disk flags, and eval run-token handoff while keeping multi-account discovery and artifact grant minting.
… prompt Land prod tool-call contract wording so Grok tool_choice=auto emits a closed submit_verdict call. Keep the joinbase REVIEW_API_BASE_URL pin.
When worker_plane is enabled the master queue no longer claims GPU work. Borderline/attach plagiarism pairs go through the OpenRouter adjudicator instead of terminal-rejecting the quarantine band.
…tures Add behavioral tests for digest path, n_concurrent, progress wire, disk guards, and dual residual signature. Update compose pin hash and lifecycle/disk/teardown expectations for the landed product changes.
…errors Omit --eval/--review-instance-type to follow the plan; explicit mismatch still fails closed with an operator-safe message instead of AttributeError. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
http:// is accepted only for 127.0.0.1/localhost/::1 when SELFDEPLOY_ALLOW_INSECURE_LOOPBACK=1; all other cases stay https-only. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…licy Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…lowlist Search KR/allowed_envs candidates so prod pin daf0f209… can be reproduced, and encrypt only secrets listed on the measured compose allowlist. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Track created ids per run and in work/owned_cvms.txt; refuse foreign ids in phala_delete_cvm; log the teardown plan; add --dry-run-teardown proof. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ship compose + example/pin fixtures; keep tokens, KR PKI, challenge.env, and runtime data/work dirs out of git. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Prefer run_staging.sh over SSH to prod; note signature-only keypairs and that real Phala CVMs must always be torn down (owned ids only). Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Deploy acks record vm_uuid while GET /cvms returns id=cvm_*; map owned track entries through listing aliases so teardown can delete what it created without selecting foreign CVMs. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Legacy GET /cvms parsing treated unrecognized envelopes as an empty list, under-reporting live spend as count 0. Pin CLI API version 2026-06-23, list via /cvms/paginated, and raise on unknown shapes so teardown confirmation never succeeds when the count is indeterminate.
Add prod-compose-upgrade.md with the derived daf0f209→9a550b2d delta (allowed_envs + image), ordered production change list, measurement capture rules, rollback, and guest_artifact_proof verification plan. Cover harvested dstack-client-trust.crt under staging gitignore with a placeholder .example. Staging runner also surfaces CVM-missing mid-poll. No production execution.
|
Important Review skippedToo many files! This PR contains 177 files, which is 77 over the limit of 100. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (178)
You can disable this status message by setting the ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
6 tasks
Contributor
Author
|
Superseded by #51 ( This PR (#50) mixed 38 commits of an older stack ( The replacement is a single focused commit on current
Closing in favor of #51. |
Contributor
Author
|
Closed: superseded by #51 (clean main-based replacement). See comment above. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Self-deploy fixes found by driving the real production eval flow today, a local staging harness, hardened Phala CVM teardown, and a documented upgrade path for the one remaining blocker.
Base branch carries earlier stack work (bucket-D, prism, constation/Lium). The new work in this PR is the last 10 commits (
8ee59da6..acc67b13).Verified fixes (selfdeploy)
format_eval_shape_mismatch_errorimplementedAttributeErrorinstead of a message — the function was called atcli.py:336but defined nowhereeval_app.measurement.vm_shape(prod:tdx.xlarge); CLI defaulted totdx.small, so every deploy died in the crash above. Explicit operator mismatch still fails closedhttp://opt-inSELFDEPLOY_ALLOW_INSECURE_LOOPBACK=1allowed_envspin;encrypt_eval_secretsscoped to the measured allowlistNo attestation check was weakened: compose_hash, measurement/RTMR, KMS digest, DCAP, run-token binding and
guest_artifact_proofare untouched.Safety: Phala CVM handling
Two real defects, both found the hard way:
run_staging.shdeleted a live production eval CVM mid-run, right after key release, permanently wedging submission 11 (eval_running,key_grant_state=granted,retryable=false, cancel/failure409). Teardown is now owned-only, refuses any CVM not provably created by the run, and logs its delete plan first.GET /api/v1/cvmsreturns[]even when inventory is non-empty; the authority is/cvms/paginatedwithX-Phala-Version: 2026-06-23. Parsing fell back to an empty list on unknown shapes, so "I don't understand the response" became "no CVMs running" — it produced a false no-spend claim while a CVM was live. Unknown shapes now raise, and an indeterminate count is treated as teardown failure, never success.Local staging harness
docker-compose.staging.yml,scripts/staging/,docs/staging.md. Drives the flow from any host with real CVMs. Secrets gitignored,.exampletemplates committed.Also confirmed: the production AC is publicly reachable at
https://chain.joinbase.ai/challenges/agent-challenge, which removes the need for SSH to the master validator and for any Cloudflare tunnel to receive guest callbacks.Blocking prerequisite for execution proof
Proven empirically on fresh submission 13 (submit → review CVM →
review_allowed→ eval prepare → eval deploy, all PASS):The production eval compose pin
daf0f209…was measured withoutCHALLENGE_PHALA_EVAL_ARTIFACT_{URL,TOKEN}inallowed_envs. The deploy'sencrypted_env_namescontained neither. Phala therefore cannot inject the artifact grant, the guest cannot download the miner ZIP, and a matchingguest_artifact_proofis structurally impossible on that pin — the earlier T8 run on the same pin produced none either.docs/prod-compose-upgrade.mddocuments the exact path: the compose delta (the artifact envs and an image change), an ordered production change list with exact setting keys, how the new measurements must be derived rather than invented, the fail-closed allowlist trap, rollback, and the verification run that would finally yield the proof.It also identifies a lower-risk intermediate: keeping the current prod image and adding only the two artifact envs yields
3a81feaf…, which unblocks the proof while changing a single variable.Not done
Verification
phala cvms list --json