Skip to content

fix(registry): migrate persisted operation ownership state - #607

Draft
wolfy-j wants to merge 3 commits into
mainfrom
fix/legacy-history-update-provenance
Draft

fix(registry): migrate persisted operation ownership state#607
wolfy-j wants to merge 3 commits into
mainfrom
fix/legacy-history-update-provenance

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Problem

Persisted registry history did not carry enough registry-owned state to replay ownership and dependency-root transitions independently of the application baseline. Update-first history could restore an entry without a corresponding ownership record, and provenance-only identity changes were not durable. Keeper's atomic snapshot then correctly rejected missing or conflicting ownership.

Fix

  • Store registry-owned transition state in one versioned Operation.Registry block. The public Entry and Lua entry shape carry no ownership/root field; format-1 decoding translates the retired wire flag directly into registry state. User entry meta and data remain unchanged.
  • Track the store-wide operation format in the existing schema migration ledger under registry_history.operations.
  • Migrate every changeset and branch atomically before replay, with exact blob digests, parent topology, complete-store coverage, and writer locking as compare-and-set preconditions.
  • Keep normal writes strict: create, update, and delete must carry exact current/previous registry state. Reconciliation records are migration-only and cannot be authored or reversed as live transitions.
  • Validate the same write contract in memory, SQLite, and PostgreSQL histories.
  • Preserve dependency-root selection through Lua registry writes and bind older dependency checkpoints to the current deployment only when their stored roots, references, selected modules, versions, and artifact identities still agree.
  • Fail closed on stale roots, partial history, concurrent changes, topology drift, malformed records, downgrades, and future formats.

There is no Keeper-specific lookup, user-metadata fallback, parallel ownership index, or second migration table.

Rollout

This is a one-way data cutover:

  1. Stop the old writer and retain a database snapshot.
  2. Start the new Runtime; it atomically migrates the registry operation stream and then boots.
  3. Rollback restores the database snapshot before starting the old binary.

An old binary is supported before cutover with the new Keeper. It is intentionally not allowed to write a migrated store it cannot understand.

Proof

  • Actual pre-cutover production binary booted the exact production baseline/history with the new Keeper: 2,914 entries, 329 history versions, /system 200.
  • Candidate migrated the exact production PostgreSQL history atomically to registry schema 1.3 / operation format 3; invalid published application graphs were rejected without partial application.
  • Fresh latest application resolved, reconciled 2,947 entries, served /system 200, shut down cleanly, then restored from persisted state and served 200 again.
  • Covered create/update/delete, dependency-root transitions, rollback/redo, sibling branches, update-first entries, 20,000-version depth, stale/partial/repointed plans, concurrent writers, SQLite/PostgreSQL parity, and unsupported formats.

Validation

  • go test ./... -count=1
  • affected packages with go test -race and PostgreSQL enabled
  • make lint — 0 issues
  • git diff --check

@wolfy-j
wolfy-j requested a review from skhaz August 27, 2026 03:20
@wolfy-j
wolfy-j requested a review from skhaz August 27, 2026 13:34
@wolfy-j
wolfy-j requested a review from skhaz August 27, 2026 19:02
@wolfy-j

wolfy-j commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Commit 3 — fix(registry): keep one resident identity across retained entries.

Second production defect after the boot fix: every keeper REST call answered 500 registry snapshot state failed: wippy/dataflow: ... module has conflicting provenance records. Root cause (reproduced in sim on the production history + app tree): the deployment lock pins wippy/dataflow 0.7.5 while the registry root resolves 0.7.14; userspace.dataflow:execution_identity was removed in 0.7.14 but is retained by operationPlanner.plan because app:dataflow_identity_binding still names it — and it stayed stamped with the 0.7.5 artifact identity, so one module carried two resident identities and ProvenancedState.Validate() refused the snapshot. Not latent in history (legacy fold yields 0 module-owned records); created at boot by baseline + resolution reconcile.

Fix in boot/deps/hub: residentIdentities folds the single artifact identity a module is resident at; adoptResidentIdentity moves a retained entry's record onto it (entry-scoped Root preserved); the planner emits a provenance-delta EntryUpdate for retained entries whose module moved. A module with no resident artifact left keeps its record. Strict invariants untouched. TDD: boot/deps/hub/retained_entry_provenance_test.go fails with the exact production string before the fix.

Proof over the production history: boot 0 errors, GET /keeper/hub/dependencies 200 (64 modules), POST .../plan 200 with a complete install payload; repeat boot converges (no re-emitted delta); pure-legacy and fresh-install conditions still pass; full boot provenance dump 70 modules / 5092 entries / zero conflicts; ./system/registry/... ./boot/deps/... ./api/registry/... green on sqlite and postgres.

Known, left visible: preserveImmutableResidentEntry compares Version only, never Digest — an immutable module resident at the same version but a different digest can fork identity the same way; not reachable on this data.

@wolfy-j

wolfy-j commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Correction to the previous comment: the retain-and-re-stamp fix ("keep one resident identity across retained entries") is withdrawn in favor of cde05d7 fix(deps): reject partial module identity updates, which refuses the fork at its source instead of re-stamping a retained entry with an artifact identity its content does not come from. The root-cause analysis stands: on the affected instance the deployment lock pins wippy/dataflow 0.7.5 while the registry root resolves 0.7.14, and app:dataflow_identity_binding still names userspace.dataflow:execution_identity, which 0.7.14 removed. The operator-side alignment for that instance is being verified in sim against the current branch HEAD; results follow. The regression test from the withdrawn commit (production shape: retained entry under a moved module) is worth carrying as an assertion of the new refusal — will propose it once the verdict is in.

@wolfy-j
wolfy-j requested a review from skhaz August 28, 2026 21:12
@wolfy-j
wolfy-j requested a review from skhaz August 28, 2026 22:47
@wolfy-j
wolfy-j requested a review from skhaz August 29, 2026 01:55
@wolfy-j
wolfy-j marked this pull request as draft August 29, 2026 02:05
@wolfy-j wolfy-j changed the title fix(registry): replay legacy histories whose entries first appear as updates; version the operation wire format fix(registry): migrate persisted operation ownership state Aug 29, 2026
@wolfy-j
wolfy-j force-pushed the fix/legacy-history-update-provenance branch 2 times, most recently from 332acbc to 4587dce Compare August 29, 2026 05:29
@wolfy-j wolfy-j closed this Aug 29, 2026
@wolfy-j
wolfy-j force-pushed the fix/legacy-history-update-provenance branch from 4587dce to 6d3f57d Compare August 29, 2026 14:19
@wolfy-j wolfy-j reopened this Aug 29, 2026
@wolfy-j
wolfy-j force-pushed the fix/legacy-history-update-provenance branch from 7a3a6cb to 4e8649d Compare August 29, 2026 18:29
@wolfy-j
wolfy-j force-pushed the fix/legacy-history-update-provenance branch from 8f55951 to 55eac93 Compare August 29, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants