Make the squashed course migration history safe for populated databases

[alexeygrigorev]

Outcome

Repair the Django courses migration graph so a release built from current main can safely migrate both a fresh database and a populated database whose migration history comes from the deployed legacy course release. Preserve course data and migration provenance without manually rewriting django_migrations, dropping/recreating tables, or importing a database dump as a workaround.

This is a narrowly bounded migration-history compatibility repair. It is not the Course → Cohort product migration, the full course-platform preservation/cutover rehearsal, or the content release digest repair in #219.

Normative authority and incident evidence

• Process and issue lifecycle
• 04 — Course-platform adoption and Course → Cohort model, especially the expand-and-contract migration and high-risk migration checks
• 09 — Migration and rollout controls
• 10 — Verification strategy
• Related course boundaries: #51 and #60

The exact incident inputs are frozen for this issue:

Evidence	Fact	Role in #220
Active deployed source 11b2bd1	The deployed course graph contains courses.0001_initial through courses.0041_courseregistrationcountsourcerun_and_more and represents the legacy migration history a populated database may record.	Compatibility lower boundary.
Squashing commit 4825aa3	Replaced the active course graph with a rewritten 0001_initial plus 0002–0010, deleting the legacy 0002–0041 modules. Its own adoption note says the squash was local-only and not a production upgrade path.	Introducing change under repair.
Current main b7c693e	The exact source that reached the failed deployment migration task.	Candidate/current target baseline.
CI run 32659885289	Application quality, Django, Playwright, container, screenshots, publish, and source-resolution jobs passed; the exact-image deployment migration stage started and the run failed.	Confirmed deployment evidence, not acceptance.
Deployment evidence artifact 9499226819	Redacted evidence records source_sha=b7c693e, migration stage contract_contradiction, failed_without_success_record, and retention of the prior release without web/worker promotion.	Failure classification and required regression evidence.

The failed deployment must not be “fixed” by retrying blindly, deleting migration rows, flushing the database, or treating a migration contract failure as an infrastructure success. Future evidence must identify whether a failure is a migration-graph/data-state defect or a runtime/infrastructure defect using the exact source SHA, immutable image identity, migration-task exit/error, applied migration-state snapshot, and controller/release result.

Frozen compatibility boundary

The implementation must support this exact boundary and no wider one:

1. Fresh: an empty database at migration zero reaches every current course leaf from the repaired graph.
2. Legacy populated: a synthetic, non-production database is created with the course migrations from 11b2bd1 applied through courses.0041_courseregistrationcountsourcerun_and_more, then populated with representative course rows and relationships. The repaired current graph must upgrade it without an inconsistent-history error, data loss, or silent schema replacement.
3. Repaired replay: a database already upgraded by the candidate can run the same migration command again and converge to an unchanged leaf with no duplicate backfill, duplicate rows, new side effects, or checksum/count drift.

The lower boundary is the exact deployed legacy migration identity above. The upper boundary is the candidate’s repaired current course leaf. Databases older than that deployed graph, future Course → Cohort migrations, and arbitrary hand-edited django_migrations tables are outside this issue.

The engineer may use a Django-compatible replacement/bridge/staged migration design after documenting its graph and state semantics. The implementation must preserve the meaning of already-applied legacy migration identities and must prove both fresh and populated behavior. It must not assume that a fresh 0001_initial name means the existing database has the new 0001 schema.

Scope

• Repair only the courses migration graph introduced by 4825aa3, including the compatibility boundary between the deployed legacy chain and the current squashed graph.
• Retain or bridge deployed migration history in a way Django can resolve for existing django_migrations state; do not silently discard the legacy chain.
• Add deterministic synthetic legacy-history fixtures/tests that exercise schema state, applied migration records, representative data, foreign keys, content types/permissions where applicable, and current leaf state.
• Prove safe migration behavior for fresh, populated-legacy, replayed-current, and failure/retry paths. Any data transformation must be an ordinary, historical-apps-only migration operation with a clear reversible/no-op policy; no raw runtime import or external side effect is permitted.
• Update the course migration adoption note only if needed to record the corrected compatibility contract and remaining HUMAN gate. Do not rewrite authoritative specs in this issue.
• Leave the current course application behavior, URLs, APIs, Studio, templates, email/Datamailer behavior, and other app migration histories unchanged.

Exact changed-path boundary

The frozen engineer candidate may change only these paths:

• courses/migrations/** — course migration graph/compatibility modules only;
• courses/tests/test_migration_history_compatibility.py — a dedicated test module if useful;
• test_support/tests/test_migrations.py — the existing migration harness, if extension is required;
• test_support/migration_seeds/courses-legacy-history-v1.json — synthetic, non-PII fixture data if a committed seed is required;
• _docs/adoption/course-platform/migration-squash-gate.md and _docs/adoption/course-platform/verification.md — factual gate/evidence updates only, if needed.

The engineer handoff must list every actual path and the exact base/head, plan digest, graph digest, evidence paths, and diff identity. No tracked path outside that allow-list is in scope. In particular, this issue must not change courses/models/**, views, templates, APIs, services, accounts/**, content/**, content_sync/**, events/**, Makefile, .github/workflows/**, pyproject.toml, uv.lock, Terraform/AWS configuration, or any database/export/protected data.

Explicit non-goals

• No Course → Cohort domain/model/relationship redesign or broader course-platform implementation from Migrate edition-like Course records to reusable Course → Cohort #51.
• No full production-like course data migration, route/API consumer migration, redirect/cutover, write-freeze/delta rehearsal, or release authorization from Preserve legacy course HTML/API compatibility and rehearse full data migration #60.
• No content release digest migration or content_sync behavior from #219.
• No destructive table recreation, DROP TABLE, TRUNCATE, flush, bulk delete/reinsert, manual django_migrations edits, or database dump restore as a compatibility strategy.
• No production database, AWS console/API, protected export, credential, real learner data, email/provider/Datamailer dispatch, or worker side effect.
• No dependency, CI workflow, Makefile, model, route, UI, API, schema-policy, or unrelated migration cleanup.
• No claim that the failed run 32659885289 was fixed, no deployment success claim, and no issue closure before the named HUMAN gates pass.

Dependencies and blocked work

• Required implementation inputs: the exact trees at 11b2bd1, 4825aa3, and b7c693e; the redacted CI/deployment evidence linked above; and the current migration/test harness. No live database is a dependency for engineering or tester work.
• Related, not blocking: Migrate edition-like Course records to reusable Course → Cohort #51 is the broader Course → Cohort migration and remains governed by its open decisions/dependencies; Preserve legacy course HTML/API compatibility and rehearse full data migration #60 is the broader preservation/cutover rehearsal and remains blocked by its own course consumer, mapping, and workflow gates. Make the squashed course migration history safe for populated databases #220 does not implement, accept, or unblock either issue.
• Release dependency: no release may deploy the current squashed course graph to a populated environment carrying the legacy chain until the automated scenarios pass and the exact-image HUMAN deployment/on-call gates below pass.
• If the repaired graph requires a product-model change or a data mapping decision owned by Migrate edition-like Course records to reusable Course → Cohort #51/Preserve legacy course HTML/API compatibility and rehearse full data migration #60, stop and re-groom rather than widening this issue.

Acceptance criteria

• Migration graph is explicit and compatible. The candidate documents the repaired graph, dependencies, replacement/bridge semantics, and current leaf. showmigrations/MigrationLoader finds no missing legacy dependency, duplicate leaf, or inconsistent applied-history state. No already-deployed migration identity is silently redefined as a different schema.
• Fresh database. An isolated empty database migrates from zero to every leaf with no error. makemigrations --check --dry-run reports no model drift, and the resulting schema matches the current models and expected course table/constraint contract.
• Populated legacy database. A deterministic fixture built from the exact 11b2bd1 legacy graph applies through courses.0041_courseregistrationcountsourcerun_and_more, inserts representative synthetic Course/enrollment/homework/project/review/score/statistics/certificate/registration data and relationships, then upgrades with the candidate. The run succeeds without InconsistentMigrationHistory, missing-table/column errors, guessed mappings, or destructive reset.
• Data and provenance preservation. Before/after evidence compares table row counts, primary keys, foreign keys, representative deterministic row checksums, content types/permissions where touched, and the complete applied migration set. No existing course row is silently dropped, duplicated, renumbered, or recreated under a new identity.
• Replay and retry are safe. Re-running the candidate migration on the upgraded populated database is a successful no-op: migrate --plan is empty at the leaf, row/checksum/migration-state snapshots are unchanged, no duplicate backfill occurs, and no email, provider, Datamailer, queue, or other external side effect is emitted. A failed/retried local migration either resumes deterministically or fails closed without partial destructive cleanup.
• Historical migration isolation. Every data migration uses historical apps models, is backend-portable, has bounded transaction/atomic behavior, and passes the repository’s migration import/side-effect isolation checks. No migration imports mutable runtime services, reads secrets, reaches the network, or depends on current application code that can drift later.
• Repository and regression verification. The engineer generates the versioned selective-verification plan from the frozen candidate, records exact identities/digests and all component dispositions, and leaves an uncommitted frozen worktree. The independent tester reruns the required focused migration/Django checks and make test-playwright-core; any render-impact classification triggers the required desktop/mobile screenshot gate, while a justified no-render classification is recorded as not_applicable per the process. No required component is partial or undocumented.
• Browser/HTTP compatibility smoke. On the synthetic migrated state, representative public course list/detail and private course workflow/API smoke requests retain expected status, canonical/compatibility behavior, no migration traceback, and no accidental data exposure. Desktop/mobile browser evidence is required when the selective plan marks those routes render-impacting.
• [HUMAN] Exact-image populated deployment. After commit, local merge, push, and a successful CI build, on-call deploys the exact immutable image/source identity to the authorized development environment and runs the migration task against an authorized populated development database copy initialized at the legacy boundary. Redacted evidence must include source SHA, image digest/version, migration task result, applied-history summary, release-controller result, and whether the prior release remained protected on failure. No production/AWS mutation is authorized by this issue’s engineering scope.
• [HUMAN] Readiness, smoke, and failure classification. On-call verifies database-aware readiness, web/worker identity parity, representative course HTTP/browser smoke, migration-task logs, and the release artifact. If the task fails, the evidence must distinguish a graph/data-state defect from a runtime/infrastructure defect; a controller rejection or missing verdict is unresolved, not green. The prior release remains the rollback target and no destructive database rollback is attempted.
• [HUMAN] CI/on-call lifecycle gate. The post-push CI run is observed by on-call to a terminal green result with no cancellation, missing required artifact, deployment contract contradiction, or unreviewed exception. The issue remains OPEN with human until these named gates are reported. If a technical commit is later authorized while the HUMAN gates remain, its body must use Refs #220, not Closes #220.

Required verification commands

Use the repository’s uv-backed commands only; all local migration databases and fixtures must be isolated under project-local .tmp/ or the deterministic test harness.

uv sync --locked
uv lock --check
make migrations-check
make test-migrations
make verification-quality
make test-django-full
make test-playwright-core
make verification-container
make verification-plan VERIFY_BASE_SHA=<frozen-base> VERIFY_HEAD_SHA=<frozen-head> VERIFY_ISSUE=220 VERIFY_WORKTREE=<frozen-worktree> VERIFY_CONSUMER=engineer VERIFY_PHASE=engineer
make verification-run VERIFY_ISSUE=220 VERIFY_WORKTREE=<frozen-worktree> VERIFY_CONSUMER=tester VERIFY_PHASE=tester
make verification-evidence-check VERIFY_ISSUE=220
make verification-report-check VERIFY_ISSUE=220 VERIFY_PHASE=tester

The handoff must include the exact commands actually run, terminal exit statuses, test counts, migration plan/output, redacted before/after snapshots, screenshot disposition/paths when applicable, and validated plan/evidence/report paths. make test-migrations and make migrations-check are required even if a broader target also invokes them. Do not run these commands against the shared development database.

Handoff and disposition

Engineering starts only after this issue is groomed and needs grooming is removed. The engineer must leave the candidate uncommitted and frozen for an independent tester; the tester must report every automated criterion separately; PM acceptance is a later lifecycle gate and is not granted by this issue rewrite. The current failed deployment evidence is preserved as the baseline incident and is not a success signal.

[alexeygrigorev]

PM grooming complete — ready for engineering

#220 is now genuinely groomed and ready for an isolated implementation lane. The disposition is a migration-history compatibility repair only: preserve the deployed 11b2bd1 courses.0001–0041 boundary, make the 4825aa3 squashed graph safe for fresh, populated-legacy, and replayed databases, and prove the result with synthetic redacted evidence. The issue has an explicit changed-path allow-list, non-goals, dependencies/related boundaries for #51/#60 and #219, required uv/Make verification, independent tester handoff, and separate exact-image deployment/on-call HUMAN gates.

The failed CI run 32659885289 and deployment artifact 9499226819 are preserved as incident evidence only; they do not constitute acceptance or deployment success. needs grooming has been removed. The issue remains OPEN with bug, P0, human, courses, operations, testing, and data-migration; no source, branch, commit, push, AWS, or database state was changed by grooming.

[alexeygrigorev]

Scope clarification for the migration repair: restoring the exact deployed courses.0001–0041 identities necessarily updates repository contracts that otherwise fail the required full Django verification. The candidate may therefore update only migration-compatibility provenance surfaces in addition to courses/migrations/**: core/tests/test_course_platform_adoption.py, scripts/validate_course_platform_schema_inventory.py, scripts/tests/test_course_platform_schema_inventory.py, _docs/adoption/course-platform/behavior-inventory.md, _docs/adoption/course-platform/integration-patched-files.tsv, and the existing migration-ID test courses/tests/test_curriculum_source_provenance_migration.py. No runtime app behavior, API, templates, CI, deployment, or dependency changes are authorized. The exact legacy graph and fresh-plan ordering must be covered by tests/docs.

[alexeygrigorev]

PM scope amendment (supersedes the earlier narrow allowlist for this compatibility repair): the migration graph cannot restore the deployed courses.0001–0041 identities while leaving the generated adoption/provenance contracts unchanged. The following eight files are therefore explicitly allowed, in addition to the migration files and six paths in the groomed issue: _docs/adoption/course-platform/behavior-inventory.md, _docs/adoption/course-platform/integration-patched-files.tsv, _docs/audits/2026-08-20-course-platform-schema-data-inventory-phase-1.md, core/tests/test_course_platform_adoption.py, courses/tests/test_curriculum_source_provenance_migration.py, scripts/render_course_platform_inventory.py, scripts/tests/test_course_platform_schema_inventory.py, and scripts/validate_course_platform_schema_inventory.py. These changes only update migration identity/provenance contracts and generated evidence; runtime application behavior, API, templates, CI, deployment, dependencies, and database operations remain out of scope. Tester should rerun the identity envelope against this amended allowlist.

[alexeygrigorev]

PM technical acceptance — automated slice ACCEPTED

I reviewed the final independent tester report at .tmp/issue-220-tester/independent-tester-report.md, the amended allow-list in comment 5388873002, and the exact uncommitted candidate at .tmp/issue-220-engineer.

Automated acceptance is complete:

• focused course migration compatibility: 6/6
• schema contract: 10/10
• adoption contract: 9/9
• make test-migrations: 18/18
• full Django: 2,485 passed, 9 skipped
• quality/evidence validation: 601/601
• final Playwright core: 209/209
• standalone container: 13/13 assertions PASS

The candidate identity and scope reconcile: base/head and origin/main are b7c693efbb4ea8c429fd3525d129bbd84504719d; the supplied candidate diff SHA is 680c52a3855f5aefc07c41b361d532538cd9e7ba7c2eb7553e4f5ad019a9c240; and the amended identity audit is 74/74 allowed paths with no violations. Source review confirms the legacy 0001–0041 compatibility boundary, the intentional unsquashed 0027 plan branch, the 0042 bridge, and repaired 0051 leaf. No runtime, API, dependency, CI, deployment, or database-operation scope drift was found.

The bounded composite runner timeout is retained as documented partial evidence, not treated as green; the completed browser-only run and standalone container artifact supply the final required component results.

Technical slice ACCEPTED. The engineer may commit exactly this frozen candidate, using Refs #220 semantics. Do not add paths or claim deployment success.

#220 remains OPEN and keeps the human label. The exact-image populated deployment, on-call readiness/smoke and failure classification, and post-push CI/on-call gates remain explicit HUMAN gates. This comment does not authorize deployment, database/AWS mutation, or issue closure.