Rebuild Phase 0 — skeleton, contracts, and CI

[goelalex]

Restructures the repo into src/python + src/csharp and adds a manifest-driven Python build driver whose migration ledger applies the ordered DDL for five schemas (etl, intake, omop, naaccr, sdc) to SQLite and SQL Server, so build is safely rerunnable. Trims C# to SDC XML import only — deleting the HL7, FHIR, and vocabulary-stub code — and removes the PostgreSQL dialect and its container wiring, keeping the vendored OHDSI files present but unapplied. Adds contracts/: the versioned envelope schema, its serialization profile, and five frozen importer outputs that preserve what the deleted C# HL7 tests asserted, plus three CI jobs running on pull requests and pushes to main. Settles the sdc_form_answer column contract (response holds the raw source lexeme, datatype names which typed column holds the parsed value), renames reponse_string_nvarchar to response_string, guards numeric parsing with TryParse, and makes --dry-run genuinely read-only on both dialects. Also renames NAACRToOMOPmaps to NAACCRtoOMOPmaps and deletes phenoml-workflows, brought forward from Phase 4 with REBUILD_PLAN.md amended to match.

Closes #90. Follow-ups noted on #91, #93, #94, and #96.

🤖 Generated with Claude Code

[rmoldwin]

I asked Copilot to summarize this PR, and will copy it here. Next step after this comment is the code review.

Here’s the user-focused summary of PR #97 (IHE-SDC-WG/SDC-CDM) in plain language.
What this PR does (in practical terms)

This PR is a major simplification and reset of the project’s first build phase so it is easier to run, verify, and maintain.

If you’re using this software, the key outcome is:

the project is reorganized,
setup/build behavior is more predictable,
only currently supported import paths are kept active,
and expected output data is now tracked with versioned contracts and fixed examples.

Why it matters to users

From a user perspective, this PR is about reliability and clarity:

Builds are safer to rerun
    Running build steps repeatedly should no longer leave the database in a bad state.

Fewer half-supported paths
    Unused or not-yet-ready pieces were removed from active code paths, so there’s less confusion about what actually works now.

More stable expected outputs
    The PR adds pinned output examples and schema contracts so behavior can be checked consistently across changes.

CI checks are expanded
    Pull requests and pushes to main now run the new checks, so breakages are caught earlier.

Biggest changes you’ll notice

1. Repository structure was rebuilt

The codebase is reorganized into:

src/python
src/csharp

This makes it clearer where to look depending on language and role.
2) Build process now follows an ordered migration list

A Python build driver now applies database schema changes in a defined order for these areas:

etl
intake
omop
naaccr
sdc

It supports SQLite and SQL Server, and is intended to be rerunnable.

User impact: running setup/build is more repeatable and less fragile.
3) C# scope is narrowed to SDC XML import only

C# paths for HL7, FHIR, and vocabulary-stub logic were removed in this phase.

User impact: less ambiguity about supported ingestion in current phase; focus is on SDC XML path.
4) PostgreSQL path removed from this phase

PostgreSQL-specific dialect/container wiring was removed from active flow.

User impact: current tested path is focused on SQLite + SQL Server only in this PR’s scope.
5) “Contracts” added for stable output checking

A contracts/ area is introduced with:

a versioned envelope schema,
serialization profile,
and frozen importer outputs (reference examples).

These preserve expected behavior previously covered by removed tests.

User impact: easier to confirm that outputs are still in the expected shape over time.
6) CI pipelines added/updated

Three CI jobs are now configured for PRs and pushes to main.

User impact: higher confidence that main workflows remain healthy before/after merges.
7) Data field behavior clarified/fixed

For sdc_form_answer:

response stores the original source value as-is,
datatype indicates which typed column holds the parsed value.

Also includes:

typo fix in column name (reponse_string_nvarchar → response_string),
safer number parsing,
and true read-only behavior for --dry-run.

User impact: more predictable data interpretation and safer test runs.
Related cleanup

NAACRToOMOPmaps renamed to NAACCRtoOMOPmaps
phenoml-workflows removed
REBUILD_PLAN.md updated
Closes #90; follow-up items referenced in #91, #93, #94, #96.

Size / risk profile (for review planning)

This is a large PR:

135 files changed
+4,571 / -16,499

So this is primarily a restructure + scope reduction + build/contract foundation PR, not a small feature tweak.

If you want, next I can give you a review checklist focused on user-facing risk (what could break for someone running imports/builds), without diving into low-level implementation details yet.

---

-------------More RM questions: ---------------------------

What CI checks are run on PR and Push?
What code functionality was removed?
What are the currently supported import paths (including data source and endpoint)?
What are the pinned output examples and schema contracts? What do you mean by the generic word "Schema"?
Describe the supported intake and ETL paths (include source, target).
Define "contract" and " versioned envelope schema"
Why was phenoml-workflows removed?
Delineate each repo function (e.g., running the DDL and import/ETL pipelines) and assess whether it is likely to work or fail in this PR. I do not expact thngs to work well at this early stage. I just want a listing of the current (probable) status for each function at this stage.

Remember to examine these questions and the status report model when I later ask you to review the PR.

---

Great set of questions. I can answer them accurately, but I need to pull the PR’s actual file diffs/check runs/comments first so I don’t guess.

I’ll do that next and then give you:

CI checks on PR vs push
Removed functionality
Supported import paths (source → endpoint/target)
Exact pinned outputs + contract files
Intake and ETL paths (source → target)
Clear definitions of “contract” and “versioned envelope schema”
Why phenoml-workflows was removed (based on evidence in PR/discussion)
A function-by-function status report: likely works / likely partial / likely broken, with rationale

And yes — I’ll keep this as the baseline model for your later formal PR review.

---------------------Mr AI needs a litle push--------------------------
do it.

Excellent — I pulled the PR content and can now answer directly.

1. What CI checks are run on PR and push?

From the workflow files added in this PR:

CI workflow (.github/workflows/ci.yml)
    Runs on:
        every pull request
        every push to main
    Jobs:
        python-sqlite: install Python deps and run pytest
        csharp-sdc: run dotnet test src/csharp/SdcCdm.Sdc.Tests

SQL Server CI workflow (.github/workflows/sqlserver.yml)
    Runs on:
        PRs only when specific files change (database/**, src/python/**, pyproject.toml, workflow file)
        pushes to main with the same path filters
    Job: 3. python-sqlserver: start SQL Server container, install ODBC + deps, run Python tests (SQLite + SQL Server)

So in plain terms: 2 always-on checks + 1 SQL Server check when relevant files change.
2) What code functionality was removed?

Major removals are very broad:

C# HL7 importer path removed
    ImportNaaccrVolV.cs deleted
C# FHIR importer/exporter removed
    FHIR/Importers.cs, ExportFhirCpds.cs, related tests removed
General C# OMOP/NAACCR pipeline implementation removed
    ISdcCdm.cs, SdcCdmInSqlite.cs, bridge helpers and many tests removed
PostgreSQL runtime path removed
    Postgres DDL + Dockerfile/docker-compose + env sample removed from active execution path
phenoml-workflows removed (plus .gitignore entries referring to it)

What remains on C# side is intentionally narrow: SDC XML import/testing only.
3) What are currently supported import paths (source → endpoint/target)?

At this PR stage:

SDC XML (C# path)
    Source: SDC XML templates/responses
    Target: SDC tables (isolated C# SQLite store; not the whole Python manifest-driven pipeline)

HL7/NAACCR path
    In C#: removed
    In Python: planned/under migration with contracts/golden outputs for conformance
    So effectively: not an actively shipping C# import path in this PR

FHIR import path
    Removed from C#

If you mean “production-ready ingestion path after merge,” it is best described as:

Actively supported now: SDC XML via C# library tests
Pipeline direction: Python-led build/ETL foundation with contracts for future importer parity

4. Pinned output examples and schema contracts; what “schema” means here
   Pinned output examples

Under contracts/golden/:

obx-Adrenal.naaccr_value.json
obx-Adrenal.measurement.json
obx-Adrenal.importer_boundary.json
obx-Adrenal.obr_date_fallback.json
24-11-000312-2.sdc_report.json
plus a README.md explaining provenance

These are frozen expected snapshots from prior behavior and used as regression targets while porting.
Schema contracts

Under contracts/:

envelope.schema.json
SERIALIZATION.md

These define:

the expected structure and required fields of canonical intake envelope JSON,
and exact serialization rules so outputs are deterministic.

“Schema” (disambiguated)

In this PR, “schema” can mean two different things:

Database schema: logical DB namespaces/tables (etl, intake, omop, naaccr, sdc)
JSON schema: formal contract for document shape (contracts/envelope.schema.json)

You asked for clarity — that’s the distinction.
5) Supported intake and ETL paths (source → target)
Intake (foundation in PR)

Source data payload → intake.inbound_message
diagnostics → intake.inbound_message_diagnostic
patient identity → intake.patient

ETL/build execution

Manifest-driven DDL application (database/manifest.json) via Python build command
    source: DDL files in database/schemas/...
    target: SQLite attached DBs / SQL Server schemas
Bridge ETL scripts remain in database/etl/sqlite and database/etl/sqlserver and are tested via Python workflow context

6. Define “contract” and “versioned envelope schema”

   Contract: a stable, testable agreement for data structure/behavior that future code must obey.
   Versioned envelope schema: the canonical intake JSON shape has an explicit version marker (e.g., envelope_version) so changes can be introduced safely over time without ambiguity.

7. Why was phenoml-workflows removed?

Based on PR docs:

The repository is narrowing scope in Phase 0.
Workflow/review functionality previously in phenoml-workflows is moved out of this repo’s active architecture plan.
Mapping/review responsibilities are being shifted toward tracked repo artifacts/contracts and later-phase SQL/seed workflows.

So: removed as part of simplification and scope control, not as a random deletion.
8) Repo functions and likely status (works/partial/fail)

Here is the practical status model you asked for:

Manifest-driven DB build (Python)
    Likely status: Works
    Evidence: new build docs, tests, and CI coverage for SQLite + SQL Server.

Migration ledger / rerunnable DDL
    Likely status: Works (core behavior)
    Evidence: etl schema_migration + rerun semantics explicitly implemented and tested.

Python tests in SQLite
    Likely status: Works
    Evidence: dedicated CI job.

Python tests in SQL Server
    Likely status: Mostly works, early-stage risk medium
    Evidence: dedicated SQL Server CI job exists; still early architecture churn.

C# SDC XML import library/tests
    Likely status: Works for scoped functionality
    Evidence: separate csharp-sdc job and narrowed scope.

C# HL7 import
    Likely status: Removed / unavailable
    Evidence: importer and tests deleted; replaced by golden contracts for Python port.

C# FHIR import/export
    Likely status: Removed / unavailable
    Evidence: related code and tests deleted.

PostgreSQL execution path
    Likely status: Unsupported
    Evidence: docker and active dialect wiring removed; retained files are reference-only.

Bridge NAACCR+SDC → OMOP scripts
    Likely status: Partial but intended to run
    Evidence: bridge scripts retained + tests mention rerun regression, but overall rebuild phase is explicitly early.

End-to-end “all old functionality” parity
    Likely status: Not yet
    Evidence: this PR is explicitly a skeleton/foundation and removes broad legacy functionality.

Yes — I’ve now captured these questions and this status-report format as the baseline for your later PR review request.

[Copilot]

Pull request overview

Restructures the repository into a Phase 0 “skeleton” with a manifest-driven Python database build, a migration ledger for safe re-runs, a reduced C# surface limited to SDC XML persistence, and CI that validates SQLite + SQL Server builds—while removing the PostgreSQL dialect and related wiring.

Changes:

• Added Python sdc_cdm build CLI with manifest validation, SQL script splitting, SQLite/SQL Server backends, and a migration ledger to make builds idempotent and rerunnable.
• Reworked the database model to five logical schemas (etl, intake, omop, naaccr, sdc), updated DDL/docs/diagrams accordingly, and removed PostgreSQL DDL + container setup from active paths.
• Trimmed C# to SDC XML template/row-data import + SQLite store only, updated solution/projects/tests, and added GitHub Actions CI for Python + C# + SQL Server.

Reviewed changes

Copilot reviewed 126 out of 135 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/tests/test_three_schema_sqlite.py	Removes PostgreSQL/Docker bootstrap assertions; updates column rename contract assertion.
tools/tests/test_load_athena_vocab.py	Drops PostgreSQL backend coverage and seed-concept behavior; updates rollback expectations.
tools/tests/test_convert_naaccr_omop_maps.py	Updates default workbook directory rename (NAACCRtoOMOPmaps).
tools/requirements-vocab.txt	Removes PostgreSQL driver dependency (psycopg).
tools/load_athena_vocab.py	Removes PostgreSQL backend, tightens “fresh target” requirements, updates bridge-required concept handling.
tools/convert_naaccr_omop_maps.py	Updates mapping-workbook default directory and removes phenoml-workflows boundary language.
tools/ccr_labreport_to_naaccr.py	Updates CLI description to “five-schema store”.
src/python/tests/test_sqlscript.py	Adds tests for SQLite/SQL Server script splitting and pins executable unit counts for manifest entries.
src/python/tests/test_omop_ddl.py	Adds SQL Server OMOP DDL identity-column regression test.
src/python/tests/test_no_postgres.py	Enforces that removed-dialect text is confined to declared reference paths.
src/python/tests/test_migration_ledger.py	Adds unit tests for migration decision policy (skip/reapply/accept-hash/mismatch).
src/python/tests/test_manifest.py	Adds manifest ordering + exclusion validation tests.
src/python/tests/test_contract.py	Adds envelope schema strictness checks and golden snapshot assertions.
src/python/tests/test_build.py	Adds CLI/build idempotency + dry-run immutability tests and schema file layout checks.
src/python/sdc_cdm/vocab/init.py	Placeholder package for Phase 1 vocabulary support.
src/python/sdc_cdm/hl7v2/init.py	Placeholder package for Phase 3 HL7 v2 parsing support.
src/python/sdc_cdm/export/init.py	Placeholder package for Phase 5 export support.
src/python/sdc_cdm/envelope/init.py	Placeholder package for Phase 3 envelope support.
src/python/sdc_cdm/db/sqlserver_backend.py	Implements SQL Server backend (pyodbc) for manifest-driven builds.
src/python/sdc_cdm/db/sqlscript.py	Adds dialect-aware SQL splitting utilities (SQLite transaction stripping, SQL Server GO batching).
src/python/sdc_cdm/db/sqlite_backend.py	Implements attached-database SQLite backend with read-only dry-run behavior.
src/python/sdc_cdm/db/run_log.py	Adds etl.run logging for build runs (SQLite + SQL Server semantics).
src/python/sdc_cdm/db/paths.py	Adds repo path helpers (repo root + manifest path).
src/python/sdc_cdm/db/manifest.py	Adds manifest loader/validator enforcing ordering and complete DDL coverage/exclusions.
src/python/sdc_cdm/db/ledger.py	Adds migration ledger + hash-change policy with --accept-changed-hashes.
src/python/sdc_cdm/db/errors.py	Adds stable error types for CLI exit semantics.
src/python/sdc_cdm/db/backend.py	Defines backend interface used by build driver.
src/python/sdc_cdm/db/init.py	Package marker for DB tooling.
src/python/sdc_cdm/cli/main.py	Implements sdc-cdm CLI with build command and dry-run/list modes.
src/python/sdc_cdm/cli/build.py	Implements manifest application, ledger decisions, and run logging.
src/python/sdc_cdm/cli/init.py	Package marker for CLI entrypoints.
src/python/sdc_cdm/main.py	Enables python -m sdc_cdm entrypoint.
src/python/sdc_cdm/init.py	Defines Python package metadata/version.
src/csharp/SdcCdm.sln	Renames solution projects to SdcCdm.Sdc + SdcCdm.Sdc.Tests and removes retired projects.
src/csharp/SdcCdm.Sdc/SdcCdm.Sdc.csproj	New trimmed C# library project with embedded SDC SQLite DDL resource.
src/csharp/SdcCdm.Sdc/ISdcCdm.cs	New trimmed interface contract reflecting SDC-only API surface and response_string rename.
src/csharp/SdcCdm.Sdc/ImportTemplateRowData.cs	Imports template row data CSV into SDC SQLite store (new implementation).
src/csharp/SdcCdm.Sdc/ImportTemplate.cs	Imports SDC template XML metadata into store (new implementation).
src/csharp/SdcCdm.Sdc.Tests/TemplateRowDataImporterTests.cs	Adds C# test verifying template row-data import creates expected template metadata.
src/csharp/SdcCdm.Sdc.Tests/SdcCdm.Sdc.Tests.csproj	Updates test data inclusion and references trimmed C# project only.
src/csharp/README.md	New C# scope documentation: SDC XML only, no pipeline/bridge/SQL Server store.
SdcCdmLib/SdcCdmInSqlite/TestImport.cs	Removes legacy manual test harness executable.
SdcCdmLib/SdcCdmInSqlite/SdcCdmInSqlite.csproj	Removes retired SQLite implementation project.
SdcCdmLib/SdcCdm/SdcCdm.csproj	Removes retired “core” C# library project (HL7/FHIR/vocab stubs).
SdcCdmLib/SdcCdm/ISdcCdm.cs	Removes retired expansive interface and the misspelled response column API.
SdcCdmLib/SdcCdm/ImportXmlForm.cs	Removes retired XML form importer implementation from old library.
SdcCdmLib/SdcCdm/ImportCsv.cs	Removes retired CSV vocab importer stub.
SdcCdmLib/SdcCdm/File/Handler.cs	Removes retired helper enum used by deleted FHIR parser.
SdcCdmLib/SdcCdm/FHIR/Parse.cs	Removes retired FHIR parsing helper.
SdcCdmLib/SdcCdm/FHIR/Importers.cs	Removes retired FHIR importer stubs.
SdcCdmLib/SdcCdm/FHIR/Converters.cs	Removes retired FHIR→OMOP converter stub.
SdcCdmLib/SdcCdm.Tests/VocabImporterTests.cs	Removes retired C# vocab importer tests.
SdcCdmLib/SdcCdm.Tests/FhirCpdsExporterTests.cs	Removes retired FHIR exporter tests and redundant template-row-data test.
SdcCdmLib/README.md	Removes outdated documentation for retired C# library layout.
sample_data/ecp_query_examples.sql	Updates comments to reflect five-schema model and removed PostgreSQL.
README.md	Rewrites top-level README for manifest-driven build and five-schema layout; documents tool support and testing.
pyproject.toml	Adds Python package definition, optional deps, scripts entrypoint, and pytest configuration.
notebooks/serve_db.py	Updates Datasette-lite URL generation for five attached SQLite schema files.
notebooks/README.md	Removes .NET notebook section and updates attached-file documentation for five-schema build.
notebooks/python_cdm_utils/import_vol_v_message_sqlite.py	Adds provenance commit reference in docstring (legacy utility).
notebooks/python_cdm_utils/import_sdc_form_sqlite.py	Updates response/typed-column contract (response + datatype + response_string).
notebooks/python_cdm_utils/crud_sqlite.py	Renames reponse_string_nvarchar to response_string in helper API/SQL.
ECP_OMOP_MAPPING.md	Updates prose for five-schema model and Phase boundaries (no active HL7 importer in Phase 0).
diagrams/three-schema/three-schema-overview.mmd	Updates ERD overview for five schemas and new intake/etl relationships.
diagrams/three-schema/sdc.mmd	Documents response/datatype contract and renames to response_string.
diagrams/three-schema/naaccr-sdc-to-omop-bridge.mmd	Updates bridge notes to reflect Phase 0 execution approach and removed Postgres text.
diagrams/README.md	Updates diagrams guidance and adds etl/intake source-of-truth DDL references.
database/vocab/README.md	Updates vocab-loading instructions for manifest build; removes PostgreSQL section and adjusts dialect guidance.
database/update-ddl-files.py	Removes tombstone script (retired generator guard).
database/schemas/sdc/ddl/sqlserver/1_sdc_sqlserver_ddl.sql	Renames reponse_string_nvarchar to response_string.
database/schemas/sdc/ddl/sqlite/1_sdc_sqlite_ddl.sql	Renames reponse_string_nvarchar to response_string.
database/schemas/sdc/ddl/postgresql/1_sdc_postgresql_ddl.sql	Removes PostgreSQL SDC DDL (dialect removed).
database/schemas/omop/VENDORED.md	Adds provenance notes and local modification rules for vendored OMOP DDL.
database/schemas/omop/ddl/sqlserver/1_OMOPCDM_sqlserver_5.4_ddl.sql	Applies SQL Server identity/type adjustments required for rerunnable builds and bridge assumptions.
database/schemas/naaccr/ddl/sqlserver/0_naaccr_dictionary_sqlserver.sql	Updates dialect text (“both supported dialects”).
database/schemas/naaccr/ddl/sqlite/1_naaccr_sqlite_ddl.sql	Updates dialect text (“both supported dialects”).
database/schemas/naaccr/ddl/postgresql/1_naaccr_postgresql_ddl.sql	Removes PostgreSQL NAACCR DDL (dialect removed).
database/schemas/intake/ddl/sqlserver/1_intake_sqlserver_ddl.sql	Adds SQL Server intake schema DDL (inbound_message, diagnostics, patient).
database/schemas/intake/ddl/sqlite/1_intake_sqlite_ddl.sql	Adds SQLite intake schema DDL for attached database layout.
database/schemas/etl/ddl/sqlserver/1_etl_sqlserver_ddl.sql	Adds SQL Server etl schema DDL (run log, migration ledger, constants).
database/schemas/etl/ddl/sqlite/1_etl_sqlite_ddl.sql	Adds SQLite etl schema DDL for ledger + run log.
database/README.md	Updates database docs to five schemas and documents manifest-driven build workflow.
database/naaccr_omop/README.md	Updates mapping workbook directory rename and removes phenoml-workflows review guidance.
database/naaccr_omop/naaccr_omop_extension_mapping_spec.json	Updates phenoml-workflows boundary string and workbook path renames.
database/manifest.json	Adds ordered per-dialect manifest plus explicit excluded reference files.
database/Dockerfile	Removes PostgreSQL container init wiring.
database/docker-compose.yml	Removes PostgreSQL docker-compose wiring.
database/.env.example	Removes PostgreSQL docker env example.
contracts/SERIALIZATION.md	Adds canonical envelope JSON serialization profile.
contracts/golden/README.md	Documents frozen “golden” importer outputs and their provenance.
contracts/golden/obx-Adrenal.obr_date_fallback.json	Adds frozen snapshot for OBR-date fallback behavior.
contracts/golden/obx-Adrenal.naaccr_value.json	Adds frozen snapshot for naaccr_value outputs.
contracts/golden/obx-Adrenal.measurement.json	Adds frozen snapshot for bridged measurement outputs.
contracts/golden/obx-Adrenal.importer_boundary.json	Adds frozen snapshot asserting importer boundary (no SDC XML form tables).
contracts/golden/24-11-000312-2.sdc_report.json	Adds frozen snapshot covering blank narrative fallback behavior.
.gitignore	Removes ignores related to deleted phenoml-workflows outputs/venv.
.github/workflows/sqlserver.yml	Adds SQL Server CI job with mssql service, ODBC driver install, and pytest run.
.github/workflows/ci.yml	Adds Python SQLite + C# SDC jobs and enforces no cross-language dependency execution.
phenoml-workflows/workflows/naaccr-to-omop.workflow.json	Removes phenoml-workflows workflow definition (package deleted).
phenoml-workflows/sample/naaccr-case.example.json	Removes phenoml-workflows sample input (package deleted).
phenoml-workflows/requirements.txt	Removes phenoml-workflows dependencies list.
phenoml-workflows/README.md	Removes phenoml-workflows documentation.
phenoml-workflows/pyproject.toml	Removes phenoml-workflows packaging metadata.
phenoml-workflows/phenoml_workflows/run_workflow.py	Removes phenoml-workflows CLI harness.
phenoml-workflows/phenoml_workflows/review_store.py	Removes phenoml-workflows review store implementation.
phenoml-workflows/phenoml_workflows/review_schema.py	Removes phenoml-workflows review schema helpers.
phenoml-workflows/phenoml_workflows/mapper.py	Removes phenoml-workflows mapping logic.
phenoml-workflows/phenoml_workflows/json_io.py	Removes phenoml-workflows JSON IO utilities.
phenoml-workflows/phenoml_workflows/config.py	Removes phenoml-workflows credential/config handling.
phenoml-workflows/phenoml_workflows/init.py	Removes phenoml-workflows package marker.
phenoml-workflows/.env.example	Removes phenoml-workflows env example.
notebooks/try_sdc_cdm_dotnet.dib	Removes .NET polyglot notebook.
.vscode/launch.json	Removes editor launch configuration for deleted .NET console target.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.