PLAN: pilot migration — one dataset per hosting pattern to validate the datasets design

[mmcky]

Part of #336. Before graduating the datasets convention to a QEP, this issue proposes a pilot: migrate one dataset per hosting pattern, chosen as the hardest representative of each class, so the design decisions get validated empirically and the edge cases surface before we write them into a standard.

Where the plan comes from

A full audit of the 7 Python lecture repos (2026-07-15) produced a complete registry of dataset usage. Headline numbers:

Metric	Count
Distinct static data files referenced by lectures	34
Data files committed across the repos	50 (20 unreferenced)
GitHub raw-URL spellings in production	6
References into the dormant pre-MyST lecture-python repo	8
Lectures fetching live API data (wbgapi, FRED ×3 access methods, yfinance)	11
Lectures referencing QuantEcon/data (which already holds 10 intro files)	0

Provenance split of the 34 files: 7 constructed with a committed builder script, 5 constructed but the pipeline was never committed, 7 verbatim third-party files, 14 author-assembled with no pipeline, 1 toy.

Working assumptions under test

The pilot adopts the current #336 direction as a strawman and tests it: canonical repo per the rename/repurpose decision, served at stable URLs (the data.quantecon.org/lectures proposal), flat layout, main pinning with corrections-in-place / new-vintages-as-new-files. If any assumption fails in practice, that is a pilot finding, not a pilot failure.

The four pilots

P1 — local-path static (simplest, first): lingcod_msy_recovery.csv, read by msy_fishery in lecture-python-intro via a relative path today.
Must answer: does the migrated lecture build green in a single PR (motivation 1 in #336)? Does the downloaded notebook run in Colab unchanged (motivation 8)? What catalog metadata does an author-assembled file need?

P2 — cross-series shared static: the pandas_panel trio (realwage.csv, countries.csv, employ.csv), used by the same lecture in lecture-python-programming and lecture-python.myst, currently fetched mostly from the dormant legacy repo.
Must answer: does the flat namespace work when two series consume one file? Does one data-repo PR cleanly update two lecture repos? This pilot also retires 5 of the 8 legacy-repo references as a side effect.

P3 — external-repo static with LFS: the heavy_tails set (forbes-global2000.csv, forbes-billionaires.csv, cities_us.csv, cities_brazil.csv) plus the SCF pair from high_dim_data.
Must answer: does the serve-it-ourselves URL make the raw-vs-media LFS trap fully invisible (open question 4 in #336)? Does the Pages deploy handle LFS objects correctly (lfs: true checkout)? Do constructed files' builders (webscrape_forbes.ipynb, generating_mini.md) migrate alongside the data?

P4 — dynamic snapshot twin (most new machinery, last): UNRATE, consumed today by 4 lectures across 3 repos via 2 different access methods.
Must answer: the full dynamic-data template — a per-dataset manifest (source, series ID, license, cadence, schema, consumers), a build script with fetch → pre-process → validate → write stages, a scheduled GitHub Action that lands refreshes as reviewable PRs, and a "sources alive" canary that detects upstream breakage in the data repo instead of in lecture CI. Also: what is the documented mechanism for a lecture to switch between the live call (where the API is the pedagogy) and the snapshot twin?

Success criteria

• Each migrated lecture builds green under -nW in a single PR, with no two-step merge
• Downloaded notebooks run in Colab without edits
• A pyodide/JupyterLite fetch of each migrated URL succeeds (CORS — relates to the WASM work in [Plan/Idea] Supporting WASM based introductory series #143)
• P4's refresh workflow produces a reviewable PR with a meaningful diff summary, and its canary catches an induced failure
• Every decision exercised gets written into the draft styleguide/datasets.md as it is validated

Out of scope

• Bulk sweep of the remaining ~25 files (mechanical once the convention is proven)
• The %%file-embedded toy datasets (5 files — proposal: these stay embedded in lecture source by design)
• Recovering pipelines for the 5 constructed-but-unscripted files and licensing review for rehosting third-party files — both need their own passes and should be recorded as QEP follow-ups

Sequence and exit

P1 → P2 → P3 → P4, each a small PR set (data repo + consuming lecture repos). Exit: findings folded into the draft manual page, then the QEP gets written recording a convention that has already worked, with the remaining sweep as its rollout checklist.

[mmcky]

P1 built and in review — 2026-07-16

P1 (local-path static, lingcod / msy_fishery): QuantEcon/data-lectures#12 (data + first manifest) → QuantEcon/lecture-python-intro#792 (lecture repoint). #792 is a draft until #12 lands, stacked behind the flat-tree restructure in QuantEcon/data-lectures#10.

Exit tests

• Downloaded notebook runs unchanged — and P1 fixed this rather than preserving it. The read was a relative path (pd.read_csv('datasets/lingcod_msy_recovery.csv')), which only resolves inside a repo checkout, so the cell raised FileNotFoundError in Colab. Reading over https is what makes the notebook self-contained. Verified by running the lecture's own cell against the data over https: 82 rows, figure renders, terminal-year null handled.
• Output unchanged — the migrated file is byte-identical to the copy removed from intro (sha256 fc2f873f…), checked before the repoint.
• Single-PR green build under -nW — blocked on merge order, which is the pilot's main finding (below).
• pyodide/JupyterLite CORS check — not done.

What P1 actually validated

Filed in full on QuantEcon/QuantEcon.manual#108. In short, the first real file broke the schema sketch in three places and exposed one process gap:

The merge order is load-bearing. The data repo's README promises "the lecture PR builds green immediately, no two-step merge" — true, but only once the data is on the data repo's default branch. Probed live: the main URL 404s, the branch URL returns 200. The branch shortcut is right there and would turn #792 green instantly — and it is precisely the bug #337 risk 1 was: a lecture pinned to a non-default branch since 2023, which turned out to live in four repos and 404'd all four the moment the branch was auto-deleted on merge. So: data PR first, lecture PR second, never a non-default ref even temporarily.

Integrity is two questions. Lingcod is a byte-perfect copy of what the lecture consumes and its upstream provenance cannot be re-derived at all — no stock id, no database version, no builder. Both true at once, so integrity split into migration and upstream, with unverifiable plus a reason as a legitimate value.

A constructed dataset with no builder needs somewhere to say so. Lingcod is one — the exact bug AGENTS.md names. The tempting fix is relabelling it verbatim, which makes the manifest lie. builder_status: unrecovered keeps it visible in the catalog and makes the Phase 9 backlog machine-countable.

Licences: check the DOI record, not the homepage. RAM Legacy is CC BY 4.0 and the extract is redistributable — but ramlegacy.org states no licence at all; the authority is the Zenodo record behind its DOI.

P2 is better positioned than it was

P2's scope was understated in one direction and overstated in another. The trio is referenced by more repos than the two listed here — but the extra ones are translations, and they self-heal: QuantEcon/action-translation syncs them from the English sources as PRs, verified today when merging QuantEcon/lecture-python-programming#576 opened sync PRs in .fa, .fr and .zh-cn within seconds. So P2 remains a two-repo job, as planned.

What did change: the trio no longer reads from the dormant QuantEcon/lecture-python repo. QuantEcon/lecture-python.myst#968 and QuantEcon/lecture-python-programming#576 repointed all five references to own-repo copies (byte-compared, published), which was #337 risk 2. P2 therefore starts from a clean base and is a straight migration rather than a migration tangled with a legacy-repo retirement.