Feature/CLIMATE-22 historical gbd versioning by wtgustafson · Pull Request #41 · ihmeuw/climate-data

wtgustafson · 2026-06-25T18:11:58Z

Implements CLIMATE-22. Stacked on #PR40 (CLIMATE-21 dry-run) — review/merge that
first, then this retargets to main.

What

GBD-hierarchy versioning for the special stage: registers gbd_2021/gbd_2025,
adds GBD_HIERARCHIES, and a --hierarchy option (default gbd_2023) on the
person-days runners.
Historical run mode for temperature_zone and the person-days steps.
draws now prefers historical ERA5 over scenario data for overlapping years.

Changed

HISTORY_YEARS extended through 2025 (so the historical-preference filter covers
the 2024–2025 overlap).
Storage root MODEL_ROOT → /mnt/share/geospatial/climate/ (AGGREGATE_ROOT
follows it).

🤖 Generated with [Claude Code]

In compile_gcm, filter scenario annual inputs to years not in HISTORY_YEARS so the extended ERA5 historical record (through 2025) takes precedence over forecast data for the overlapping 2024-2025 years. Prevents duplicate/conflicting year coords in combine_by_coords. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Support scenario="historical" (gcm_member "era5"): read the raw historical annual mean_temperature directly and produce a pure-ERA5 10-year rolling zone spanning 1990-2025, instead of the compiled historical+forecast series. Switch the runner/task option from --cmip6-experiment to --scenario and enumerate era5 for historical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- constants: register gbd_2021/gbd_2025 in HIERARCHY_MAP and add GBD_HIERARCHIES; point AGGREGATE_ROOT at the geospatial working tree (MODEL_ROOT/aggregates). - data: add ClimateAggregateData.person_days_path / compiled_person_days_path; generalize the hierarchy conditionals to GBD_HIERARCHIES; note the gbd_2025 parent-drop check; drop a stray debug print in save_compiled_annual_results. - cli_options: let with_hierarchy take a default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Add a historical run mode (scenario="historical", era5, years 1990-2025; ERA5 daily used for all years). - compile_person_days: aggregate against gbd_2023 (matches the inputs, which are generated under gbd_2023). - Parameterize --hierarchy (default gbd_2023); version outputs by appending the hierarchy to output_dir, and route reads/writes through the centralized person_days_path / compiled_person_days_path helpers. - _main creates its own per-block output dir so standalone cdtask runs don't depend on the runner pre-creating it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…climate HISTORY_YEARS now runs through 2025 so the draws 'prefer historical ERA5 over scenario' filter actually excludes the overlapping 2024-2025 forecast years. MODEL_ROOT moves to /mnt/share/geospatial/climate/ (AGGREGATE_ROOT already follows MODEL_ROOT). Both were previously entangled in the CLIM-21 dry-run commit; consolidated here so CLIM-21 stays pure infra. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The CLIM-22 rewrite of the compile_person_days runner introduced an un-annotated 'jobs = []' that fails strict mypy (var-annotated). Annotate as list[tuple[str, str]] (gcm_member, scenario). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add a standalone `special` step that downloads whole-year ERA5 `reanalysis` and `ensemble_spread` 2m-temperature files for 2024/2025 from the Copernicus CDS, matching Katrin Burkart's `era5_{product_type}_{variable}_{year}.nc` layout. Fills the 2024/2025 uncertainty gap that GBD temperature PAF work depends on. - Pulls both products 3-hourly from reanalysis-era5-single-levels (the only dataset exposing ensemble products); ensemble_spread/EDA is only 3-hourly, reanalysis/HRES is subsampled to match. - Reads credentials from the caller's ~/.cdsapirc, avoiding the shared copernicus.yaml keyring. - Registered in special/__init__.py RUNNERS/TASK_RUNNERS as `download_era5_uncertainty`; CHANGELOG updated. ensemble_spread is not yet consumed downstream -- this is an input gap-fill only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…rtainty Add methodology docs for the CLIMATE-22 features: - New "Temperature person-days exposure" section (special-stage chain: temperature_zone -> temperature_person_days -> compile_person_days -> draw assembly), with subsections for the historical ERA5-only mode (1990-2025 GBD deliverable, distinct from the 1990-2100 forecast) and GBD --hierarchy versioning. - New "ERA5 ensemble uncertainty" subsection (reanalysis/ensemble_spread 2m-temp gap-fill, single-levels only, not yet consumed by the pipeline). - Version-dependent reference-climatology and historical-daily windows: 2019-2023 for FHS-2023, 2021-2025 for FHS-2025; post-2023 extension is currently GBD-only. - Wired docs/methods.md into the mkdocs nav (was orphaned). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

`choices` was annotated `Sequence[str]` but defaults to a dict (`cdc.HIERARCHY_MAP`), a pre-existing mypy error. Widen to `Collection[str]` — what a dict is and what `with_choice` accepts. No behavior change; drop the now-unused `Sequence` import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review #1 (critical): the branch repointed MODEL_ROOT/AGGREGATE_ROOT to the geospatial run area, which would repoint the whole pipeline on merge. Revert both to the production roots (erf/climate_downscale + climate-aggregates) as the module-level defaults, and add model_root()/aggregate_root() helpers plus a --run-mode CLI option (forecast|historical, default forecast) so the historical GBD run opts into the geospatial roots explicitly rather than via a hardcoded constant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review #2/#4/#5: extending HISTORY_YEARS to 2025 created a 2024/2025 overlap with FORECAST_YEARS (double-claimed years), which drove compile_gcm hole risk, a shifted reference-climatology baseline, and wrong-zone person-days binning. Revert HISTORY_YEARS to end at 2023 (production) so the overlap disappears at the source: FORECAST_YEARS starts 2024 with no double-claim, REFERENCE_YEARS is back to 2019-2023, and ALL_YEARS has no duplicates. Add EXPOSURE_START_YEAR=1990 for the person-days/zone analysis window (used by the special stage, which derives its upper bound from the data on disk). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review #3/#8: drive the person-days loop from the zone's actual `year` coordinate, indexed by matching year (was hardcoded 1990..2025 + positional), skipping years with missing daily/pop inputs instead of raising. Derive FORECAST_START_YEAR and the zone slice from constants. Add --run-mode {forecast,historical} to the special person-days chain to route the storage roots (production vs geospatial); an explicit --climate-data-dir/--output-dir still wins. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review #6: if the unzip failed part-way, out_path was left truncated and the except only removed download_path, so the next run skipped it as "already downloaded". Write the unzipped payload to a temp and atomically replace onto out_path as the last step; clean up partial out_path/temp on failure. Review #9: comment ERA5_UNCERTAINTY_YEARS as the historical-extension gap-fill (fixed path, run-mode-independent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review #10: draws.compile_gcm_main and temperature_zone duplicated the `raw_annual_results glob + open_mfdataset(combine="by_coords").sortby("year") .compute()` idiom, which could diverge. Add ClimateData.combine_raw_annual (the open/sort/compute) and load_raw_annual_mfdataset (glob + combine); use them in both callers. Behavior-preserving; drops the now-unused xarray imports there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review #7: the widened most_detailed override applies the UK/India/NZ parent-drop list to gbd_2025. Verified against the hierarchy parquets: 39/41 ids are present in gbd_2025 (4854 J&K/Ladakh reorganized out, harmless no-op), so keep the widening. While verifying, found 4919 is absent in all rounds -- a typo for 4619 ("North West England"), so its UTLA children were never dropped; correct it and update the NOTE. Checked 4919 explicitly: it appears as neither a location_id nor a parent_id in any raking hierarchy parquet, and 4618-4626 are exactly England's nine regions with only 4619 otherwise missing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review #5 + run-mode docs: note that the reference window is the trailing five years of history, so changing the history end shifts it and the reference climatology and bias correction must be regenerated in lockstep. Document that --run-mode selects the storage roots (forecast=production, historical=geospatial) and that the historical product's span is derived from the ERA5 data on disk (1990-2025 as produced). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wtgustafson · 2026-07-14T23:31:51Z

@rmbarber — pushed a round of changes addressing an independent code review of this branch, plus the methodology docs. Ready for your look; summary below.

Core design change

The branch had made two global changes a main merge shouldn't carry: it repointed the storage roots to geospatial and pushed HISTORY_YEARS to 2025. Both are now reverted to the production defaults, and a new --run-mode {forecast,historical} flag routes the storage roots explicitly — default forecast = production (erf/climate_downscale + climate-aggregates), historical = the geospatial working area for the GBD product. The historical 1990–2025 span is now derived from the ERA5 data on disk, not a global constant.

Review findings → resolution (7 gated commits, `52c3998`..`a5d265d`)

🔴 Feature/add dependencies #1 leaked geospatial root → reverted roots + --run-mode routing
🟠 Feature/ncei extraction #2/Add stubs for other extractions #4/Feature/other extraction stubs #5 2024/2025 historical↔forecast overlap → resolved at the source by reverting HISTORY_YEARS to 2023 (no double-claim; REFERENCE_YEARS back to 2019–2023; forecast 2024/2025 is GCM on both daily and zone)
🟠 Feature/era5 extraction #3 historical person-days crash → drive the year loop from the temperature zone's own year coord, index by matching year, and skip years with missing daily/pop inputs instead of IndexError/FileNotFound
🟡 Model data prep #6 truncated netCDF left on a partial unzip → atomic finalize (temp → replace, clean up partials)
🟡 Added vscode to git ignore #7 gbd_2025 parent-drop list → verified against the hierarchy parquets (39/41 ids present, safe to keep); fixed a 4919→4619 typo (North West England's UTLA children were never being dropped, in every round)
🔵 Fix linting and type checking #8/CLIMATE-2: Draft draw generation #9/Add precipitation days #10 → de-hardcode year literals, document the ERA5-uncertainty gap-fill years, factor the duplicated raw_annual open/combine idiom into a ClimateData helper

Docs

Methodology page updated (person-days product, historical/--run-mode behavior, reference-climatology lockstep). This rendered preview highlights the documentation changes vs main — green = added, ~~struck~~→new = reworded — so you can see exactly what this branch adds/changes:
https://docs.sae.ihme.washington.edu/billg/previews/climate-data/methods/

It links out to the main methodology doc page (where the docs render without highlighting):
https://docs.sae.ihme.washington.edu/billg/previews/climate-data

Full review write-up (the 10 findings): https://docs.sae.ihme.washington.edu/billg/reviews/climate-22/

Verification status (being explicit)

mypy / ruff / inspection clean on all touched files; run-mode resolution, download atomicity, and the gbd_2025 ids are verified in isolation. The end-to-end person-days / compile_gcm pipeline was NOT re-run on the cluster — logic is verified in isolation only, and the already-delivered outputs are untouched. Flagging in case you want an e2e pass before merge.

rmbarber · 2026-07-16T19:54:19Z

-            / block_key
-            / f"{cmip6_experiment}_{gcm_member}.parquet"
-        )
+        path = ca_data.person_days_path(block_key, scenario, gcm_member)


Probably preferable to just handle the io in the data module (i.e., write a ca_data.load_person_days rather than generate the path and then call read_parquet here)

rmbarber · 2026-07-16T20:02:18Z

+                    scenario, "mean_temperature", year, gcm_member
+                ).sel(**climate_slice)
+            pop_arr = pm_data.load_results(f"{year}q1", block_key)._ndarray.flatten()  # noqa: SLF001
+        except FileNotFoundError as exc:


What is the impetus for this? I would have thought we need squareness across time/scenario/ensemble member, and that missingness would be something that should fail loudly rather than be skipped.

Yeah, that is a very good point. I may be leaning on claude too heavily here, because I missed that point.

Review (rmbarber): the per-year loop caught FileNotFoundError and skipped any year missing its daily or population input. For a product that must be square, that silently punches holes instead of surfacing a real gap. Drop the try/except so a missing file propagates from load_daily_results / load_raw_daily_results / load_results. The span is still derived from the temperature_zone on disk. Reword the docstring and the no-output guard to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review (rmbarber): compile_person_days built the block path and called pd.read_parquet inline. Move the I/O into ClimateAggregateData.load_person_days (mirrors the load_raw_results convention) so the data module owns the layout. Keep the block-loading loop imperative and promote ruff PERF401 to the global ignore list (it was already ignored per-file for jobmon_utils.py with the note "billg hates list comprehensions"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

temperature_person_days loaded the raw population array (nodata = NaN) and summed it into the compute_person_days accumulator (out += pop), so a NaN population pixel poisoned every output cell it touched and was read back as 0 -- zeroing small and coastal locations (American Samoa, the England regions, etc.). Zero-fill nodata before the kernel; a modeled-population-free pixel contributes no person-days. Adds a regression test (NaN pixel must not poison the output) and a CHANGELOG entry. Latent bug exposed by a population-model nodata-encoding change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Bill Gustafson and others added 7 commits June 25, 2026 10:47

CLIM-22: add changelog entries for historical mode + GBD versioning

97b585d

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

wtgustafson requested a review from rmbarber June 25, 2026 18:13

wtgustafson changed the base branch from feature/CLIMATE-21-dry-run to main June 25, 2026 23:34

wtgustafson changed the title ~~Feature/climate 22 historical gbd versioning~~ Feature/CLIMATE-22 historical gbd versioning Jun 25, 2026

Bill Gustafson and others added 10 commits July 13, 2026 16:40

rmbarber reviewed Jul 16, 2026

View reviewed changes

Bill Gustafson and others added 2 commits July 16, 2026 14:06

rmbarber approved these changes Jul 16, 2026

View reviewed changes

wtgustafson merged commit 6e00542 into main Jul 20, 2026
7 of 8 checks passed

This was referenced Jul 21, 2026

chore: green up pre-commit CI #42

Merged

ci: move workflows off the departed maintainer's personal token #43

Merged

wtgustafson mentioned this pull request Jul 21, 2026

Fix deploy docs #36

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/CLIMATE-22 historical gbd versioning - #41

Feature/CLIMATE-22 historical gbd versioning#41
wtgustafson merged 20 commits into
mainfrom
feature/CLIMATE-22-historical-gbd-versioning

wtgustafson commented Jun 25, 2026

Uh oh!

wtgustafson commented Jul 14, 2026 •

edited

Loading

Uh oh!

rmbarber Jul 16, 2026

Uh oh!

wtgustafson Jul 16, 2026

Uh oh!

rmbarber Jul 16, 2026

Uh oh!

wtgustafson Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

wtgustafson commented Jun 25, 2026

What

Changed

Uh oh!

wtgustafson commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Core design change

Review findings → resolution (7 gated commits, 52c3998..a5d265d)

Docs

Verification status (being explicit)

Uh oh!

rmbarber Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

wtgustafson Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

rmbarber Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

wtgustafson Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wtgustafson commented Jul 14, 2026 •

edited

Loading

Review findings → resolution (7 gated commits, `52c3998`..`a5d265d`)