cedarkit-test-data

Important

This repository has changed purpose. It is no longer a Python package — it is now the data-generation project and release-asset store for frozen test datasets used by the cedarkit / reki documentation and tests.

• The old installable package is archived on the legacy/test-data-package branch. Do not pip install it: its reki.sources entry point shadows reki's built-in test source. Its functionality was merged into reki (built-in test source + reki-test-data CLI).
• The actual data files are GitHub release assets, not files in this repo. See Releases.

What this repository is

1. Generation scripts (scripts/) that produce small, frozen test datasets from ECMWF IFS open data (CC-BY-4.0).
2. Release assets: each data version is published as a GitHub release tagged with an independent calver (v<year>.<month>.<rev>, e.g. v2026.8.0), decoupled from reki releases. Data semantics (model / domain / run date / forecast step) live in the asset file names, not in the version number.

Data inventory

Asset naming: ifs_<domain>_<run-date><run-time>_f<step>.grib2

Asset	Model	Run	Step	Parameters	Domain	Intended use
ifs_eastasia_2026090712_f024.grib2	ECMWF IFS HRES 0.25°	2026-09-07 12Z	+24h	2t, 2d, 10u, 10v, msl, tp; gh/t at 500 hPa, t/u/v at 850 hPa	0–60N, 60–150E	core read / plot examples
ifs_time_docdomain_2026090712.grib2	ECMWF IFS HRES 0.25°	2026-09-07 12Z	0/6/12/24h	2t instant; tp accumulations	20–50N, 100–130E	time / valid-time examples
ifs_ensemble_docdomain_2026090700.grib2	ECMWF IFS 0.25°	2026-09-07 00Z	0/24h	2t, control (oper/fc) plus PF 1–20 (enfo/pf)	20–50N, 100–130E	control/member-selection and ensemble-statistics examples
ifs_layers_docdomain_2026090712.grib2	ECMWF IFS HRES 0.25°	2026-09-07 12Z	0h	soil temperature (sot), layers 1/2	20–50N, 100–130E	layer-bound examples
ifs_global_2026090712_f024.grib2	ECMWF IFS HRES 0.25°	2026-09-07 12Z	+24h	2t	global	regrid / area operator examples

The published v2026.9.0 release contains the earlier two-PF-member ensemble asset. The prepared v2026.9.1 candidate is a complete five-asset release that carries forward the four unchanged files and replaces its ensemble asset with control plus PF 1–20, using the same filename in a new immutable release. Its manifest is the machine-readable source of file identities, complete SHA-256 values, source requests and attribution. The prior 2026-08-18 assets remain available in v2026.8.1 for compatibility; they are not part of the current dataset.

Using the data

The recommended entry point is reki's built-in test source (ecmwf_ifs dataset), which downloads from the release URL and opens the file in one call:

import reki

ds = reki.from_source("test", "ecmwf_ifs")           # frozen East-Asia subset
field = ds.sel(parameter="2t", level_type="heightAboveGround", level=2).to_xarray()

or the CLI:

reki-test-data download ecmwf_ifs --domain eastasia -o ./data

Direct release-URL reference after v2026.9.0 is published (advanced):

https://github.com/cemc-oper/cedarkit-test-data/releases/download/v2026.9.0/manifest.json

Regenerating the data

ECMWF open data only keeps the last few run cycles, so regeneration means: pick a current run, pin it in the script, generate, tag, release.

uv venv && uv pip install -r requirements.txt
# also required: the ecCodes shared library (apt install libeccodes0 or conda);
# all GRIB2 processing is done through the ecCodes Python API — no cdo needed

# pin DATE/TIME/STEP in scripts/generate_ifs.py first, then:
uv run python scripts/generate_ifs.py --all --output-dir build --manifest build/manifest.json
uv run python scripts/validate_assets.py \
  --contract contracts/asset-contract.yaml \
  --assets-dir build --manifest build/manifest.json --check

Follow release/approval-checklist.md for the required review, commit, tag, upload and rollback steps. The checklist is deliberately procedural: this repository never creates a tag, push or release automatically.

License and attribution

• The scripts in this repository are under the repository's original license (see LICENSE).
• The data assets contain modified ECMWF IFS open data, © ECMWF, licensed under CC-BY-4.0. See NOTICE. When using the data, attribute ECMWF.

Historical note

Releases up to v2026.7.0 (and PyPI package cedarkit-test-data) were the old Python test-data package; see the legacy/test-data-package branch. Data releases start at v2026.8.0.