ecCKD/ecRad radiation platform and validation harness

[glwagner]

Summary

Renames and rebuilds AnalyticBandRadiation.jl into the standalone, ecRad/ecCKD-compatible radiation package NumericalRadiation described in radiative_heating.md: staged runtime API, ecCKD ingestion + forward gas-optics, cloudless/cloud-overlap solvers, cloud + aerosol optics scaffolding, NCDatasets and RRTMGP extensions, and an Artifacts.toml wiring for official ecRad/ecckd data. Adds an executable validation/ audit harness — accuracy gates against ecRad references, reduced-model search/optimizer artifacts, RRTMGP comparison, AD calibration, ecCKD recovery preflights, plus prompt-to-artifact and recovery-goal audits writing paired .json + .md to validation/results/. Removes the in-repo Breeze extension and lands an independent benchmarking/ project here that path-deps onto a developing Breeze checkout for the production H100 benchmark.

Full narrative, status against each gate, and step-by-step reproduction are in PR_WORK_SUMMARY.md.

Package rename history

The Julia package has been renamed twice in this PR. UUID cd8119b0-1744-44d6-9ede-6ad1ad750b26 is stable across both renames, so downstream environments only need to update the package name in their manifests.

1. AnalyticBandRadiation → Lightflux
2. Lightflux → NumericalRadiation (current)

The GitHub repository URL (github.com/NumericalEarth/AnalyticBandRadiation.jl) reflects the original name and has not been renamed on GitHub yet. The Julia package name in Project.toml is now NumericalRadiation. Source/extension paths:

• src/NumericalRadiation.jl (was src/Lightflux.jl, was src/AnalyticBandRadiation.jl)
• ext/NumericalRadiation{NCDatasets,RRTMGP,SpeedyWeather}Ext.jl

Downstream packages with manifests referencing Lightflux or AnalyticBandRadiation under UUID cd8119b0-... need to regenerate or update their manifests to pick up the new name. The Breeze rename plumbing (BreezeLightfluxExt → BreezeNumericalRadiationExt, path-fallback constants in benchmarking/) lands in a separate Breeze commit.

Headline figures

Regenerated from committed artifacts by figures/make_pr_figures.jl.

Accuracy: official 32×32 ecCKD matches ecRad on the clear-sky tropical column

LW flux RMSE ≈ 0.003 W m⁻², SW flux RMSE ≈ 0.007 W m⁻², heating-rate RMSE ≈ 0.006 K day⁻¹. Hard ecCKD cloudless gate passes.

Performance: H100 speedup across published ecCKD k-models at production scale

Measured by the new independent benchmarking/ project (path-deps onto the dev Breeze checkout) through Breeze's update_radiation! call surface. Both bars use post-warmup medians from 5 samples on a single H100 + the same 512×512×128 / 262,144-column RCEMIP-style workload.

k-model	RH median [ms]	RRTMGP median [ms]	Speedup
32 LW × 32 SW (fsck-32b × rgb-32b)	794.7	5863.3	7.4×
32 LW × 64 SW (fsck-32b × window-64b)	1047.3	5864.2	5.6×
32 LW × 96 SW (fsck-32b × vfine-96b)	1279.3	5867.7	4.6×
64 LW × 32 SW (narrow-64b × rgb-32b)	1352.7	5869.8	4.3×
64 LW × 64 SW (narrow-64b × window-64b)	1603.6	5870.1	3.7×

RRTMGP is essentially constant — it runs its own 256-LW × 224-SW lookup independent of the ecCKD k-counts, so the dashed baseline is genuinely one number. RadiativeHeating scales sub-linearly with total g-points thanks to the new streaming kernel in BreezeRadiativeHeatingExt._tabulated_ecckd_streaming_radiation!, which fuses g-point optical depth into the column transport loop and never materializes (ngpt, Nx, Ny, Nz) four-dimensional intermediates (which at 512×512×128 × 96 g-points would have been ~26 GiB per buffer). The 64 LW × 64 SW row at 3.7× is just below the 4× gate at this scale — an honest reading of where the 64-g production path stands today.

A more recent fully-coupled H100 sweep (post-rename) is recorded in HANDOFF.md R-10: 164.4 ms/step at 100×100×74 and 2457.2 ms/step at 512×512×128, both with radiation scheduled at IterationInterval(1). That's 5.5–7.2× faster than RRTMGP coupled at the same cadence, and 52–57× slower than dynamics-only — the latter ratio is the target of the in-flight performance campaign scoped by HANDOFF.md R-7.

Training: Reactant + Enzyme calibration of ecCKD coefficients

Left: 13-epoch finite-difference / Enzyme-checked gradient descent on a 4-parameter toy ecCKD fixed-topology fixture — loss × 0.006, flux RMSE × 0.077, heating-rate RMSE × 0.178. Right: 8 epochs of Reactant-compiled, Enzyme reverse-mode gradient descent against the package-native RRTMGP shortwave loss for a 48-parameter 16-g model.

What's left: the reduced 16-g hard gate

Worst boundary-forcing error after each accepted optimizer move in the greedy / constrained-table / slot-blend / weight-refit chain. From 0.0042 W m⁻² at the 32×32 baseline up to ≈7.2 W m⁻² for naive 16-g, then down to ≈2.14 W m⁻² after the full chain — still ≈7× above the 0.30 W m⁻² hard gate.

Acceptance gate status

Gate	Status
ecRad parity (full + reduced)	partial — full 32×32 passes hard cloudless and all-sky IFS gates; reduced 16-g shortwave still fails
Reduced vs. RRTMGP on representative states	partial — direct package-native RRTMGP comparison emitted; reduced 16-g still fails hard accuracy
Dynamic Breeze integration + ≥4× H100 RRTMGP speedup	partial — 32×32, 32×64, 32×96, 64×32 pass the 4× gate at 512×512×128 / 262K cols; 64×64 lands at 3.7× and is the next perf-work target
Reactant/Enzyme original-objective ecCKD recovery	blocked — teacher-student recovery passes for all six published definitions; exact recovery is gated on locally derived ecCKD CKDMIP training flux products, generation in progress under Slurm

Notable code changes

• src/runtime_interfaces.jl, src/abstract_types.jl: staged optics/solver/backend API.
• src/io/ecckd_definition.jl, src/gas_optics/ecckd_forward.jl: official ecCKD schema + tabulated forward gas-optics.
• src/io/cloud_scattering.jl, src/solvers/cloud_optics.jl, src/solvers/cloud{less,overlap}_{longwave,shortwave}.jl: cloud/aerosol scaffolding + solvers.
• src/metrics.jl: shared validation metrics + thresholds.
• ext/NumericalRadiationNCDatasetsExt.jl, ext/NumericalRadiationRRTMGPExt.jl: NetCDF loaders + RRTMGP comparison.
• Artifacts.toml: pinned lazy ecrad_data and ecckd_source artifacts.
• validation/: ~50 audit scripts + paired JSON/MD results.
• benchmarking/: independent H100 benchmark suite (env-controlled, path-deps onto developing Breeze).
• figures/: PR-narrative figure generator + four PNGs.
• ext/AnalyticBandRadiationBreezeExt.jl deleted; per-package Breeze extension now lives in the Breeze checkout as BreezeNumericalRadiationExt (rename in flight there).

Test plan

• julia --project=. -e 'using Pkg; Pkg.test()'
• julia --project=test validation/recovery_goal_audit.jl — expect not_complete until the live derived-flux Slurm job finishes.
• julia --project=test validation/ecrad_accuracy_gate.jl + validation/ecrad_all_sky_ifs_gate.jl — hard ecRad gates on official ecCKD.
• julia --project=figures figures/make_pr_figures.jl — regenerate the four PR figures.
• (H100) sbatch benchmarking/h100_kmodel_coverage.sbatch — reproduce the cross-k-model H100 sweep at 512×512×128.
• (Heavy, optional) PR_WORK_SUMMARY.md §4.3 for the live ecCKD derived-flux generation.

🤖 Generated with Claude Code