NestedSimulation abstraction + ERA5 regional hindcast example

[ewquon]

Summary

@glwagner @kaiyuan-cheng

Adds examples/era5_breeze.jl — a building block for a regional modeling example that will eventually couple Breeze (compressible solver, in development) to forthcoming SlabLand and SlabOcean components.

We download ERA5 reanalysis over an SGP-centered LAM bounding box (HI-SCALE 2016-09-10 case day) and interpolate onto a LatitudeLongitudeGrid sized for ~3 km horizontal cells at the domain center latitude. Tᵥ is computed as a derived field using Breeze.ThermodynamicConstants for the Rᵥ/Rd − 1 coefficient.

Punch list

• Breeze model construction
• open boundary conditions
• dynamical initialization (DFI)
• terrain
• acoustic substepping over terrain
• land/ocean coupling

Notes

• Surface pressure is read onto a 3-D grid with Nz=1 rather than a 2-D (Bounded, Bounded, Flat) grid, sidestepping CliMA/Oceananigans.jl#5473 (Flat↔non-Flat interpolate! errors with a cryptic BoundsError; #5474 will convert that to a clear ArgumentError but does not lift the restriction). Mirrors the pattern in examples/ERA5_hourly_data.jl.
• This example uses a LatitudeLongitudeGrid. A future variant on a projected Cartesian grid would benefit from the set! projection support proposed in Add map projection support to set! for RectilinearGrid targets #232.

Test plan

• Run end-to-end against the CDS API on a machine with ~/.cdsapirc configured.

[codecov]

Codecov Report

❌ Patch coverage is 13.88889% with 93 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...els/NestedSimulations/interpolated_fts_boundary.jl	0.00%	32 Missing ⚠️
...arthSystemModels/NestedSimulations/nested_model.jl	0.00%	22 Missing ⚠️
...SystemModels/NestedSimulations/child_simulation.jl	0.00%	12 Missing ⚠️
src/Atmospheres/prescribed_atmosphere.jl	65.21%	8 Missing ⚠️
...ls/NestedSimulations/parent_boundary_conditions.jl	0.00%	8 Missing ⚠️
...hSystemModels/NestedSimulations/parent_forcings.jl	0.00%	8 Missing ⚠️
...NumericalEarthBreezeExt/breeze_child_simulation.jl	0.00%	2 Missing ⚠️
...ystemModels/NestedSimulations/nested_simulation.jl	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[ewquon]

Okay, the model state should be ready for dynamic initialization.

A couple of important notes:

1. As noted in ERA5 pressure-level ingest: per-column geopotential and surface-pressure masking #236, we need to better address geopotential height variability and we should discuss the approach to this. I've prototyped per-column vertical interpolation to unblock the ICs work and to have as a reference when developing the data wrangling improvement.
2. Because we don't have terrain yet, I'm using the height above ground level to interpolate onto the flat-bottomed Breeze grid. (Otherwise, portions of the Breeze domain would be essentially underground and I imagine this would be problematic for the initialization)

[glwagner]

@ewquon should we add terrain? Terrain following should work for the fully compressible formulation (just not substepping)

[ewquon]

@ewquon should we add terrain? Terrain following should work for the fully compressible formulation (just not substepping)

Hallelujah! I'm happy to add that in. Anything that gets us closer to reality sooner rather than later is a win in my book.

[glwagner]

@ewquon I added the "build all examples" label so that it will build even if the "build always" tag is false.

[glwagner]

@giordano do you know what the GPU error is?

https://github.com/NumericalEarth/NumericalEarth.jl/actions/runs/25867031203/job/76011977037?pr=233#step:2:211

[giordano]

I believe

  failed to register layer: write /usr/local/share/julia/artifacts/75ae42466e7dd21641ff521f63cfb456d09433e2/lib/libcudnn_engines_precompiled.so.9: no space left on device
  Warning: Docker pull failed with exit code 1, back off 8.515 seconds before retry.

simply happens when the runner was reused from a previous run from somewhere else in the organisation, but the new build uses a new Docker image than the previous run, and then it fails to pull the second one because the combination of the two images fills the partition.

In summary, it's an inoffensive, albeit annoying, issue, restarting the job should succeed (unless you rerun in the same situation 😄)

[giordano]

NumericalEarth/Breeze.jl#715 recovers some space on the VMs also in the jobs we run in the Breeze.jl repository (at the moment we do the cleanup only in this repo), hopefully this will make the error less frequent.

[glwagner]

working syntax (updated for the latest API on the branch — child_simulation + NestedSimulation wrap a NestedModel):

using NumericalEarth, Oceananigans, Breeze
using Breeze.AnelasticEquations: AnelasticDynamics
using Breeze.Thermodynamics: ReferenceState

# Lamb-Oseen vortex translating at U in x
const Γ, a, U, x₀, y₀ = 5e3, 200.0, 10.0, 600.0, 1000.0
const ρ̄ = 1.225

function lamb_oseen_uv(x, y, t)
    dx, dy = x - x₀ - U*t, y - y₀
    r² = dx^2 + dy^2
    uθ_over_r = r² < eps() ? 0.0 : (Γ / (2π * r²)) * (1 - exp(-r² / a^2))
    return (U - uθ_over_r * dy, uθ_over_r * dx)
end

# Parent: coarser and strictly larger than the child, so the FTS brackets every
# child sampling node (Interpolated BC + Relaxation-on-FTS both require this).
parent_grid = RectilinearGrid(size = (40, 16, 8),
                              x = (-500, 4500), y = (-500, 2500), z = (-20, 120),
                              topology = (Bounded, Periodic, Bounded))

parent = PrescribedAtmosphere(parent_grid, collect(0.0:5.0:305.0))
set!(parent.velocities.u, (x, y, z, t) -> ρ̄ * lamb_oseen_uv(x, y, t)[1])  # ρ̄·u → Breeze momentum BC
set!(parent.velocities.v, (x, y, z, t) -> ρ̄ * lamb_oseen_uv(x, y, t)[2])

# Child: finer LAM grid
child_grid = RectilinearGrid(size = (128, 32, 4),
                             x = (0, 4000), y = (0, 2000), z = (0, 100),
                             topology = (Bounded, Periodic, Bounded))

# Sponge mask: ramps from 0 in the interior to 1 in the outer 15% of x.
const Lx, x_sponge = 4000.0, 0.85 * 4000.0
sponge_mask(x, y, z) = x < x_sponge ? 0.0 : ((x - x_sponge) / (Lx - x_sponge))^2

# child_simulation builds the Breeze child model, wires Open BCs from the parent
# on the requested sides, and adds Relaxation-on-FTS interior nudging.
# It returns the constructed AtmosphereModel.
ref   = ReferenceState(child_grid; surface_pressure = 101325.0, potential_temperature = 300.0)
child = child_simulation(AtmosphereModel, child_grid, parent;
                    sides           = (:west, :east),
                    relaxation_rate = 1 / 60,        # 60-second sponge timescale
                    relaxation_mask = sponge_mask,
                    dynamics    = AnelasticDynamics(ref),
                    formulation = :LiquidIcePotentialTemperature,
                    advection   = WENO(order = 5))

set!(child; θ  = 300.0,
            ρu = (x, y, z) -> ρ̄ * lamb_oseen_uv(x, y, 0)[1],
            ρv = (x, y, z) -> ρ̄ * lamb_oseen_uv(x, y, 0)[2],
            ρw = 0.0)

# NestedSimulation(parent, child; sim_kwargs...) is a convenience for
# `Simulation(NestedModel(parent, child); sim_kwargs...)`. The NestedModel's
# `time_step!` steps the child, then ticks the parent's clock to match.
nested = NestedSimulation(parent, child; Δt = 0.2, stop_time = 300.0)
run!(nested)

What's in play:

• child_simulation(modeltype, grid, parent; …) builds the child model and wires up its BCs and forcings from the parent. Returns the constructed AbstractModel. Forwards model_kwargs to the model constructor.
• Variable mapping is chosen by dispatch on default_parent_variables(modeltype, parent). Oceananigans.NonhydrostaticModel → (u, v) ← parent.velocities.(u, v) (core). Breeze.AtmosphereModel → (ρu, ρv) ← parent.velocities.(u, v) (in NumericalEarthBreezeExt, so Breeze stays a weakdep).
• BCs go through an internal Interpolated(fts) wrapper that lets OpenBoundaryCondition consume an FTS directly. Standard Oceananigans regularization tags it with side/dim/location; our getbc does trilinear-in-space + linear-in-time interpolation against the parent grid.
• relaxation_rate / relaxation_mask route through Oceananigans' native FTS-Relaxation path (Support FieldTimeSeries targets in Relaxation CliMA/Oceananigans.jl#5575) for interior nudging.
• NestedModel(parent, child::AbstractModel) <: AbstractModel pairs the two and overrides time_step! to step the child, then advance the parent. NestedSimulation(parent, child; sim_kwargs...) is a one-liner for Simulation(NestedModel(parent, child); sim_kwargs...).

Needs Breeze on eq/atmosphere-model-open-bc-velocity (PR NumericalEarth/Breeze.jl#722) and Oceananigans main (post-0.108.0) for FTS-Relaxation.