PROJECT: QuantEcon benchmarking programme (code performance & execution)

[mmcky]

Scope

Code benchmarking and execution performance for QuantEcon computational work. This programme is deliberately focused: AI/LLM evaluation benchmarks (e.g. benchmark.translate-zh-cn) are a separate domain and out of scope here (naming conventions for the two families to be settled separately). Early focus is the lecture series — they are user-facing and where implementation choices are most visible — expanding later to library and infrastructure benchmarks.

Motivation

The review thread on QuantEcon/lecture-python.myst#717 (ge_arrow JAX conversion) produced a quantitative evaluation system for deciding whether an acceleration rewrite actually improves a lecture, and demonstrated why our current practice misleads: warm %timeit-style benchmarks showed the JAX version 1.4–4.8× faster, while replaying the lecture's actual call sequence in a fresh process (so compile time counts) showed it ~45× slower. Getting the measurement methodology right changes conversion decisions.

We already have two scoped benchmarking projects — #264 (kernel/backend suite) and #289 (lecture build monitoring) — plus repos created from them. This umbrella issue organises them into one programme with shared methodology, and adds the missing middle layer that #717 surfaced.

Structure — three workstreams

A. Kernel/backend suite — existing, #264. Library-level benchmarks (NumPy / Numba / JAX CPU / JAX GPU / Dask) across problem sizes, producing crossover curves ("GPU wins above size N for operation X"). This is the reference data layer. Repo: benchmarks.

B. Per-lecture implementation evaluation — new, from QuantEcon/lecture-python.myst#717. Given a lecture's actual problem sizes and call pattern, which implementation should it use — and did a conversion PR improve it? Deliverables: the evaluation scripts (being gathered by @xuanguang-li from the #717 thread), a reusable agent skill (working name /eval-py-acceleration) built with him, and the rubric documented in the manual (tracked in QuantEcon/QuantEcon.manual#104). Both the scripts and the skill live in the new QuantEcon/skills repository as a benchmark plugin, so the harness is versioned once and distributed everywhere (see #304 for the marketplace mechanism). This layer consumes workstream A's crossover data. Validation cases: lecture-python.myst#717 and lecture-python.myst#654.

C. Longitudinal monitoring — existing, #289. Per-lecture execution-time tracking across publish-* releases and environment.yml changes, with regression detection (e.g. the annual Anaconda upgrade cycle). Repo: tool-lecture-benchmark.

Shared foundations

• Measurement standard: the as-used, fresh-process methodology from #717 (actual problem sizes, actual call sequence, JIT compile time included) becomes the house standard across all three workstreams; warm/amortised numbers are reported alongside, never alone.
• Environment descriptors: every stored result records hardware, library versions, and runner type, so results are comparable across time and infrastructure changes.
• Results storage & reporting: one shared format/home for benchmark results (relates to Server & build telemetry — operational dashboard for runner health, build performance & cost #328 build telemetry and the Proposal: GitHub Projects as QuantEcon's PM + reporting layer (central dashboard) #332 dashboard direction).

Sequencing

1. Workstream B first (lectures are user-facing): gather the #717 scripts, build and validate the /eval-py-acceleration skill in QuantEcon/skills, document in the manual.
2. Extract the harness patterns from B to seed the workstream A suite.
3. Wire workstream C into CI for release/environment regression tracking.

Tasks

• Attach PROJECT: Performance Benchmarks Suite for Numba, Dask, JAX (CPU/GPU) #264 and Project: Lecture Performance Benchmarking & Monitoring System #289 as sub-issues of this umbrella
• Scaffold QuantEcon/skills as a Claude Code plugin marketplace with a benchmark plugin (skill + scripts)
• Workstream B: collect the evaluation scripts from the lecture-python.myst#717 thread into QuantEcon/skills (benchmark plugin)
• Workstream B: build /eval-py-acceleration and validate against lecture-python.myst#717 and lecture-python.myst#654
• Workstream B: manual documentation (tracked in QuantEcon/QuantEcon.manual#104)
• Roll out marketplace auto-install settings (extraKnownMarketplaces + enabledPlugins in checked-in .claude/settings.json) across the lecture repos once the skill stabilises
• Adopt the as-used measurement standard in a short written note all workstreams reference
• Define the shared result + environment-descriptor schema (commit, environment hash, runner class, hardware, library versions) — first blocking deliverable: telemetry reports, harness results, and kernel-suite results all write to it (extends Proposal: emit a machine-readable execution_status.json for lecture build/execution status #314; see the telemetry design comment below)
• Telemetry: post-build per-cell execution report from .jupyter_cache (nbclient already records metadata.execution by default) + PR-diff screening comment on preview builds
• Prose feature (separable): an {exec-time} role resolving measured cell durations into lecture text (Sphinx extension now, mystmd plugin post-migration)
• Revisit expansion beyond lectures (QuantEcon.py benchmarks, wasm comparison, runner/cost benchmarking)

Related

• [ge_arrow] Update to JAX and compare runtime lecture-python.myst#717 — origin thread (rubric + evaluation report)
• QuantEcon/QuantEcon.manual#104 — documentation tracking
• Proposal: A QuantEcon Claude Code toolkit (agents, skills, plugins) #304 — Claude toolkit / skills distribution (delivered by QuantEcon/skills)
• Repos: QuantEcon/skills, benchmarks, tool-lecture-benchmark

[mmcky]

Design update: an always-on telemetry layer, and how the three workstreams divide

Working through how the /eval-py-acceleration skill should extract and time lecture code surfaced a cheaper, always-on complement that reshapes the programme's architecture. Recording the design here.

Key discovery: per-cell timings are already being captured

nbclient — the executor under MyST-NB / Jupyter Book — records per-cell execution timestamps by default (record_timing = True, verified on nbclient 0.10.2 in our build environment). Every executed cell already carries metadata.execution timestamps (duration ≈ iopub.status.idle − iopub.execute_input), and jupyter-cache stores the executed notebook with this metadata intact. So every CI build is already generating per-cell timing data; nothing currently surfaces it.

The gap is provenance: the notebook only carries kernelspec/language_info — no hardware, runner class, library versions, or jax_enable_x64 state. Provenance must be attached at report time by the build workflow (commit SHA, environment.yml hash, runner class, CPU/GPU model, key library versions), which knows all of it.

Proposed telemetry pipeline

• Capture — already happening via nbclient (no work).
• Persist — a post-build step reads .jupyter_cache and emits a machine-readable execution report: per cell (label, duration) plus the provenance stamp. This is an extension of Proposal: emit a machine-readable execution_status.json for lecture build/execution status #314 from per-document status to per-cell timing — same artifact, richer schema.
• Compare — PR preview builds already execute changed lectures, so diffing cell timings against the last main build gives an automatic screening comment ("cell solve-ex4 went 8 ms → 950 ms") at zero extra execution cost. This would have flagged the regression in [ge_arrow] Update to JAX and compare runtime lecture-python.myst#717 the day the PR was opened.
• Monitor — reports accumulate into build history and execution statistics (this is workstream C's data source): slowest cells per repo, post-merge drift, before/after distributions for environment upgrades (turns the Anaconda-cycle validation sweeps into a query), runner-class cost attribution.
• Prose (future, separable) — a role like {exec-time} resolving a labelled cell's measured duration into lecture prose. {code-cell} :name: already provides the reference identity; MyST-NB precedents exist (nb-exec-table, glue). Needs a small Sphinx extension now and a mystmd plugin post-migration.

Division of labour across the workstreams

• A calibrates — kernel crossover curves on standardized code, stable over time, independent of lecture edits.
• B diagnoses — the skill: episodic, comparative, controlled. Two implementations of one lecture, same host, K replications, float32/float64 toggles, cold/warm decomposition, recompile audits. Telemetry cannot do counterfactuals — a build never runs two versions of a lecture — so the clinical harness stays.
• C observes — continuous, observational, accumulating; n=1 per build on shared runners, so it screens and trends but does not render verdicts.

Telemetry triggers and pre-loads B (a flagged cell hands the skill its as-used numbers for free); B's verdicts annotate C's history (explaining why a trend line stepped).

Sequencing consequence

The shared result + environment-descriptor schema is the first blocking deliverable — telemetry reports, harness results, and kernel-suite results should all write to it so the datasets join. Task list updated accordingly.

Caveats recorded

Telemetry is n=1 per build on shared runners — good for order-of-magnitude regressions, not fine verdicts. jupyter-cache mixes runner vintages for unchanged notebooks, so benchmark-grade comparisons need cache-busted runs or per-notebook provenance. Cell-level timing measures the whole cell, so keeping solves and plots in separate cells becomes measurement hygiene — arguably a style-guide point of its own.

Cheap first probe

A ~50-line script reading a CI .jupyter_cache and emitting the per-cell timing report for one repo, then diffing the lecture-python.myst#717 preview build against main to see whether the known regression falls out.