Repository navigation
Roadmap
#288
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
CTParser Roadmap v1
Last updated: July 7, 2026
Context
CTParser (v0.8.15) turns the
@defDSL into executable model-building code, through two parsing backends living side by side in onepass.jl (~1500 lines)::fungenerates function-based CTModels code,:exagenerates a full ExaModels transcription (variables on a grid, scheme-dependent dynamics defects, cost quadrature, constraints). The package is a flat module (4 files:defaults.jl,utils.jl,onepass.jl,initial_guess.jl), with generated code addressed through mutable prefixRefs (PREFIX_FUN/PREFIX_EXA).This roadmap is the companion of the CTDirect roadmap v1: its single most structural item — moving the discretization out of the parser (§2) — is the CTParser side of CTDirect roadmap §5 ("Own the ExaModels transcription in CTDirect"). The guiding principle:
The rest re-scopes the open issues into: ExaModels 0.11 migration (§1), "1-D = scalar" simplification (§3), prefix removal (§4), parsing robustness (§5), syntax extensions (§6), and the usual architecture/tests/docs alignment with the Handbook (§7–§9).
Current pain points (evidence)
:exabackend is a second discretizer:p_dynamics_exa!hard-codesif scheme == :euler … :midpoint … :trapezechains in generated code (onepass.jl:918-941, 1022-1035),p_lagrange_exa!duplicates the quadrature per scheme (1095-1103) — a parallel, restricted (4 schemes, uniform grid) implementation of what CTDirect does.PREFIX_EXA = Ref(…),prefix_exa!) route generated code to backend modules — fragile global state, #224.onepass.jlis a 1500-line monolith mixing the pattern-matching front-end, both codegen backends, and the@defentry points; header TODO comments (# - exa: …) accumulate.control_funvs:control_fun,as_rangereturning aVectorwhere an expression-level range is expected,__wrapscoping).x/usymbols afterreplace_calldon't error (#193);x(t0)witht0undefined in the calling scope parses without a clear error (#139); slice notation support is partial (#260, PR #251 marksend-keyword and variable-range cases broken).test_onepass_*.jl,*_bis.jlduplicates) not following the Handbook template; docs are a singleindex.mdon plain Documenter.1. ExaModels 0.11 migration
Status: to do first — it is a compat blocker for the whole ecosystem (CTDirect and CTSolvers also pin
ExaModels = "0.9").The generated
:exacode callsExaModels.variable,constraint,constraint!,objective,solution,multipliers, plus theExaCore/ExaModellifecycle (through the prefix indirection). Work items:p_variable_exa!,p_state_exa!,p_control_exa!,p_constraint_exa!,p_dynamics_exa!,p_lagrange_exa!,p_mayer_exa!, and the solution getter block around onepass.jl:1426-1434) against the 0.11 API; replace deprecated forms.ExaModels = "0.11"intest/Project.tomlhere, and coordinate the bump in CTDirect / CTSolvers / OptimalControl (one PR train, since the generated code and the caller must agree).:exatest suite (test_onepass_exa*.jl,test_dynamics_exa.jl,test_exa_linalg.jl) + CTDirect's Exa problems as the cross-package check; GPU smoke test if available.2. Move the discretization out of the parser
Status: the structural item; joint with CTDirect roadmap §5 — the decision (function tracing vs expression-level IR) is made in the CTDirect §3/§5 design notes, this section is the CTParser side. Absorbs: #65 (non-uniform grid for exa), #161 (endpoint-control constraints per scheme), #134 (exa multipliers for boundary/path constraints).
Goal: the
:exaparsing backend stops generating scheme-dependent transcription code. What remains in CTParser depends on the route chosen::exabackend at all —@defemits only the:funpath, and the parser's job ends at building the CTModels model. The:exacode paths,PARSING_EXA, prefix machinery for Exa,get_build_examodelplumbing are deleted once parity is reached.if scheme == …remains here.Consequences for open issues
time_gridhandling (already supported on the ADNLP path) — nothing to implement in CTParser; close it against the repatriation.u_N = u_{N-1}for euler/midpoint): scheme knowledge → CTDirect's recipe (final-control trait, CTDirect §2.2/§3). If a short-term fix is wanted before the repatriation, implement it in the generated code behind the existing scheme branches, but mark it temporary.Transition discipline
The current
:exabackend stays frozen (bug fixes only, no new features) and serves as the comparison oracle for CTDirect's prototypes: same NLP dimensions, same solutions on the shared problem suite. It is deleted (route 1) or reduced to the IR emitter (route 2) once CTDirect's golden tests pass on both paths.3. "1-D = scalar" simplification
Reference: Handbook dimension-and-shape.md. Status: to align; the parser is the emitter of user-facing code, so its compliance shapes everyone downstream.
Per the Handbook, the parser may emit
[1]-indexed code (works with scalars) or scalar code; either honours the convention — what it must not do is force 1-vectors on the callers or wrap scalars into vectors.:funcodegen for 1-D state/control/variable: emitted dynamics/cost/constraint closures should work when callers (CTFlows, CTDirect after its §7) pass scalars —[1]-indexed emission is the safe common denominator; drop anyx → [x]vectorization at emission boundaries.dim == 1with wrapping/unwrapping should collapse to the singleonly/identitydimension-driven rule; remove now-useless scalar/vector duplication inutils.jlsubstitution helpers where applicable.4. Remove prefixes
Issue: #224. Status: decided in principle; small, high-leverage cleanup.
Generated code currently reaches CTBase/CTModels/ExaModels through mutable prefix
Refs (PREFIX_FUN,PREFIX_EXA,prefix_fun!,prefix_exa!) so that the caller (OptimalControl) can point them at re-exported modules. Instead:@reexport(or plain re-export) the needed module names from OptimalControl.jl, and prefix the generated code directly withCTModels.…/CTBase.…/ExaModels.…ab initio.Refs and setters — removes global mutable state, makes generated code readable and greppable, and kills thetest_prefix*.jlspecial suite.:funprefix removal regardless.5. Parsing robustness — bugs, strictness, slices
Issues: #197, #193, #139, #260 (finalize, with PR #251).
5.1 Bug sweep (#197)
Work through the catalogued items with a test per fix:
@matchpatterns using barecontrol_funinstead of:control_fun— fix everywhere, add a regression test that would have caught the mis-route.as_rangereturning[x]where quoted code expects a range expression — introduceas_range_expr(x) = is_range(x) ? x : :($x:$x)and use it consistently in codegen paths.__wrapscoping hazard (local ex+throw(ex)) — tighten the generated error-handling block.5.2 Stricter parsing (#193, #139)
Silent mis-parses must become parse-time errors (
ParsingErrorwith position and hint):replace_call, any leftover state/control symbol in an expression is necessarily a typo → error (not a gensym that fails later at runtime; the issue suggests gensym as the weaker fallback).x(t0)whent0is not the declared time boundary ([Dev] Improve parsing #139): detect at parse time that the argument of a state/control evaluation is neither the declaredt,t0,tfnor a literal within the time range semantics — error with "did you meanx(0)?"-style hints. (The issue notes the tricky case wheret0exists in the eval scope — hence a parse-level rule on declared symbols, not an eval-level one.)5.3 Slice notation — finalize #260
PR #251 added comprehensive slice tests with known-broken cases. Finish the feature:
x[1:3](t),x[1:end](t),x[1:n](t)(withnthe declared dimension) uniformly in constraints, dynamics (∂(x[1:3])(t)), and costs — both backends.endkeyword and symbolic bounds require resolving against declared dimensions at parse time — the parser knows them; lowerend/nto literals during normalization.@test_brokencases; keep the test matrix from Add comprehensive slice notation tests #251 as the acceptance criterion.6. Syntax extensions
Grouped because they are independent, user-visible, and each needs an explicit go/no-go with JB.
6.1 Coordinate-derivative aliases (#71)
Accept
ẋ₁(t) == …/ẋ1(t) == …as sugar for∂(x₁)(t) == …. Pure front-end: recognize the combining-dot-above character on state (component) names during normalization (Unicodedep already present). Small; needs a rule for whatẋmeans whenxis a vector (whole-state dynamics) vs a component alias.6.2 Tensors / matrix-valued state and control (#181)
Go from
x[i, j](component × grid) tox[i, ii, (k,) j]— matrix (or complex-matrix) valued states, matrix-valued controls (extdet examples). This is not a parser-only feature: CTModels (storage, dimensions), CTDirect (layout flattening, sparsity), CTFlows (coercion) must all agree on the representation.x ∈ M(n, m),ℂcase), how it is stored (flattened with a shape trait? — relates to the Handbook dimension-and-shape philosophy), what the generated functions receive (a reshaped view? scalars stay scalars).6.3 Incremental
@def!(#32)Build a problem incrementally, interpolating an existing one (
@def! $ocp begin … end, à la@benchmark). Useful for parametric studies and continuation. Design questions: what is re-parsed vs mutated (CTModels models are immutable — so this is "parse extra clauses, rebuild"), interaction withinitial_guess, and whether@defgains$-interpolation of scalar parameters as a first step (cheaper, frequently requested). Prototype the$-interpolation first.7. Parsing engine redesign — staged pipeline over a typed IR
Reference: Handbook modules.md, types-traits-interfaces.md. Status: design proposal, not started. Best done after (or jointly with) the §2 route decision, since the IR is exactly the "neutral expression representation" of route 2.
The core problem of the current engine: the "one pass" does everything at once — syntax recognition, semantic analysis and code generation are interleaved inside each
@matcharm and duplicated per backend. Robustness bugs (§5) and extension cost both stem from this.Proposal: a staged compiler-style pipeline where each
@defline is parsed into a typed IR node (data, not code), validated, and only then emitted:Why more robust
LineNumberNode; nothing is silently swallowed. The strictness rules of §5.2 become centralized stage-2 checks on data instead of guards scattered through codegen.@defcan report all its errors with positions in one shot, instead of dying on the first.Why still simple to extend
validate!method +emitmethods — plain open multiple dispatch (OCP), no edit to a central@match. MLStyle stays, inside each rule, for what it is good at.emitmethods, not duplicating the grammar.PARSING_DIRidea, at production granularity, returning data) suffices.Synergies
@def!(Incremental @def! #32) becomes IR merging (re-validate, re-emit), not surgery on generated code.end([Bug] Comprehensive Slice Notation Tests #260, §5.3) resolve at stage 2 where dimensions are known.@defsyntax gives "did you mean" error hints, an auto-derivable grammar reference (§9.2), and doctests that cannot drift.Submodule split (follows the stages)
IRGrammarparse_linerulesonepass.jl, parts ofutils.jlSemanticsvalidate!methods, strictness (§5.2), error accumulation/reportingParsingInfo(p), scattered checksFunBackend:funhalf ofonepass.jlExaBackend(transitional):exacodegen, then deletion (route 1) or replaced by the IR handoff itself (route 2):exahalf ofonepass.jlInitialGuess@initparsing (same pipeline, own productions)initial_guess.jlDefaultsdefaults.jlHandbook rules apply: one directory + same-named manifest per submodule, top-level
CTParser.jlexports nothing, qualified imports (using MLStyle: MLStyle— currently bareusing MLStyle,using Parameters,using OrderedCollectionsall pollute scope), naming conventionssymbol/_helper/__default, dependency DAG (IR→Grammar→Semantics→ backends).Also fold in: move the header TODO comments of
onepass.jlinto tracked issues; kill the*_bis.jlfile-duplication pattern (belongs to §8).Deliverable order: (1) short design note fixing the IR node set and stage contracts, written together with the §2 route decision; (2)
IR+Grammarfor the declaration productions only, oracle-tested against the current parser; (3) semantics + strictness rules (§5.2 lands here, on the new engine); (4)FunBackendemission, full-suite parity; (5) switch + delete one-pass.8. Test suite restructuring
Reference: Handbook testing.md. Status: not compliant (flat scripts,
_bisduplicates, sharedutils.jl).test_<name>()entry redefined at outer scope, top-level fakes,VERBOSE/SHOWTIMINGviaMain.TestOptions.suite/grammar/(productions, normalization, aliases, slices — the Add comprehensive slice notation tests #251 matrix lives here),suite/strictness/(error paths of §5.2 — everyParsingErrorwith@test_throwsand message checks),suite/codegen_fun/(generated CTModels code: build + call the functions, scalar/vector 1-D equivalence of §3),suite/codegen_exa/(frozen oracle suite while §2 transitions),suite/initial_guess/,suite/integration/(end-to-end@def→ solve smoke tests, shared problems with CTDirect).:exatests double as the comparison baseline for CTDirect's repatriation (§2) — keep them green and pinned until the switch.*_bis.jlshadow suites into their primaries; remove order dependence; keepcoverage.jltooling.9. Documentation
Reference: Handbook VITEPRESS-DOC.md. Status: minimal (single
index.md).make.jl,generate_template,config.mtspatches).onepass.jlpattern matches.@init), backend story after §2 (what:exameans for users — ideally: nothing, it's a solver-side choice), migration notes for removed prefixes (§4) and any syntax changes (§6).Priority summary
Priority 0:
High priority:
:exameanwhile.Medium priority:
FunBackendemission.:exaoracle suite first (it guards §2 and §7).$-interpolation prototype (Incremental @def! #32) — small syntax wins (#71is a stage-0 normalization rule in the new engine).Lower priority:
Dependency graph
Coordination points
All reactions