Skip to content

[Tracking] Modeler-agnostic transcription: one recipe in CTDirect, any NLP modeler #673

Open
0 of 5 issues completed

Description

@ocots

Written by Claude (Claude Code) at the request of ocots, from a design discussion with ocots.

Objective

CTDirect writes each transcription once, without knowing the NLP modeler underneath. Adding a modeler (JuMP, for instance) then means implementing two contracts next to the modeler, in CTSolvers, with no change in CTModels, CTDirect or CTParser.

This must be done before §3 (transcription recipe) and §5 (own the Exa transcription) of the CTDirect redesign (#610, discussion #609): it fixes the NLP emission axis that the recipe is written against.

Problem: the transcription exists three times

Modeler Where the transcription lives Problem
ADNLP CTDirect (src/ode/*.jl) calls CTModels through a single convention, the in-place f!(r, t, x, u, v)
Exa CTParser, code generated at @def time (onepass.jl, if scheme == … chains) 4 schemes, no custom grid, no path/boundary multipliers, only for @def problems; the result is stored in Model.build_examodel, so CTModels holds an already discretized object
JuMP OptimalControlProblems, by hand (control-toolbox/OptimalControlProblems.jl#383) a third copy

Each new modeler needs a new transcription, and each new scheme must be written in every copy. The scheme conventions already drift between copies (final control, aliases: control-toolbox/CTParser.jl#355, time grid: #638).

Principle

One transcription, in CTDirect. CTDirect holds the modeler and passes it to every API call; that argument drives the dispatch. CTDirect uses two APIs, each taking the modeler:

  1. Evaluate the OCP: get the dynamics, costs and constraints in the form the modeler consumes. API of CTModels.
  2. Build the NLP: declare variables, families of constraints and objective terms. API of CTSolvers, where the modelers live.

Roles

CTModels: describe the problem, select the representation

  • Each component (dynamics, Lagrange, Mayer, path and boundary constraints) stores several redundant representations, consistent by construction when CTParser fills them from one expression:
    struct Dynamics{F,G,E,S}
        inplace::F      # f!(r, t, x, u, v): always present
        traceable::G    # generic out-of-place form, per component, or nothing
        expression::E   # Expr per component, or nothing
        sparsity::S     # dependencies of each component on (t, x, u, v), or nothing
    end
  • Traits tell what is available (has_traceable, has_expression, has_sparsity).
  • Forms and selection: CTModels defines the forms (InPlaceForm, TraceableForm) and the generic function required_form. The API takes the modeler, for dispatch and for its options (Strategies.options(m), e.g. base_type on GPU):
    dynamics(ocp, modeler) = dynamics(ocp, required_form(modeler), modeler)
  • The API stays continuous (t, x, u, v): no grid, no scheme.
  • A hand-written model only gives inplace: it works with ADNLP, and Exa or JuMP throw a clear error unless a traceable form is declared.
  • The expression is not used for evaluation (that would need eval): it serves analysis (true sparsity, linear or box constraints), display, and as a fallback for code generation.

CTSolvers: the modelers (they stay there)

A modeler is an NLP concept (it already serves non-OCP problems, e.g. test/problems/rosenbrock.jl) and goes with the solvers (MadNLPGPU needs Exa on GPU). Thanks to required_form, CTModels does not need to know the modelers, and the dependency graph CTBase ← CTModels ← CTSolvers ← CTDirect does not change.

  • Each modeler declares the form it consumes: CTModels.required_form(::Modelers.Exa) = CTModels.TraceableForm().
  • Each modeler implements the generic NLP-building API, modeler as argument:
    X = add_variables!(nlp, modeler, (n, N+1); lvar, uvar, start)   # opaque handle
    c = add_constraints!(nlp, modeler, body, indices; lcon, ucon)   # a family
    add_objective!(nlp, modeler, body, indices)
    model = build(nlp, modeler)
    Exa: add_var/add_con/add_obj on generators, traced once at build time (the tree is evaluated, also on GPU). ADNLP: families accumulated, then f and c! loop over them. JuMP: @variable/@constraint by operator overloading.
  • The handles are used to read back the primal values and the multipliers the same way for every modeler.

CTDirect: the recipe, written once

Schemes (Butcher tableaux), layout, grids, families, quadrature. Optimized schemes (midpoint, FSAL trapeze) stay specializations by dispatch (#609 §3.3).

f = CTModels.dynamics(ocp, modeler)
X = add_variables!(nlp, modeler, (n, N+1); …)
U = add_variables!(nlp, modeler, (m, N+1); …)
defect(j) = X[j+1] - X[j] - h/2 * (f(t[j], X[j], U[j], v) + f(t[j+1], X[j+1], U[j+1], v))
add_constraints!(nlp, modeler, defect, 0:N-1; lcon=0, ucon=0)

CTDirect would then no longer depend on ADNLPModels, ExaModels or CTParser.

CTParser: from expressions to the model

Translates the @def expressions into a Model with every representation of each component. No discretization any more: no grid, no scheme, no build_examodel.

Where each thing goes

Concept Package
OCP description, multi-representation components, traits CTModels
Forms, required_form (generic), dynamics(ocp, modeler) and the selection CTModels
Modelers (strategies and options) and their required_form method CTSolvers.Modelers
NLP-building API, handles, reading the multipliers CTSolvers.Modelers (contract), one extension per modeler
Linear algebra on Exa nodes (today CTParserExaModels, control-toolbox/CTParser.jl#365) CTSolversExaModels, or upstream in ExaModels
AbstractDiscretizer contract CTSolvers.DOCP (unchanged)
Schemes, layout, grids, recipe, solution rebuild CTDirect
Expressions to every representation CTParser
build_examodel removed

Open technical point: one calling convention

To write defect(j) once, the callable CTDirect gets from dynamics(ocp, modeler) must have the same calling convention for every modeler. Proposal: CTParser generates an out-of-place form returning a tuple or an SVector (scalar in 1-D, DIM1): no allocation, ForwardDiff-compatible, traceable, so one function could serve both ADNLP and Exa. For a hand-written model with only f!, CTModels adapts it. Benchmarks on midpoint and trapeze against the current in-place code must show no loss.

Work items

Order: #674 → control-toolbox/CTModels.jl#516 → control-toolbox/CTSolvers.jl#319 and control-toolbox/CTParser.jl#415 → CTDirect recipe → control-toolbox/CTSolvers.jl#320.

Related

Activity

  1. ocots commented on Nov 6, 2026

    @ocots
    MemberAuthor

    Written by Claude (Claude Code) at the request of ocots.

    The JuMP models of OptimalControlProblems are now a small prototype of this design, on the JuMP side. Notes for #674, control-toolbox/CTModels.jl#516 and control-toolbox/CTSolvers.jl#319.

    What exists there.

    Points for the design.

    1. Calling convention. The functions of ProblemData return a Vector. That is fine for JuMP, but it allocates; [Tracking] Modeler-agnostic transcription: one recipe in CTDirect, any NLP modeler #673 proposes a tuple or an SVector, and Prototype: one transcription for ADNLP and Exa (trace-based Exa, out-of-place ADNLP) #674 should settle it.

    2. Constraint order is not portable. MOI passes the constraints to the solver grouped by type (function × set). The order of creation is kept only within a type. On 21 problems, grouping the constraints by kind changed the order seen by the solver on one problem only; the order of the variables is what changes. The NLP-building API should promise an order of the variables, not of the constraints across families.

    3. Handles. build stores model[:constraints], the constraints in CTDirect's order whatever the layout, and the dynamics constraints by component (model[:∂x₁]) for the costate. This is the role of the handles of Generic NLP-building API dispatched on the modeler, and required_form for ADNLP and Exa CTSolvers.jl#319 for reading the multipliers.

    4. Scaled models. (Updated.) space_shuttle used to write the JuMP dynamics on the true altitude and velocity, through ProblemData.state_expressions, so its rows were scaled by 1e5 and 1e4 with respect to CTDirect's. It now writes them on the scaled state, as the OptimalControl model does, and state_expressions is gone ([Tests] NLP equivalence tests between JuMP, ADNLP and Exa OptimalControlProblems.jl#371). Nothing to represent here, then.

    5. Oracle. The hand-written JuMP models stay an independent oracle until JuMP modeler (after the generic NLP-building API) CTSolvers.jl#320 exists. Afterwards, comparing them with JuMP-through-CTDirect will test CTDirect's transcription itself.

    6. Initial point of the Exa model. CTParser declares N+1 controls for every scheme, and it initialises the control at the last node with the value of node N-1, for the trapezoidal rule too (chain: -2, 0, 2, 4, 4 against CTDirect's -2, 0, 2, 4, 6). The initial point therefore differs from CTDirect's, even when the equations are the same. The new transcription should take the initial point from the scheme, as CTDirect does. [Tests] NLP equivalence tests between JuMP, ADNLP and Exa OptimalControlProblems.jl#371 reports this by computation, without failing.

    OptimalControlProblems tracks its side in a theme issue, one source of truth per problem, with this issue as the target:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions