Roadmap + spikes (R1/R3/R5) + M1 tail calls + M2 value-model/GC migration (S0–S2) - #118
Open
pmatos wants to merge 20 commits into
Open
Roadmap + spikes (R1/R3/R5) + M1 tail calls + M2 value-model/GC migration (S0–S2)#118pmatos wants to merge 20 commits into
pmatos wants to merge 20 commits into
Conversation
Milestone ladder (M0..M17 interpreter/self-hosted-expander track, B0..B5 NIR/LLVM AOT backend track) over a shared value-model+GC+tail-call substrate, targeting a standalone AOT executable via the self-hosted expander and the NIR/MLIR backend. Produced via a multi-agent research+synthesis+critique pass.
tools/freevars.py is a scope-aware s-expression walker that performs a real free-variable analysis over expander/expander.rktl (tracking lambda/case-lambda/ let-values/letrec-values/define-values binders, skipping quoted data). Robust to the M3 lexing gap that stops the production C++ parser from ingesting the artifact. It reports the exact 544 distinct kernel primitives the expander references (22,173 refs), ranked and categorized in docs/expander-primitive-surface.md. Seeds M4-M7 (issues #96-99).
…closure) Standalone spike (spike/r1-value-model/, not in the norac build) that freezes the shared runtime representation M2 and B0/B2 both consume: - tagged 64-bit nr_value (odd=fixnum, 000=8-aligned heap ptr, 010=singleton, 110=char), 8-byte ObjHeader, flat NrClosure with B2's nr_code signature; - Boehm-Demers-Weiser conservative GC committed for interpreter AND compiled code (moving/precise GC explicitly out of scope). Proofs (all pass, -O2 and ASan): a tree-walking interpreter and a compiled C++ function compute identical results over the same heap and entry points; a flat closure applies identically in both; a garbage loop churns 2.98 GiB through a 0.1 MiB live heap (~24000x) with RSS 4 MiB. ABI frozen in docs/value-model-abi.md.
…keleton Rewrites the broken/stale NIR scaffold into a clean, minimal, representation- agnostic MLIR dialect (the seed of B0): - delete stale duplicate Ops.td (obsolete list<OpTrait> API); - NirOps.td: nir.constant (i64 skeleton) + nir.return terminator, no nr_value-tagged types yet (those follow the M2 ABI freeze); - add nir-opt.cpp (minimal mlir-opt clone registering the dialect) + CMake wiring; add round-trip test test/mlir/nir-roundtrip.mlir. NOT YET VERIFIED: MLIR 22 is not installed on this machine (LLVM 22 is; MLIR is a separate package), so this cannot be built or round-tripped yet. All of it is behind NORA_ENABLE_MLIR=OFF; the default norac build + tests remain green (21/21). See src/mlir/README.md for the one-command verify once MLIR is installed. Issue #90 stays open.
Resolves the build against the now-installed MLIR 22: - src/include/nir/CMakeLists.txt: pass -dialect=nir to the dialect-decls/defs generators (NirOps.td transitively pulls in the builtin dialect); - src/mlir/CMakeLists.txt: PARTIAL_SOURCES_INTENDED on nirLib and nir-opt (two targets share the dir); link MLIRRegisterAllDialects/AllPasses; - fix the nir.constant custom-form example (result type i64 is fixed, so it prints 'nir.constant 42', not '... : i64'); update the round-trip test. Verified: cmake --preset mlir && build nir-opt; nir-opt test/mlir/nir-roundtrip.mlir | nir-opt | FileCheck passes. Default norac build + tests unaffected (still green).
The machine grew Kont by one Frame::Call per call (popped only when the activation returned), so tail loops were O(depth) in continuation space. Two local changes make self- and mutual tail recursion O(1): - continueStep(Frame::Seq): pop the sequence frame before its final expression (non-begin0), mirroring IfBranch, so a tail call there sees the enclosing activation frame on top; - applyProcedure(): when the top of Kont is that activation (Frame::Call), reuse it (move in the new callee, clear its marks) instead of pushing a new Call frame. Added test-first (Catch2 test_interpreter target + lit): tail loop returns the right value; peak Kont is constant across depths (100 vs 100000) and < 16; non-tail recursion still grows the continuation; mutual ev/od recursion is bounded and correct. Exposed Interpreter::getPeakKont() for the assertion and a minimal zero? predicate so a terminating loop can be written (full numeric predicates are M4). 25/25 tests green under debug, asan and ubsan.
First slice of the shared value model. The interpreter clones values on every environment lookup, so mutation and object identity were impossible. Introduce a Box value whose single-slot cell is heap-allocated and *shared*: cloning a Box shares the same cell, so set-box! through one reference is visible through another, and (eq? b b) holds while (eq? (box 0) (box 0)) does not. - ast::Box (ClonableNode) holding a shared_ptr<Cell>; get/set/identity. - box / unbox / set-box! / eq? primitives (eq? uses cell identity for boxes, the structural valueEq otherwise). - visitor plumbing (ASTVisitor/Interpreter/AnalysisFreeVars), AST_Box kind. Built test-first: box round-trip; set-box! visible through a shared reference; eq? distinguishes identity; + integration test box.rkt. 28/28 green under debug/asan/ubsan. The shared_ptr cell is interim; a later M2 slice moves cells onto the Boehm GC heap behind these same tests.
…q? (TDD) Second slice of the shared value model, mirroring the Box slice. ast::Pair is a cons cell whose car/cdr live in a heap-allocated shared_ptr<Cell>; cloning a Pair shares the cell, so set-car!/set-cdr! through one reference are visible through another and (eq? p p) holds while (eq? (cons 1 2) (cons 1 2)) does not. - ast::Pair (ClonableNode) + car/cdr/setCar/setCdr/identity; AST_Pair kind and visitor plumbing. - cons/car/cdr/set-car!/set-cdr! primitives; eq? extended with a pair-identity branch. Built test-first: cons/car/cdr round-trip; set-car!/set-cdr! through a shared reference; eq? distinguishes pair identity; + integration test pair.rkt. 31/31 green under debug/asan/ubsan. (List/Pair unification and car/cdr on quoted lists are a later slice; the shared_ptr cell still moves onto the Boehm GC heap in a subsequent slice.)
Symbols now have object identity, not just name equality. An interned symbol is canonical by name (a global intern table hands out one stable pointer per name); an uninterned symbol (gensym / string->uninterned-symbol) carries a unique token shared across its clones. eq? compares that identity, so: - (eq? 'a 'a) => #t (interned, canonical) - (eq? (string->uninterned-symbol "s") (string->uninterned-symbol "s")) => #f - (eq? (gensym) (gensym)) => #f - ast::Symbol gains identity()/isInterned()/makeUninterned() + a shared_ptr uninterned token; eq? extended with a symbol-identity branch. - string->uninterned-symbol and gensym primitives. - Wired the existing parseString into parseExpr: string literals now parse as expressions (a pre-existing gap that blocked string->uninterned-symbol "s"; a '"'-token matches no other expression parser, so this is safe). Built test-first: symbol eq? is identity not name; gensym distinctness; + integration symbol.rkt. 33/33 green under debug/asan/ubsan.
Multi-agent design pass. Decision: GC-backed ValueNode (keep the class hierarchy + visitor RTTI; GC-allocate, strip RAII members, share instead of clone), reusing only R1's nr_value *immediate* encoding (the vptr sits at offset 0, not an ObjHeader, so R1's object accessors are NOT reused in M2). Transition scaffolding (legacy pin table + GC keep-alive root) keeps every slice green with no destructor-leaking / cross-heap-dangling intermediate. Forcing seam: a GC-heap-size hook (depth-independent live-heap plateau), not RSS. Slices S0..S18; capstones at S13 (value garbage) and S17 (scope garbage, deletes AllScopes).
… (TDD)
First slice of the value-model+GC migration (docs/value-model-gc-migration.md).
Brings the collector up without touching the value representation yet:
- src/nora_rt.{h,cpp}: the R1 nr_value IMMEDIATE ABI (fixnum/char/bool/singleton
tags) promoted into the build. R1's ObjHeader-based object accessors are
deliberately NOT promoted (M2 cells are polymorphic, vptr at offset 0, not an
ObjHeader) and GC_set_all_interior_pointers is dropped.
- CMake: pkg-config bdw-gc (PkgConfig::BDWGC) wired into norac + both unit exes.
- GC_INIT() first in norac main and a shared CATCH_CONFIG_RUNNER test main
(both test exes switched off CATCH_CONFIG_MAIN) so Boehm records the stack
bottom on the main thread.
- Interpreter::getGCHeapSize()/getGCTotalBytes() hooks for the forcing seam.
Test-first: test_gc.cpp exercises the immediate ABI + GC_MALLOC/heap-size (RED
before libgc was linked). 34/34 green under debug, asan, ubsan; norac end-to-end
unchanged. GMP-through-GC hook is deferred to S11 per the plan.
Move the Kont vector's backing store onto the Boehm heap so that, as values migrate onto the GC heap in later slices, in-flight values held in continuation frames stay reachable (the Kont header lives in the stack-resident Interpreter, so Boehm's stack scan roots the buffer). Behaviour-preserving; frame elements are still legacy unique_ptr/shared_ptr and are destructed normally. - src/include/gc_alloc.h: GcAllocator<T>, a minimal exception-free allocator over GC_MALLOC (scanned). Boehm's own gc_allocator needs -fexceptions, which this codebase disables; deallocate is a no-op (conservative-GC-safe). - Kont: std::vector<Frame> -> std::vector<Frame, GcAllocator<Frame>>. Scoped to Kont (self-contained: Interpreter.cpp uses only the vector API on it). Frame::Done and the mark containers move in later slices (they flow into default-allocator params / have the keep-alive-root backstop). Forcing test at the GC-heap seam: a deep non-tail loop now churns >100 KB of GC bytes (~0 before, since nothing was GC-allocated during eval). 35/35 green debug/asan/ubsan.
Thread a Value handle (src/include/Value.h) through the machine's value register (Val) and linklet Result. Value is the migration vehicle (docs/value-model-gc-migration.md §3): it will become a bare nr_value word (immediate | GC pointer | legacy pin-index) so GC cells can hold it, but in this phase it simply carries the legacy heap ValueNode by unique_ptr and is behaviourally identical to it. Move-into-register sites (deliver, = nullptr, = Last) are unchanged via implicit ctors; the 14 move-out sites use Value::takeLegacy() to hand the unique_ptr to the still-legacy frame/env slots (those migrate in S3/S4). Behaviour-preserving refactor; the whole suite is the guard: 35/35 green under debug/asan/ubsan.
This was referenced Jul 9, 2026
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1740c9d94e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // activation (Frame::Call), reuse it instead of stacking a new one. Together | ||
| // with popping Seq/if/let-body frames before their tail sub-expression, this | ||
| // makes self- and mutual tail recursion run in O(1) continuation space. | ||
| if (!Kont.empty() && Kont.back().K == Frame::Call) { |
There was a problem hiding this comment.
When the tail call occurs as the result expression of with-continuation-mark, Frame::WcmMark is still on top of Kont while that expression is evaluated, so this check fails even though the application is in tail position and the code falls through to Kont.emplace_back(Frame::Call). A loop such as (lambda (n) (with-continuation-mark 'k n (if (zero? n) 0 (loop (- n 1))))) therefore grows one activation per iteration rather than satisfying M1's proper-tail-call guarantee for this existing form.
Useful? React with 👍 / 👎.
| auto const *SB = llvm::dyn_cast<ast::Symbol>(B); | ||
| Eq = (SB != nullptr) && SA->identity() == SB->identity(); | ||
| } else { | ||
| Eq = ast::valueEq(*A, *B); |
There was a problem hiding this comment.
When one operand to eq? is a quoted symbol, it reaches this fallback as an ast::QuotedExpr, so the symbol identity branch above is skipped and valueEq unwraps it then compares Symbol::operator== by name. This makes (eq? 's (string->uninterned-symbol "s")) return #t, contradicting the new uninterned-symbol identity semantics; unwrap quoted datums before dispatching or make this path use Symbol::identity() for symbols.
Useful? React with 👍 / 👎.
| const llvm::SmallVector<const ast::ValueNode *> &Args) const override { | ||
| static unsigned Counter = 0; | ||
| std::string Base = "g"; | ||
| if (Args.size() == 1) { |
There was a problem hiding this comment.
For calls with two or more arguments this condition is simply skipped, leaving the default base and returning a fresh symbol. That means (gensym 'a 'b) succeeds instead of reporting the same invalid-arguments error pattern used by the other primitives, so arity mistakes are silently accepted; reject Args.size() > 1 before constructing the symbol.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
A foundational branch that lays out the road to a compiled
#lang racket/basehello-world and starts down it: the roadmap + design docs, the three
de-risking spikes (R1/R3/R5), M1 (proper tail calls), the first three
M2 value-model slices, and the first three slices of the M2 Boehm-GC
migration (S0–S2). All work was done test-first (TDD); the suite is 35/35
green under
debug,asan, andubsanat every commit, and the default buildplus
noracend-to-end are unchanged.This is intentionally a large, mostly-additive PR (54 files, ~+3.5k/−250). M2 is
the multi-month "pivot" and is not finished here — remaining GC-migration
slices S3–S18 are filed as tracked, dependency-ordered issues (see below).
What's done
Planning & design
ROADMAP.md— staged milestone ladder (M0–M17 interpreter/expander track, B0–B5NIR/LLVM AOT backend track) with per-milestone tracking issues (Spike R1 — value-model / GC ABI (tagged nr_value + Boehm + flat closure) #88–B5 — ./hello from #lang racket/base (FINAL GOAL) #115).
docs/value-model-abi.md— the frozen taggednr_valueABI (from R1).docs/expander-primitive-surface.md— the expander's exact primitive surface.docs/value-model-gc-migration.md— the 19-slice (S0–S18) value-model + Boehm-GCmigration plan (from a multi-agent design pass): representation decision,
transition scaffolding, forcing seam, and adversarial critique folded in.
Spikes (closed issues)
tools/freevars.py: a scope-aware free-variable walk overexpander.rktlmeasuring the real primitive surface = 544 distinctidentifiers (corrected the ~350 estimate).
spike/r1-value-model/: proves a taggednr_value+ Boehm GC +flat closure; a tree-walker and compiled C++ compute identical results over one
heap; a garbage loop churns 2.98 GiB through a 0.1 MiB live heap.
nir.constant/nir.returndialect +nir-opt;cmake --preset mlirbuilds itand the round-trip test passes (MLIR 22). Opt-in; default build untouched.
M1 — proper tail calls (#93, closed)
Seqearly-pop +Callframe reuse).
Interpreter::getPeakKont()hook; minimalzero?. Unit tests +tailcall.rkt.M2 — shared value model (#94, in progress)
eq?identity (shared cell survives clone-on-lookup).cons/car/cdr/set-car!/set-cdr!) + paireq?.eq?,gensym,string->uninterned-symbol; also wiredparseStringintoparseExpr(string literals now parse as expressions).
pkg-config bdw-gc),GC_INIT, promote thenr_valueimmediate ABI tosrc/nora_rt.*, heap hooks.Kont) now lives on the GC heap (exception-freeGcAllocator, since Boehm's own needs-fexceptions).Valuemigration handle in theVal/Resultregisters.Not done here / follow-ups
of clone in the environment; immediates; the collectable-leaf scaffolding;
containers-as-GC-cells; the scope cutover that deletes
AllScopes/clone()).Filed as dependency-ordered issues blocking M2 — Shared value model + GC + closure representation (PIVOT) #94; see
docs/value-model-gc-migration.md.List/Pairunification socar/cdrwork on quotedlists;
Frame::Done/mark containers onto the GC heap (later slices).Testing
ctest --preset debug/asan/ubsan: 35/35.noracend-to-endintegration tests (
box.rkt,pair.rkt,symbol.rkt,tailcall.rkt) pass.clang-format clean. The
mlirpreset (opt-in) builds and round-trips.Closes #88, closes #89, closes #90, closes #93. Advances #94.