feat(tailcalls): opt-in return_call codegen via --experimental-tailcalls

[ggreif]

Summary

End-to-end opt-in support for wasm tail calls (return_call / return_call_indirect AST in wasm-exts; TailCallPrim IR; producer pass; codegen for direct calls), gated behind the new --experimental-tailcalls flag. With the flag on, all tail calls — both self-recursive and cross-function — lower to wasm return_call, giving bounded stack for mutually-recursive / VM-dispatcher-shaped code at a small cycle benefit relative to the existing self-tail-call → loop rewrite. With the flag off, current behaviour is preserved exactly.

This PR was originally scoped as a broader wasm-exts sync to upstream wasm-2.0.2 (SIMD, reference types, multi-memory, …). After confirming that tail calls are not in wasm-2.0.2 (the OCaml interpreter at that release predates the proposal merge — see .claude/plans/wasm-exts-update.md), the tail-call work was carved out as its own self-contained slice on top of the existing wasm-exts. The broader 2.0.2 sync remains future work.

What's delivered

layer	commit	what
wasm-exts	f2c69f3e7	ReturnCall / ReturnCallIndirect AST variants, smart constructors, encoder (opcodes 0x12 / 0x13), decoder (opcodes removed from the illegal-list)
CLI	a7ebc369d	--experimental-tailcalls flag wiring in flags.ml + moc.ml (no behaviour change yet)
IR	dd2d8337a	TailCallPrim of Type.typ list declaration + exhaustive plumbing through interpreter, type-checker, effect inference, async lowering, both backends
codegen	ef2c7fb0c	compile_classical.ml and compile_enhanced.ml direct-call arm: is_tail derived from prim, emits ReturnCall <fi> and skips trailing FakeMultiVal.load when the prim is TailCallPrim
producer	4dbdb1712	tailcall.ml gains a third arm: in tail position, when the self-recursion loop-rewrite doesn't apply and --experimental-tailcalls is set, emit TailCallPrim instead of CallPrim
Shared-fn fix	6491d605c	tailcall.ml FuncE arm gates tail_pos = true on s = Type.Local. Shared bodies are descended with tail_pos = false, so the producer cannot label the wrapper-to-body call as tail-position. Fixes a cleanup-bypass trap that surfaced once gauss exercised the post-update-message path. Also passes --enable-tail-call to wasm-validate in the test-runner (test-runner/src/run_test.rs).
flag-gated loop elision	e84769da5	tailcall.ml self-tail-call → loop rewrite is now gated not !Flags.experimental_tailcalls. With the flag on, self-tail calls fall through to TailCallPrim and become return_call; with the flag off, the legacy loop rewrite still fires.
bench	fb38c1b24, e24ebc003, 6491d605c, e84769da5	test/bench/tailcall.mo: Hutton/Bahr-style stack VM running fak (mutual TCO) + naïve self-recursive foldLeft running 5-Year-Old Gauss (loop-rewrite vs return_call). Self-documents via //MOC-FLAG --experimental-tailcalls.

Empirical (EOP, --experimental-tailcalls on)

benchmark	shape	cycles
hutton	mutual-tail-recursive VM dispatcher, fak(10) × 1_000	25_416_088
gauss	self-tail-recursive foldLeft (+) 0 [1..100] × 10_000	101_400_270

Comparisons (same tree, same EOP setting, only the flag flipped):

• hutton (mutual): −636_000 cycles (~2.5%) vs call. Pure cost-table delta (Call=5 → ReturnCall=3 × dispatch count). Also bounded-stack — VM dispatchers can run forever without growing the wasm stack.
• gauss (self-tail): −9_120_000 cycles (~8.2%) vs the loop rewrite. The loop rewrite has its own overhead (mutable arg-temps + loop/block wrapper) that return_call skips. Removing the loop rewrite under the flag is cheaper, not just stack-bounded.

Design notes

• Pipeline order matters. tailcall_optimization runs after async_lowering (pipeline.ml:796-797), so by the time the producer arm sees a CallPrim, awaitable / IC calls have already been desugared. The producer therefore cannot mistakenly tag a shared call as a tail call.
• Codegen scope: direct calls only. TailCallPrim is honoured in the SR.Const Const.Fun (..., mk_fi, _) arm (known function index → emit ReturnCall <fi>). Closure calls (Type.Local via call_indirect) and shared calls fall through to the regular path even if the prim is TailCallPrim. Extending to return_call_indirect for computed tail-calls is described in the plan as a follow-up.
• Validation argument. Wasm return_call requires the callee's wasm result type to match the enclosing function's. Motoko's all-multival-via-side-channel ABI gives every wasm function the result type [] (or a single i64), so the constraint holds trivially for direct calls between Motoko functions.
• Shared-function bodies excluded. A Motoko public func foo() : async () = body compiles to a wasm wrapper of the form message_start ; user-body ; message_cleanup (state-machine transition + GC). The body is not in tail position from the wasm wrapper's perspective. Letting return_call escape the body would skip the cleanup, leaving the lifecycle stuck at InUpdate — the next update message then traps with 'internal error: unexpected state entering InUpdate'. The fix is to descend Shared FuncE bodies with tail_pos = false.

Future work (scoped out, kept in the plan)

• Broader wasm-exts 2.0.2 sync — the original ambition of this branch (SIMD, reference types, multi-memory, exception handling, …). Tail calls are not in 2.0.2 anyway, so the slice was carved out independently.
• Source-level annotation (with tailcall) — per-call surface knob (presence-form via record-punning on a let tailcall = true) with a compile-time Bool-const constraint and a tail-position warning. Useful workaround for mo:core recursive algorithms.
• return_call_indirect for computed tail-calls — VM dispatchers that index into a handler table, CPS continuations, dynamic dispatch in interpreters. Requires extending Closure.call_closure. The wasm-exts AST/codec already supports it; codegen lowering does not.

See .claude/plans/wasm-exts-update.md for the full design treatment, including open design questions for the annotation and the IC pricing measurements that informed the empirical comparison.

Related

• .claude/plans/abstract-interpreter.md — background on the ConstTrack abstract interpreter (relevant when emitting new instructions).
• fix: instructions for Operator::ReturnCallIndirect dfinity/ic#10086 — concurrent IC fix bringing Operator::ReturnCallIndirect cost from 60 → 6 (10× cut), affecting the proposed return_call_indirect follow-up but not this PR's direct-call path.

🤖 Generated with Claude Code

[github-actions]

Comparing from f8ad7b0 to a3ff621:
The produced WebAssembly code seems to be completely unchanged.
In terms of gas, no changes are observed in 5 tests.
In terms of size, no changes are observed in 5 tests.

[ggreif]

@crusso — pinging you on the tail-call slice. Self-contained and opt-in behind --experimental-tailcalls; the PR description and .claude/plans/wasm-exts-update.md carry the full design. Two empirical wins on the IC instruction counter under the flag:

bench	shape	flag-off → flag-on	delta
hutton	mutually-tail-recursive Hutton/Bahr stack VM, fak(10) × 1k	26.05M → 25.42M	−2.5%
gauss	self-tail-recursive foldLeft (+) 0 [1..100] × 10k	110.52M → 101.40M	−8.2%

So return_call beats both ordinary call (mutual case) and the existing self-tail → loop rewrite (gauss case) on cycles, contradicting the intuition that "loop rewrite is strictly cheaper than return_call for self-tail."

Two concrete future directions worth noting alongside what's already in the plan (§ Source-level annotation (with tailcall) and § return_call_indirect for computed tail-calls):

1. return_call_indirect for tail-position invocation of dynamic closures. Today the codegen specialises only the direct call arm (SR.Const Const.Fun (..., mk_fi, _) → ReturnCall <fi>). Closure-call sites (_, Type.Local via Closure.call_closure) silently degrade to non-tail call_indirect even when the IR carries TailCallPrim. Extending Closure.call_closure to take an is_tail parameter and conditionally emit return_call_indirect would close that gap — the wasm-exts AST/codec already grew the variant, only the lowering is missing. Now also unblocked on the IC side: fix: instructions for Operator::ReturnCallIndirect dfinity/ic#10086 just merged, cutting Operator::ReturnCallIndirect cost 60 → 6 (10× reduction), so once a pocket-ic release picks it up, indirect tail-calls measure at parity with ordinary call_indirect.

2. Destination-passing style for TCMC (Tail Call Modulo Cons). Functions like map f (x :: xs) = f x :: map f xs are not in tail position under standard TCO (the recursive call sits inside a cons constructor), and currently stack-overflow on long lists. The OCaml-5 [@tail_mod_cons] / Koka / Lean trick: allocate the constructor ?(f(x), <hole>) eagerly on the heap, pass the address of <hole> to the recursive call as a destination argument, and have the recursive call write its result into the destination slot before tail-calling itself. The recursive call thereby becomes tail (modulo a heap write), and return_call can apply uniformly. Implementation would touch the IR (a pre-pass recognising tail-mod-cons patterns), the calling convention (extra destination operand), and the allocator (exposing unfilled slots through the write barrier). Complementary to (1): (1) handles "tail call to a computed callee," (2) handles "tail call to self with a constructor in the way."

Together with the (with tailcall) annotation idea from the plan, the long-term shape is: explicit user intent → producer pass classifies every recursive call site (direct, indirect, or modulo-cons) → uniform return_call / return_call_indirect lowering → bounded stack everywhere, default-on once IC support is universal.

Happy to chase any of these as separate follow-up PRs if there's interest.

[crusso]

Description all sounds sensible but haven't looked at code at all. Impressive!

[ggreif]

Description all sounds sensible but haven't looked at code at all. Impressive!

The code is surprisingly straightforward too!

[ggreif]

As soon as the cycle counts are stabilised and the tail-call instructions are consistently cheaper than the stack-eaters, we'll flip the default for --experimental-tailcalls. Then we'll remove the flag from the algorithm (and with it the loop emulation) and reuse the flag for TCMC and other clever stuff.

[ggreif]

When the return_call* is cheaper (and better) in all regards, (with tailcall) won't make sense.

[ggreif]

It might make sense to understand it as a directive: error out if the tail-call is not realisable by the compiler mid-end transform.

[ggreif]

Do we want to document the flag to the user?

[ggreif]

Update: extending TCO to indirect calls + linker fix

Spotted a real gap while drafting a follow-up bench: the original PR's TCO covered direct call only — closure-dispatched calls (anything reached via call_indirect) silently degraded to non-tail dispatch. Mutually tail-recursive async* Bool chains stack-overflowed at ~1k hops because every await* goes through the closure dispatch path (the lambda async-lowering produces from [v_ret; v_fail; v_clean] -->* call user_body krb).

Fixed in commit 113d56dc9 plus plan housekeeping in c3fb49ed3. Three coupled changes:

1. Closure.call_closure (both backends) gains an optional ~is_tail parameter and emits ReturnCallIndirect when set.
2. The _, Type.Local arm in compile_*.ml (the closure-call producer in CallPrim _ | TailCallPrim _ as cp) threads is_tail through.
3. The linker. linkModule.ml's rename_types pass renumbers type-table indices when merging RTS + user module — and its instruction matcher handled CallIndirect but not ReturnCallIndirect, so the new tail-call sites kept stale pre-merge type indices and the binary failed wasmtime's validator with bogus type signatures (e.g. (func (result i64)) on a 4-i64-no-result call site). One-line fix.

New mccarthy bench

Added a third row to test/bench/tailcall.mo: mutually tail-recursive isEven/isOdd in async* Bool, chained via await*.

Bench	Mode	Cycles
hutton	direct call → return_call	25,416,088
gauss	self-recursive foldLeft → return_call	101,400,270
mccarthy	indirect call_indirect → return_call_indirect (new)	54,700,107

100k mutual hops, no stack growth. ~547 cycles per hop.

tailcall.valid.{ok,ret.ok} quirk

wabt's wasm-validate rejects return_call_indirect with an i64 table-index under table64 even though it accepts the same i64 index for plain call_indirect. wasmtime treats them symmetrically and accepts our binary; the canister runs end-to-end. The new valid.ok files capture wabt's error messages so the test harness stays honest about that toolchain asymmetry — if wabt closes the gap upstream we'll re-accept.

[ggreif]

Filed the upstream fix for the wabt return_call_indirect + table64 asymmetry referenced in 113d56dc9: WebAssembly/wabt#2744. Three-line patch + a test/typecheck/ regression.

Once that lands and we bump our pinned wabt, the test/bench/ok/tailcall.valid.{ok,ret.ok} files added in this PR can be removed (or re-accepted to a clean validate), since they only exist to document wabt's stricter-than-wasmtime reading of return_call_indirect under table64.