route switched I/O through completion-owned UDP (#4814) by mariusae · Pull Request #4814 · meta-pytorch/monarch

mariusae · 2026-09-04T15:17:37Z

Summary:

add RoutedUdpPacketIo so quiche, CID routing, pacing, GSO/GRO, and io_uring waits share one endpoint owner
route non-local CID aggregates directly between completion-owned UDP carriers while retaining a Unix datagram fallback for task-local leaves
remove the threaded CompletionUdpSocket adapter and preserve route admission through actual submission
use the roofline-validated eight-segment switched GSO frontier and wait for response FIN before benchmark teardown

Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba

Walkthrough document: https://mdoc.internalmeta.com/doc/ef7df54e-b047-4e9a-96b5-6055b2a411da

Reviewed By: shayne-fletcher

Differential Revision: D117242784

meta-codesync · 2026-09-04T15:18:30Z

@mariusae has exported this pull request. If you are a Meta employee, you can view the originating Diff in D117242784.

Summary: Pull Request resolved: meta-pytorch#4814 - add `RoutedUdpPacketIo` so quiche, CID routing, pacing, GSO/GRO, and io_uring waits share one endpoint owner - route non-local CID aggregates directly between completion-owned UDP carriers while retaining a Unix datagram fallback for task-local leaves - remove the threaded `CompletionUdpSocket` adapter and preserve route admission through actual submission - use the roofline-validated eight-segment switched GSO frontier and wait for response FIN before benchmark teardown Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/ef7df54e-b047-4e9a-96b5-6055b2a411da Reviewed By: shayne-fletcher Differential Revision: D117242784

Summary: Introduce the dependency-free wire types at the base of Chrysalis and document the architecture and review sequence. PIDs name process incarnations, while fixed-width QUIC connection IDs embed the destination PID and a process-local demultiplexing suffix. Walkthrough: Start in `chrysalis-core`, where `Pid` defines process identity and `ConnectionId` defines the routable QUIC wire layout. Then read `chrysalis/DESIGN.md` for the complete model and `chrysalis/WALKTHROUGH.md` for the order in which the stack assembles it. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/58f98269-78c5-42b3-a1a4-6e9a7a1adaf2 Differential Revision: D117242759

Summary: Add the carrier layer that moves atomic datagrams without interpreting QUIC. A common socket contract supports UDP, Unix datagrams, and in-process channels, while `DatagramSocketSet` selects a carrier from an opaque destination address. Walkthrough: Read `datagram.rs` for the transport-independent address, receive, and transmit contracts. Then follow the concrete implementations in `udp.rs`, `unix.rs`, and `inproc.rs`; `socket_set.rs` composes those carriers behind one socket and `shutdown.rs` provides structured teardown. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/aebf6303-aaad-470c-ae40-e0a0f60550bf Differential Revision: D117242760

Summary: Route datagrams by the destination PID embedded in each QUIC connection ID. Routes reference lifecycle gates, so expiring one gate fences every derived destination without synchronously deleting a subtree of forwarding entries. Walkthrough: Begin with `route.rs`: `RouteGate` supplies the shared fence, `Route` binds a destination to a next hop and gate, and `Router` performs forwarding. `switch.rs` then demultiplexes local PIDs and delegates every non-local datagram to that router without inspecting QUIC streams. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/15fc8742-10cf-4dbd-af8d-70a218e2d8c7 Differential Revision: D117242761

Summary: Add the ownership and completion model shared by transport drivers and runtime adapters. Bounded ingress, completion credits, ordered submissions, operation IDs, and explicit wakeups make resource retention and backpressure part of the API without depending on Tokio or a QUIC implementation. Walkthrough: `command.rs` and `submission.rs` define accepted work and ownership transfer. `completion.rs` defines every resource-return and lifecycle event. `credit.rs`, `queue.rs`, and `ingress.rs` enforce bounded admission, while `id.rs` and `notify.rs` provide driver-scoped identity and runtime-neutral wakeups. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/a39624e5-45e3-4696-8d96-77c9e2f95eb6 Differential Revision: D117242762

Summary: Add a runtime-neutral UDP engine that owns stable packet slots on a dedicated driver thread. The engine submits batched receive and transmit operations through io_uring and exposes the packet-I/O boundary needed by the QUIC state machine. Walkthrough: `config.rs` defines bounded ring and buffer sizing. `socket.rs` transfers a bound UDP socket into the engine. `driver.rs` owns the ring, receive and transmit slot lifecycles, GSO submissions, GRO splitting, completion processing, and structured shutdown. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/96ad08dc-3b3a-4fb4-9c4f-59d5a9415577 Differential Revision: D117242763

Summary: Define the final identity, admission, buffer-ownership, command, and packet-I/O boundaries used by the runtime-neutral quiche endpoint. The crate binds quiche to Chrysalis CIDs and completion transport without introducing an async runtime. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/6ae92406-f8be-46ae-8b73-6d4b869276c2 Differential Revision: D117242765

Summary: Implement the final runtime-neutral quiche endpoint over bounded completion transport and packet slots. The driver authenticates certificate-derived PIDs, routes fixed CIDs directly, bounds server admission and retained completions, schedules only runnable streams, preserves directional operation order, returns ownership on cancellation, and reclaims terminal state. When a client server name is an IP literal, create the quiche connection without DNS SNI and call `Connection::set_host_ip_addr` before the first packet. This makes verified connections match the certificate `iPAddress` SAN. DNS names retain the existing SNI and DNS SAN path. Exercise the IP-literal path with a verified end-to-end IPv6 handshake. Walkthrough: `driver.rs` owns endpoint admission, completion credits, shutdown, packet-loop orchestration, and the lifecycle overview. `driver/network.rs` owns CID routing, Retry, connection admission, packet generation, and lifecycle statistics. `driver/connection.rs` owns authenticated connection state plus ordered send and receive queues, runnable-stream scheduling, cancellation, half-state transitions, and reclamation. `driver/tests.rs` exercises handshake, multiplexing, backpressure, shutdown, authentication, cancellation, and stream control. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/c932f625-1719-4ce0-9502-6074ffcb7f68 Reviewed By: shayne-fletcher Differential Revision: D117242766

Summary: Translate the completion-driven Chrysalis transport into ergonomic Tokio connections and streams without moving protocol ownership into the runtime. The adapter preserves bounded admission, caller buffer ownership, operation cancellation, and structured driver shutdown. Name every transport constructor `spawn_client`, `spawn_server`, `spawn_duplex`, or `spawn_duplex_routed` so the API makes creation of the driver thread and completion pump explicit. Walkthrough: `Transport::spawn_*` constructs the low-level endpoint, starts its dedicated driver thread, and starts the Tokio completion pump. Connection and stream handles translate command receipts and operation completions into Tokio futures while cancellation remains an explicit message to the runtime-neutral driver. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1c2c7a60-06fb-493d-9ce2-96f18e099b75 Differential Revision: D117242767

Summary: Compose routed datagram sockets, direct io_uring UDP, pooled QUIC connections, and link-local stream protocols behind one carrier-neutral transport. Application and control streams share physical carriers while retaining distinct process routing identities. Document the synthetic `SocketAddr` projection that lets unmodified quiche identify paths over opaque carrier addresses, including its stable bidirectional mapping and the fork that removing it would otherwise require. Walkthrough: `packet_io.rs` adapts switched datagrams to the quiche packet-slot contract. `quic_transport.rs` owns endpoint tasks, connection pooling, dialing, and accepted streams across routed and direct UDP carriers. `link_local.rs` multiplexes registered control protocols over reserved streams, and `lib.rs` exposes the composed transport surface. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/c70ac010-2206-4e75-b662-cee3411ff5a8 Differential Revision: D117242768

Summary: Define the versioned delegated-namespace messages and a bounded deterministic frame codec. The protocol covers publication snapshots and deltas, cumulative acknowledgements, resolution, enumeration, rejection, and link-scoped identity. Walkthrough: `protocol.rs` defines typed identifiers, locators, process entries, consistency modes, and every message variant. `codec.rs` assigns the wire tags, enforces frame and collection bounds, and encodes or decodes each field without depending on session or network tasks. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1d9e8e3d-57d2-443e-b9ae-0bc87a88ff30 Differential Revision: D117242769

Summary: Implement the authoritative delegated namespace and parent-scoped resolver cache as deterministic state machines. Publication snapshots, contiguous deltas, ownership, revisions, replay, and negative-cache expiry can now be tested without transport timing. Walkthrough: `state.rs` validates commands, stages snapshots, applies atomic directory changes, assigns revisions, and emits cumulative link responses. `cache.rs` accepts ordered resolution updates from one authority and retains positive or expiring negative results without extending replayed deadlines. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/f890ab4b-ac9f-423a-bb41-5e818abc6dd2 Differential Revision: D117242770

Summary: Connect the deterministic nameserver protocol to ordered framed streams. Parent sessions negotiate versions, preserve request ordering, translate committed state effects into acknowledgements, correlate resolution and enumeration responses, and bind learned state to one link lifetime. Walkthrough: `stream.rs` performs bounded asynchronous frame reads and writes over Chrysalis streams. `session.rs` owns handshake state, pending request correlation, committed-command callbacks, publication responses, enumeration, and terminal protocol errors without owning socket tasks. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/9a1ade49-3a74-4c22-825f-62e8cb812987 Differential Revision: D117242771

Summary: Export one complete child process subtree through a link-scoped publication session. The child sends chunked snapshots followed by contiguous deltas, advances only through cumulative acknowledgements, and resnapshots when the parent detects a gap. Walkthrough: `ChildSession` owns publication sequence, snapshot incarnation, chunking, pending changes, and request correlation. Its event interface separates deterministic protocol progress from the task that carries messages and supplies the current subtree. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/60c62871-5eb2-48da-aa2c-3734a5d78ae2 Differential Revision: D117242772

Summary: Compose deterministic namespace state with framed parent and child tasks, gated route projection, cached resolution, paginated enumeration, and supervised parent reconnection. Expiring one link gate immediately fences every route learned from that subtree while asynchronous cleanup proceeds. Walkthrough: `service.rs` serializes authoritative commands and queries. `routes.rs` projects directory changes into shared-gate router entries. `link.rs` drives one parent publication and resolution session, `server.rs` accepts child links, `upstream.rs` presents the active parent, and `manager.rs` supervises configuration, reconnect, status, resolution, and enumeration. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/650999f8-d5fd-4927-93bb-0f22b18c437a Differential Revision: D117242773

Summary: Present identity, carriers, transport, nameserver state, parent linkage, and registered link-local protocols through one embeddable `Node`. The facade owns structured startup and shutdown while exposing a small serve, join, dial, accept, enumerate, and protocol API. Walkthrough: `node.rs` composes transport and namespace services and supervises their lifetimes. `link_protocol.rs` registers singleton link-local handlers and starts them for every live link. `socket.rs` adapts accepted Chrysalis streams to Tokio I/O, while `lib.rs` exposes the public embedding surface. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/94be4b47-26d5-4988-a16b-3c91a0d8a068 Differential Revision: D117242774

Summary: Obtain Meta mTLS material and issue a distinct leaf certificate for each Chrysalis process. Certificate-derived PIDs remain self-certifying while the core transport and Node crates stay independent of Meta identity infrastructure. Walkthrough: The provider loads the host identity chain, creates process-specific key material, signs a constrained leaf certificate, validates the resulting chain, and returns the certificate, key, and derived endpoint identity consumed by Chrysalis. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/d06242da-b4c6-40e8-9ef1-71a3232d5884 Differential Revision: D117242775

Summary: Discover replicated table schemas, initialize peer metadata, stream bounded cr-sqlite change batches, and acknowledge only durable application. Per-origin frontiers retain enough state to restart without skipping unacknowledged changes or replaying acknowledged work. Walkthrough: `lib.rs` owns schema hashing, site scopes, persistent origin/frontier metadata, transactional change reads, and durable application. `protocol.rs` frames schema, site-set, change, acknowledgement, and error messages. `replica.rs` drives one bidirectional stream and coordinates publisher and receiver progress. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/238b8581-9ba5-437c-8aee-503392f793e0 Differential Revision: D117242778

Summary: Run bidirectional SQLite synchronization on every adjacent Chrysalis link. Durable per-origin frontiers and incremental origin discovery prevent loops and preserve progress when subtree membership changes without assuming that relayed sites share one scalar database version. Walkthrough: `topology.rs` registers the reserved SQLite link protocol with `Node`, advertises each child subtree explicitly, advertises the complement toward children, and republishes aggregate membership upstream. The README follows scope changes, durable frontiers, origin discovery, schema propagation, and operational usage. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/8625eb6f-b458-4863-b259-db69d2a218e0 Differential Revision: D117242779

Summary: Resolve direct addresses and `mast://` names into join settings, then expose serving, process enumeration, stream piping, and replicated SQLite operations through the public Node API. The production wrapper packages the hermetic cr-sqlite extension for the in-process shell. Walkthrough: `chrysalis-resolver` parses deployment references and resolves MAST tasks. In `chrysalis-cli`, `address.rs` normalizes carrier and join syntax, `identity.rs` selects identity providers, `socket.rs` bridges streams, `sqlite_shell.rs` owns embedded database commands, and `main.rs` assembles the command surface. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/7ace3758-d098-4a0c-9837-e49b902c9cee Differential Revision: D117242780

Summary: Launch many real Chrysalis processes in flat or task-head topologies, record lineage, wait for namespace convergence, exercise fresh streams, and run direct network baselines. Only task heads require UDP; local leaves use Unix datagrams under one supervisor. Walkthrough: `benchmark.rs` owns process topology, worker supervision, convergence, echo, throughput, and experiment execution. `persist.rs` defines replicated experiment and node records. `network_baseline.rs` isolates carrier and QUIC measurements, `mast.rs` builds MAST launch specifications, and `main.rs` exposes run, persist, baseline, and worker commands. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/4e601137-b930-44b0-914f-6ca3b85705b0 Differential Revision: D117242781

Summary: Let any process submit, inspect, and target persistent scale experiments through replicated SQLite state. Nodes claim work addressed to their PID, run echo or streaming-transfer measurements after warmup, and publish status, timing, and failure details for remote inspection. Walkthrough: `admin.rs` implements experiment creation, targeted source/destination selection, node listing, and result inspection over a remote Chrysalis SQLite endpoint. `main.rs` exposes those commands alongside the runner, and the README walks through MAST launch, topology, experiment submission, bandwidth tests, and result collection. Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/1dbe7d3b-7aab-48ad-be8b-01bd98b9e9a7 Differential Revision: D117242783

Summary: Pull Request resolved: meta-pytorch#4814 - add `RoutedUdpPacketIo` so quiche, CID routing, pacing, GSO/GRO, and io_uring waits share one endpoint owner - route non-local CID aggregates directly between completion-owned UDP carriers while retaining a Unix datagram fallback for task-local leaves - remove the threaded `CompletionUdpSocket` adapter and preserve route admission through actual submission - use the roofline-validated eight-segment switched GSO frontier and wait for response FIN before benchmark teardown Design document: https://mdoc.internalmeta.com/doc/e6918c35-b3d1-4118-ac39-6308b0cb55ba Walkthrough document: https://mdoc.internalmeta.com/doc/ef7df54e-b047-4e9a-96b5-6055b2a411da Reviewed By: shayne-fletcher Differential Revision: D117242784

meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 4, 2026

meta-codesync Bot added the meta-exported label Sep 4, 2026

meta-codesync Bot changed the title ~~route switched I/O through completion-owned UDP~~ route switched I/O through completion-owned UDP (#4814) Sep 4, 2026

mariusae force-pushed the export-D117242784 branch from 9900d38 to 35bdd91 Compare September 4, 2026 16:03

mariusae force-pushed the export-D117242784 branch from 35bdd91 to 99b7f27 Compare September 4, 2026 16:08

mariusae force-pushed the export-D117242784 branch from 99b7f27 to d09f7bb Compare September 4, 2026 16:13

mariusae force-pushed the export-D117242784 branch from d09f7bb to 59ba883 Compare September 4, 2026 16:39

mariusae force-pushed the export-D117242784 branch from 59ba883 to 10bce18 Compare September 4, 2026 16:45

mariusae force-pushed the export-D117242784 branch from 10bce18 to f3143f3 Compare September 4, 2026 16:59

mariusae force-pushed the export-D117242784 branch from f3143f3 to 001f63d Compare September 4, 2026 17:14

mariusae force-pushed the export-D117242784 branch from 001f63d to b9a537d Compare September 4, 2026 17:18

mariusae force-pushed the export-D117242784 branch from b9a537d to 970e945 Compare September 4, 2026 17:35

mariusae force-pushed the export-D117242784 branch from 970e945 to 599ec75 Compare September 4, 2026 17:40

mariusae force-pushed the export-D117242784 branch from 599ec75 to aaf3c80 Compare September 4, 2026 17:52

mariusae force-pushed the export-D117242784 branch from aaf3c80 to 3bccbf7 Compare September 4, 2026 17:57

mariusae force-pushed the export-D117242784 branch from 3bccbf7 to c9e9305 Compare September 4, 2026 19:55

mariusae force-pushed the export-D117242784 branch from 3deff0f to d4339d8 Compare September 10, 2026 21:13

mariusae force-pushed the export-D117242784 branch from d4339d8 to 1c73555 Compare September 10, 2026 21:21

mariusae force-pushed the export-D117242784 branch from 1c73555 to 2c1182e Compare September 10, 2026 22:07

mariusae force-pushed the export-D117242784 branch from 2c1182e to 9591c1e Compare September 10, 2026 22:15

mariusae added 22 commits September 10, 2026 19:39

Navigation Menu

Uh oh!

route switched I/O through completion-owned UDP (#4814) - #4814

route switched I/O through completion-owned UDP (#4814)#4814
mariusae wants to merge 22 commits into
meta-pytorch:mainfrom
mariusae:export-D117242784

mariusae commented Sep 4, 2026 •

edited by meta-codesync Bot

Loading

Uh oh!

meta-codesync Bot commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

mariusae commented Sep 4, 2026 • edited by meta-codesync Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

meta-codesync Bot commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mariusae commented Sep 4, 2026 •

edited by meta-codesync Bot

Loading