quic: route a migrated client to the reactor that owns its connection - #210
Merged
Conversation
…honour it The receive/send buffer asked for on every UDP socket was a private const of 8 MiB, with a comment noting the kernel clamps it to net.core.rmem_max and calling the result "best-effort headroom". The comment was right and the consequence was invisible: on a stock Linux box that ceiling is 212,992 bytes, so the 8 MiB request is granted at about a fortieth of its size, and the only symptom is datagrams dropped under load - which reads as a bug in whatever is running on top. Measured here, unmodified: 4.2% of datagrams dropped to rcvbuf overflow on the h3 benchmark at saturation. setsockopt does not fail in this case, it clamps and reports success, so nothing said so. Two changes. SocketBufferBytes becomes a UdpOptions knob (default unchanged at 8 MiB), because an operator who raises the ceiling has no way to tell ioxide to use it. And the granted size is read back with getsockopt and reported once per process when it falls well short - once, not per reactor per port, since the clamp is a property of the machine. The message deliberately stops at the fact and does not tell anyone to raise the cap. Granting the full 8 MiB on this machine cost about 45% of h3 throughput at saturation on unmodified main: the drops stopped, a deep standing queue took their place, and peers timed out and retransmitted instead - 1.2 datagrams per request became 5.2. A shallow buffer drops early, and early drops are the signal congestion control is built to read. Which behaviour is wanted depends on the deployment, so this reports and leaves the judgement. A zero or negative request is refused rather than clamped, because it would leave the socket on the kernel minimum and look identical to the clamp it is trying to make visible.
…s connection (#205) Fixes #205. Every reactor binds the QUIC port with SO_REUSEPORT and the kernel chooses between them by hashing the sender's address. That is the right answer only while the address holds still, so a NAT rebind or a client changing network re-hashes to a reactor that has never heard of the connection, whose short-header packets it then drops. #209 taught the transport to migrate; this is what lets the packets reach the connection that could act on it. It bites the default configuration, since ReactorCount defaults to one per core. The connection cannot move to meet the packet. The ngtcp2 conn, the picotls session, the open streams and their ring-bound buffers are owned by one reactor thread and documented reactor-thread- only throughout; moving live state to whichever reactor a datagram landed on is precisely what shared-nothing forbids. So the datagram moves instead, which is ordinary message passing and rides ScheduleOnReactor - already public, already used for exactly this. Every connection id the server mints now carries its owning reactor in the first byte, chosen so cid[0] % ReactorCount is that reactor while the rest stays random (iq_stamp_shard). A reactor that receives a short header for an id it does not have reads that byte, copies the datagram and posts it to the owner. The copy is the point, not an inefficiency: the payload lives in the receiving reactor's io_uring provided-buffer ring, which is returned as soon as dispatch ends, so handing the owner a pointer into it would be a use-after-free under load. What crosses a thread is bytes, never reactor state. Only short headers are forwarded. A short header means the handshake finished, so the id is one this server minted and its first byte really does name the owner. A long header carries an id the CLIENT chose, and routing on a byte the peer controls would let anyone aim traffic at a reactor of their choosing. QuicOptions.Routing offers the alternative. KernelFilter additionally attaches a classic-BPF program to the reuseport group so the kernel routes by connection id directly. It is not the default, and the measurements say why - h3 benchmark, two reactors, one machine: Forward nothing at all until a client moves, then ~8.5us per datagram KernelFilter free with CPU headroom, ~-12% throughput at saturation, nothing for migrated The forward cost is a cross-thread wake, not work: CPU per request moved 2.44 -> 2.46us with every datagram forwarded, while throughput halved at fixed concurrency, and the reactors sat 36% idle. So Forward charges only the connections that actually migrate; KernelFilter charges every packet a little kernel work, invisible until there is no headroom left. Unless a large share of clients migrate, Forward is cheaper in aggregate. KernelFilter needs reactors to open their UDP sockets in ShardIndex order, since the program answers with a position in the reuseport group and that position is bind order. That rendezvous exists only under KernelFilter; the default leaves startup untouched. It also degrades rather than fails: if the kernel refuses the program, ioxide says so and forwarding stays underneath. Correctness never depends on the filter, only cost does. Both modes are tested and each discriminates in the opposite direction - Forward asserts datagrams were forwarded, KernelFilter asserts none were and that the program actually attached, so it cannot pass vacuously as Forward under another name. Confirmed both ways: suppressing the forward makes the Forward test fail. StartQuicSharded is new because every other QUIC entry point in the harness pins ReactorCount = 1, where a datagram has nowhere wrong to land. QuicForwardsSent/Received/Dropped and QuicStaleDatagrams are exposed for operators. The distinction matters and only the shard byte makes it possible: a short header for an unknown id that belongs elsewhere is a routing event, while one addressed here is ordinary - a migration retires ids and packets in flight still carry them. Also here, found on the way and not separable from the shim rebuild: iq_sync_path was calling ngtcp2_sockaddr_eq, which lives in lib/ngtcp2_addr.h and is NOT shipped under lib/includes. It compiled only by implicit declaration and stops building outright on GCC 14+, where that is an error. Replaced with a local mirror over the public types; a plain memcmp will not do, since sockaddr padding and flowinfo would read as an address change and fire migration callbacks on a connection that never moved. iq_abi is a new exported surface revision the managed side checks when it builds an engine, so a stale libioxide_ngtcp2.so fails at startup with a clear message instead of passing garbage across a boundary that just grew two parameters. Stale .so files have silently invalidated audit runs here before.
The learn section had QUIC & HTTP/3, which walks a datagram through the transport and is written for someone reading the code. What it does not answer is the set of questions a deployment actually has: which certificate is served for which name, how one gets renewed without dropping traffic, how client certificates are checked, what happens when a client's address changes, and what a fleet of reactors does about it. Those were spread across doc comments and samples or nowhere at all. /how-ioxide-does-h3 collects them. It covers the three layers and what each owns, why TLS on QUIC is a separate stack from TLS on TCP with its own configuration (the thing most likely to catch someone running both), SNI and why the host table closes at CreateFactory, ReplaceCertificates and its three surprising properties, mutual TLS and why PeerCommonName exists rather than substring-matching the subject, connection migration, and the routing modes added in #205 with the measurements behind the default. It links to QUIC & HTTP/3 rather than repeating it: this page is what the server does, that one is how the code does it. quic-h3.html gains the same routing material in short form, because its ingress walkthrough said unknown short-header packets are dropped as stale traffic - true when it was written, and no longer true for a fleet, where such a packet may belong to a sibling reactor. The full comparison lives on the new page so the two cannot drift.
… the panes Every sample carries its full knob set at the shipping default, so a reader can see what is configurable without going looking. QuicOptions.Routing was missing from all twelve QUIC and HTTP/3 samples. Set explicitly to QuicRouting.Forward - the default - with what the choice actually means: Forward costs nothing until a client changes address, KernelFilter has the kernel route by connection id instead and costs a little on every packet. The comment points at /how-ioxide-does-h3 rather than restating the measurements, so there is one place for them to be wrong. Site panes regenerated from the samples, since they are generated rather than written.
H3TestClient built a fresh HTTP/3 session inside Request() - ih3_client_new plus a new control, QPACK encoder and QPACK decoder stream - on every call. HTTP/3 permits exactly one control stream per peer, and RFC 9114 6.2.1 requires a second one to be treated as a connection error of type H3_STREAM_CREATION_ERROR. So from the second request onward this client was speaking invalid HTTP/3. The server was right and said so. nghttp3 returned H3_STREAM_CREATION_ERROR from read_stream on the duplicate stream (type 0x00 carrying SETTINGS, and later a duplicate QPACK decoder), the run loop set _protocolFailed, exited, and closed the connection - about a millisecond after the first response. Every nghttp3 run loop did it: buffered sync, buffered async, streaming, and streamed response. The pure-C# stack did not, because it never saw a second control stream: it only ever ran one request per connection in these tests. None of that was visible, for the worst possible reason. The requests already in flight kept being answered off state the transport had already dismantled - the CIDs unregistered, the connection out of _quicConnSet, PeerAddr freed and zeroed, the transport's reference dropped - so three more requests returned 200 apiece and every test stayed green. A test client that provokes a connection error and then passes anyway is worse than one that fails. Found while investigating why a migrated connection's pinned socket died instantly: the pin is released in QuicRemoveConnection, which this was triggering after the first response. Fixing this is a prerequisite for that work, and for any test that means to exercise more than one request over one connection. The session now stands up once per connection; only the bidi request stream and the response state are per request. 431 pass across E2E, Unit, Chaos, Http, Tls and File. Connections now live to the end of their test and exit clean (isClosed=True, protocolFailed=False) rather than on a protocol failure.
H3TestClient never called iq_conn_handle_expiry. The shim has exported it, and iq_conn_expiry beside it, since the engine binding landed - the client simply never fired either. So it had no loss recovery whatsoever: a dropped datagram left its packet unacked forever, nothing retransmitted, the congestion window filled, and writev_stream began answering 0 with nothing consumed. Both ends then sat silent until the 10 s write deadline called it a stall. The diagnosis took a while because the symptom accuses the server. What settled it was that the client's own sent and received datagram counts were FROZEN across every spin - it was neither sending nor receiving, so it was not blocked on anything the peer had done. Flow control looks different and recovers: the streaming-upload test shows n=-208 (STREAM_DATA_BLOCKED) and comes back. Here it was n=0 with consumed=-1, which is ngtcp2 saying it has nothing to send at all. Nothing here ever lost a packet before, which is why an omission this size survived: every connection served one request over loopback and was gone. Once connections started living across several requests, and an address change started discarding datagrams in flight to the forwarder's old socket, the gap became a hang - and it presented as a server that stopped answering after a migration, which is exactly the bug it is not. Timers fire from PumpIn, which every wait loop already drives. Three consecutive full E2E runs green; the migration tests had been failing roughly two runs in three before this.
…d to mean The mtls test "a client certificate that is not valid yet is refused" failed today, reporting that the server had served a not-yet-valid certificate. It had not. The fixture had gone stale. Spec fixtures are cached by their spec rather than by freshness, deliberately, because most of them are supposed to be invalid and re-minting an expired one every run would defeat the test. That is safe in one direction only. An expired certificate stays expired forever; one minted to be NOT YET VALID becomes valid the moment its notBefore arrives. This fixture is minted with notBefore = now + 1 day, so the copy left in /tmp yesterday became a perfectly good certificate overnight, the server accepted it exactly as it should, and the test called that a product bug. A test that fails once a day has passed is a bug in the test, and one that accuses the product of a security defect is worse than most. The cached file is now reused only while its actual validity state still matches the intent - not yet started if the spec asked for that, expired if it asked for that. Confirmed both ways: planting an already-valid certificate in the not-yet-valid fixture's place makes the cache re-mint it (CN=alice back to CN=future-alice, notBefore tomorrow), where before it was served as-is and the test failed. Unrelated to the QUIC work in flight; found because the date rolled over mid-session.
…r one hop Cross-reactor forwarding (#205) is correct but permanent. The kernel picks a socket by hashing the sender's address, and after a migration that address does not change back - so every one of that client's datagrams keeps landing on the wrong reactor and keeps paying a cross-thread hop for the life of the connection. A datagram socket bound to the same address as the others but connect()ed to one peer is a MORE SPECIFIC match than a wildcard bind, and the kernel's lookup takes the narrowest match before it ever reaches the reuseport hash. So the owning reactor opens one toward the peer's new address and the datagrams arrive there directly. Three things about that, each measured rather than assumed. connect() on a datagram socket puts nothing on the wire - it is a local declaration, and the peer is never told. Adding one does not re-scatter anybody else: eight established peers, none moved. And it cannot bootstrap itself, since the owning reactor only learns the new address from a datagram and the datagrams are going elsewhere - forwarding has to deliver the first one. The two are complements: forwarding makes the claim possible, the claim stops forwarding being forever. The claim is made from the reactor's sweep, NOT from the engine's path-change report, and that distinction is the whole difference between working and thrashing. ngtcp2 reports a path many times while it validates one, alternating between the old address for data and the new one for PATH_CHALLENGE probes. Claiming on each report tore down a working socket and built another, with datagrams already queued on the one it closed - losing packets in order to avoid a hop, which is the wrong way round. Measured: five claims for a single address change, flip-flopping between two ports. From the sweep it is one claim per address, because by then the path has settled. Only connections that actually moved are claimed. The sweep visits every connection every 250 ms, so a guard that asked merely "is this address claimed yet" would spend a descriptor and an armed receive on every connection on the server to change nothing - they are already being delivered here by the hash. PeerAddressMoved is the signal, set where the address is adopted. Bounded and best-effort throughout. A ceiling of 512 concurrent claims, because ngtcp2 adopts a new path BEFORE it finishes validating it, so a forged datagram can reach here; past the ceiling the connection simply keeps forwarding. A bind that fails is swallowed for the same reason - this runs under a callback whose exceptions fault the connection, and faulting a working connection to skip an optimisation is a bad trade. Ignored entirely under KernelFilter, where the kernel already routes by connection id. QuicPinsCreated and QuicPinsOpen expose it. Tests cover both directions: that the forwarding stops after a claim, that one address change produces one claim rather than one per path report, and that a fleet whose clients never move claims nothing and forwards nothing. Also here, because the migration tests could not be trusted until it was proven: both h3 stacks now assert that the SAME connection object serves before and after the address changes, and that the factory ran exactly once. Status codes cannot tell migration from a client quietly re-handshaking, and a reconnect would take the h3 session, the QPACK tables and any application state with it while the tests stayed green. Confirmed both ways - asserting a DIFFERENT connection makes both fail. 436 tests pass; E2E stable across three consecutive runs. Benchmarks against main: -0.4% Tls/OpenSsl, -0.3% Nghttp3Response, -1.1% ManagedBuffered, -0.1% ManagedStreamedBoth, and Nghttp3Buffered repeats at +2.5%/+1.9% against a baseline that measures +2.1% against itself - so no regression on either h3 stack, buffered or streamed.
Roughly half the prose added across #205 and the claim work said the same thing twice, or explained the investigation rather than the code. Kept: why the packet moves instead of the connection, why the copy is not optional, why the claim runs from the sweep, and the classic-BPF listing, which is unreadable without it. One comment was also wrong - it said the claim runs inside ngtcp2's path-change callback, which stopped being true when it moved to the sweep. No behaviour change; 436 tests pass.
Two pages covering one subject, with the routing material duplicated across both. Now one: deployment first (layers, TLS on QUIC, SNI, rotation, mTLS, migration, routing, the counters), then the packet-level walk - ingress, read surface, egress, the engines, timers, and the invariants. learn/quic-h3.html is deleted and every sidebar and inbound link repointed. No broken links.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #205.
Every reactor binds the QUIC port with
SO_REUSEPORT, and the kernel picks between them by hashingthe sender's address. That is the right answer only while the address holds still. A NAT rebind or a
client changing network re-hashes to a reactor that has never heard of the connection, whose
short-header packets it drops - so the connection dies on the reactor that could still have served
it. #209 taught the transport to migrate; this is what lets the packets reach it. It bites the
default configuration, since
ReactorCountdefaults to one per core.The connection cannot move to meet the packet: the
ngtcp2_conn, the picotls session, the openstreams and their ring-bound buffers are owned by one reactor thread. So the datagram moves instead,
which is ordinary message passing over the queue reactors already expose.
Every server-minted connection id now carries its owner in the first byte, chosen so
cid[0] % ReactorCountis that reactor with the rest left random.QuicOptions.RoutingForward(default)KernelFilterForwardhas the receiving reactor copy the datagram and post it to the owner. The copy is thepoint: the payload lives in that reactor's provided-buffer ring, returned as soon as dispatch ends,
so passing a pointer would be a use-after-free under load. Only short headers are forwarded - a long
header's id is chosen by the client, and routing on a byte the peer controls would let anyone aim
traffic at a reactor of their choosing.
KernelFilterattaches a classic-BPF program to the reuseport group. It needs reactors to opentheir UDP sockets in shard order (the program answers with a position in the group, and that is bind
order) - a rendezvous that exists only under this mode. It degrades rather than fails: if the kernel
refuses the program, forwarding stays underneath. Correctness never depends on the filter, only cost
does.
The forward cost is a cross-thread wake, not work: with every datagram forwarded, CPU/request moved
2.44 → 2.46 µs while throughput halved at fixed concurrency and the reactors sat a third idle. So
Forwardcharges only connections that migrate;KernelFiltercharges every packet a little kernelwork. Unless a large share of clients migrate,
Forwardwins in aggregate.Tests
StartQuicShardedis new because every other QUIC entry point in the harness pinsReactorCount = 1, where a datagram has nowhere wrong to land. Each mode has a test thatdiscriminates in the opposite direction -
Forwardasserts datagrams were forwarded,KernelFilterasserts none were and that the program attached, so it cannot pass vacuously as
Forwardunderanother name. Confirmed both ways: suppressing the forward makes the
Forwardtest fail.431 pass, 0 fail. Benchmarks vs
main: +0.4% / -2.0% / +1.4% / -1.0% / +0.8% - all inside noise.Also here
A latent build break.
iq_sync_pathcalledngtcp2_sockaddr_eq, which lives inlib/ngtcp2_addr.hand is not shipped underlib/includes. It compiled only by implicitdeclaration and fails outright on GCC 14+. Replaced with a local mirror over the public types; a
plain
memcmpwill not do, since sockaddr padding and flowinfo would read as an address change andfire migration callbacks on a connection that never moved.
iq_abi, a exported-surface revision the managed side checks when building an engine, so a stalelibioxide_ngtcp2.sofails at startup with a clear message rather than passing garbage across aboundary that just grew two parameters. Stale
.sofiles have silently invalidated audit runs herebefore.
UdpOptions.SocketBufferBytes(default unchanged at 8 MiB) plus a read-back.SO_RCVBUFclampsto
net.core.rmem_maxrather than failing, so on a stock box the 8 MiB request is granted at ~1/39thand the only symptom is 4.2% of datagrams dropped under load. ioxide now reports the shortfall once.
It deliberately does not advise raising the cap: granting the full 8 MiB cost ~45% of h3
throughput at saturation on unmodified main, trading early drops for a deep standing queue. The
message states the fact and leaves the judgement.
Docs.
/how-ioxide-does-h3covers the whole surface - layers, TLS on QUIC being a separate stackfrom TLS on TCP, SNI,
ReplaceCertificates, mutual TLS, migration, and both routing modes.learn/quic-h3.htmlsaid unknown short-header packets are dropped as stale traffic, true whenwritten and no longer so.