quic: let a connection survive its peer changing address - #209
Merged
Conversation
Part of #204. A client that changed network - or, far more commonly, whose NAT recycled a UDP mapping while it was idle - was blackholed: the server kept answering the address the connection was accepted on, and the connection died at the idle sweep with no error to either side. That is the failure QUIC's connection id exists to prevent. The cause was one line hiding the input from the library that already solves this: (void)remote_sa; (void)remote_salen; /* milestone: no migration; path is fixed at accept */ ngtcp2 implements path validation in full. It was simply never told an address had changed, so it had nothing to validate and kept writing to the path it was given at accept. So this does not implement migration - it stops concealing it. iq_conn_read now builds the path from the address the datagram actually arrived on, and everything after that is ngtcp2's decision: PATH_CHALLENGE out, PATH_RESPONSE back, and only then is the new path adopted. Adoption is reported through a new on_path_change callback, fired only after ngtcp2 has VALIDATED the path - never merely because a datagram claimed a new address. That distinction is the whole security of it: adopting an unvalidated address turns this server into an amplification reflector for whoever spoofed it. QuicConnection.UpdatePeerAddress, public and documented for exactly this since it was written, finally has its first caller. QuicConnection.OnDatagram gained an address-carrying overload rather than a changed signature. It is virtual and forwards to the two-argument form, so every existing subclass - including the test doubles - keeps working untouched. The test is h2o's and nginx's: a UDP forwarder between client and server whose UPSTREAM socket is swapped mid-connection, so the server sees one connection id arrive from a new source port. Getting that test to mean anything took two goes, which is worth recording. The first version swapped lazily, on the next relayed datagram, and the request under test completed BEFORE the swap ever happened - so it passed against a build with no migration support at all. The second asserted only that the request succeeded, which a server that ignores the change also does, because it had already answered. What discriminates is whether the server ever sends to the client's NEW address: measured with the fix at 3 datagrams back, and without it at 0. Confirmed red then green, with a control through a forwarder that never swaps so the forwarder itself is not the variable. SCOPE, deliberately: this fixes a single-reactor server. #205 remains - on a multi-reactor server the datagram is delivered by the kernel's 4-tuple hash and may not reach the reactor that owns the connection at all. Closing that needs a cross-reactor registry, which does not exist today and is an architectural decision for a shared-nothing runtime rather than something to bolt on here, and it competes with eBPF reuseport steering that would make the forwarding path unnecessary. E2E 148 passed, 0 failed.
…es not hold Two independent reviews of the previous commit. Between them they found five real problems, one of which was a regression that commit introduced, and one of which invalidated its own test result. THE CLAIM WAS FALSE. Six places - the commit message, the C callback comment, the interop doc, the managed callback doc, the core doc, and the shim's own inline reasoning - said on_path_change fires "only after PATH_CHALLENGE/PATH_RESPONSE succeeded". It does not. conn_recv_non_probing_pkt_on_new_path copies the new dcid into conn->dcid.current BEFORE creating the path validation, and for a plain remote-address change ngtcp2's pref_addr guard is false, so adoption is unconditional. get_path2 therefore moves on the first decryptable non-probing packet from a new address, and validation follows. All six now say so. What actually makes this safe is not the ordering I asserted: the packet must decrypt under 1-RTT keys, which an off-path spoofer cannot forge, and ngtcp2 caps an unvalidated path at 3x what it received. Both reviewers reached that independently. Both also suggested ngtcp2's path_validation callback as "the correct primitive". It is not, for this job: ngtcp2 addresses its OWN output to dcid.current from the moment it adopts, so a transport that waited for validation to move PeerAddr would send ngtcp2's new-path packets to the old address - the next bug inverted. Following get_path2 is right; only the claim about it was wrong. A PATH CHANGE IS NOT ONLY VISIBLE FROM READ. ngtcp2 changes dcid.current in four places and only three are inside read_pkt. The fourth is conn_on_path_validation_failed, reached from inside a WRITE, restoring the previously validated path when a probe times out - and because ngtcp2 also rewrites our remote_addr on every write, the next read compared equal and never fired again. The caller stayed pinned to an address that had just failed validation, permanently. A connection that survived before that commit died at the idle sweep after it. There is now one iq_sync_path, called from read, write and expiry. writev_stream's path argument is an OUT parameter and was being discarded. ngtcp2 writes the destination it chose for THAT datagram into it - for a PATH_RESPONSE, the address the challenge arrived from, which RFC 9000 8.2.2 requires. Syncing after the write is what lets those go where ngtcp2 addressed them, and the GSO batch is flushed before the address moves so datagrams queued for one peer are not readdressed to another. The batch's "single destination by construction" comment said something no longer true and now states the condition. THE CALLBACK TABLE HAD NO WAY TO DETECT SKEW. iq_callbacks grew a ninth pointer; it is passed BY VALUE and mirrored by hand in EIGHT managed declarations - a reviewer found five, the compiler found three more. The callee read 72 bytes where callers wrote 64, so on_path_change was whatever was on the stack. Latent only because those callers pass no remote address, but the previous commit's "E2E 148 passed" was collected against a mismatched ABI. The struct now leads with its own size and the shim refuses a table it does not recognise, which is the only thing that can catch this - nothing else on either side can see it. Also: active_connection_id_limit was ngtcp2's default of 2, the RFC minimum. A migrating client must use a fresh CID and ngtcp2 pops one from that pool; during a validation window 2 is current + fallback + nothing spare, so a second migration finds it empty, ngtcp2 swallows it, the path never moves and the connection blackholes silently. Now 8. Smaller: UpdatePeerAddress guards a freed PeerAddr the way its sibling Send always has (a null destination there is an access violation, not a catchable exception); remote_salen is bounded below as well as above; the change detector is ngtcp2_sockaddr_eq rather than memcmp, so padding and flowinfo cannot fake a change. The test's forwarder set ReceiveTimeout on a stale local capture while receiving through the volatile field, so a swapped-in socket kept the default of block-forever and the pump wedged until disposal - a background thread outliving its test, which is how one suite starts perturbing another. It reads the field once per pass now, and sockets get their timeout at birth. MY OWN REGRESSION, found by running the suite rather than by either review: moving the path sync ahead of the error check, I replaced `return rv` with `return 0` and swallowed every ngtcp2 error. Twelve E2E tests failed, all "the peer was never told" - connections that should have been closed never were. reported_addr is also seeded at accept now, so the first sync is a no-op rather than a path change every connection never had. Migration re-verified red then green after all of it. All suites: 430 passed, 0 failed, 19 pending.
This was referenced Aug 19, 2026
MDA2AV
added a commit
that referenced
this pull request
Aug 20, 2026
…#210) * udp: make the socket buffer a knob, and say when the kernel does not honour it The receive/send buffer asked for on every UDP socket was a private const of 8 MiB, with a comment noting the kernel clamps it to net.core.rmem_max and calling the result "best-effort headroom". The comment was right and the consequence was invisible: on a stock Linux box that ceiling is 212,992 bytes, so the 8 MiB request is granted at about a fortieth of its size, and the only symptom is datagrams dropped under load - which reads as a bug in whatever is running on top. Measured here, unmodified: 4.2% of datagrams dropped to rcvbuf overflow on the h3 benchmark at saturation. setsockopt does not fail in this case, it clamps and reports success, so nothing said so. Two changes. SocketBufferBytes becomes a UdpOptions knob (default unchanged at 8 MiB), because an operator who raises the ceiling has no way to tell ioxide to use it. And the granted size is read back with getsockopt and reported once per process when it falls well short - once, not per reactor per port, since the clamp is a property of the machine. The message deliberately stops at the fact and does not tell anyone to raise the cap. Granting the full 8 MiB on this machine cost about 45% of h3 throughput at saturation on unmodified main: the drops stopped, a deep standing queue took their place, and peers timed out and retransmitted instead - 1.2 datagrams per request became 5.2. A shallow buffer drops early, and early drops are the signal congestion control is built to read. Which behaviour is wanted depends on the deployment, so this reports and leaves the judgement. A zero or negative request is refused rather than clamped, because it would leave the socket on the kernel minimum and look identical to the clamp it is trying to make visible. * quic: route a migrated client's datagrams to the reactor that owns its connection (#205) Fixes #205. Every reactor binds the QUIC port with SO_REUSEPORT and the kernel chooses between them by hashing the sender's address. That is the right answer only while the address holds still, so a NAT rebind or a client changing network re-hashes to a reactor that has never heard of the connection, whose short-header packets it then drops. #209 taught the transport to migrate; this is what lets the packets reach the connection that could act on it. It bites the default configuration, since ReactorCount defaults to one per core. The connection cannot move to meet the packet. The ngtcp2 conn, the picotls session, the open streams and their ring-bound buffers are owned by one reactor thread and documented reactor-thread- only throughout; moving live state to whichever reactor a datagram landed on is precisely what shared-nothing forbids. So the datagram moves instead, which is ordinary message passing and rides ScheduleOnReactor - already public, already used for exactly this. Every connection id the server mints now carries its owning reactor in the first byte, chosen so cid[0] % ReactorCount is that reactor while the rest stays random (iq_stamp_shard). A reactor that receives a short header for an id it does not have reads that byte, copies the datagram and posts it to the owner. The copy is the point, not an inefficiency: the payload lives in the receiving reactor's io_uring provided-buffer ring, which is returned as soon as dispatch ends, so handing the owner a pointer into it would be a use-after-free under load. What crosses a thread is bytes, never reactor state. Only short headers are forwarded. A short header means the handshake finished, so the id is one this server minted and its first byte really does name the owner. A long header carries an id the CLIENT chose, and routing on a byte the peer controls would let anyone aim traffic at a reactor of their choosing. QuicOptions.Routing offers the alternative. KernelFilter additionally attaches a classic-BPF program to the reuseport group so the kernel routes by connection id directly. It is not the default, and the measurements say why - h3 benchmark, two reactors, one machine: Forward nothing at all until a client moves, then ~8.5us per datagram KernelFilter free with CPU headroom, ~-12% throughput at saturation, nothing for migrated The forward cost is a cross-thread wake, not work: CPU per request moved 2.44 -> 2.46us with every datagram forwarded, while throughput halved at fixed concurrency, and the reactors sat 36% idle. So Forward charges only the connections that actually migrate; KernelFilter charges every packet a little kernel work, invisible until there is no headroom left. Unless a large share of clients migrate, Forward is cheaper in aggregate. KernelFilter needs reactors to open their UDP sockets in ShardIndex order, since the program answers with a position in the reuseport group and that position is bind order. That rendezvous exists only under KernelFilter; the default leaves startup untouched. It also degrades rather than fails: if the kernel refuses the program, ioxide says so and forwarding stays underneath. Correctness never depends on the filter, only cost does. Both modes are tested and each discriminates in the opposite direction - Forward asserts datagrams were forwarded, KernelFilter asserts none were and that the program actually attached, so it cannot pass vacuously as Forward under another name. Confirmed both ways: suppressing the forward makes the Forward test fail. StartQuicSharded is new because every other QUIC entry point in the harness pins ReactorCount = 1, where a datagram has nowhere wrong to land. QuicForwardsSent/Received/Dropped and QuicStaleDatagrams are exposed for operators. The distinction matters and only the shard byte makes it possible: a short header for an unknown id that belongs elsewhere is a routing event, while one addressed here is ordinary - a migration retires ids and packets in flight still carry them. Also here, found on the way and not separable from the shim rebuild: iq_sync_path was calling ngtcp2_sockaddr_eq, which lives in lib/ngtcp2_addr.h and is NOT shipped under lib/includes. It compiled only by implicit declaration and stops building outright on GCC 14+, where that is an error. Replaced with a local mirror over the public types; a plain memcmp will not do, since sockaddr padding and flowinfo would read as an address change and fire migration callbacks on a connection that never moved. iq_abi is a new exported surface revision the managed side checks when it builds an engine, so a stale libioxide_ngtcp2.so fails at startup with a clear message instead of passing garbage across a boundary that just grew two parameters. Stale .so files have silently invalidated audit runs here before. * docs: a page for how ioxide does HTTP/3, end to end The learn section had QUIC & HTTP/3, which walks a datagram through the transport and is written for someone reading the code. What it does not answer is the set of questions a deployment actually has: which certificate is served for which name, how one gets renewed without dropping traffic, how client certificates are checked, what happens when a client's address changes, and what a fleet of reactors does about it. Those were spread across doc comments and samples or nowhere at all. /how-ioxide-does-h3 collects them. It covers the three layers and what each owns, why TLS on QUIC is a separate stack from TLS on TCP with its own configuration (the thing most likely to catch someone running both), SNI and why the host table closes at CreateFactory, ReplaceCertificates and its three surprising properties, mutual TLS and why PeerCommonName exists rather than substring-matching the subject, connection migration, and the routing modes added in #205 with the measurements behind the default. It links to QUIC & HTTP/3 rather than repeating it: this page is what the server does, that one is how the code does it. quic-h3.html gains the same routing material in short form, because its ingress walkthrough said unknown short-header packets are dropped as stale traffic - true when it was written, and no longer true for a fleet, where such a packet may belong to a sibling reactor. The full comparison lives on the new page so the two cannot drift. * playground: show the QUIC routing knob at its default, and regenerate the panes Every sample carries its full knob set at the shipping default, so a reader can see what is configurable without going looking. QuicOptions.Routing was missing from all twelve QUIC and HTTP/3 samples. Set explicitly to QuicRouting.Forward - the default - with what the choice actually means: Forward costs nothing until a client changes address, KernelFilter has the kernel route by connection id instead and costs a little on every packet. The comment points at /how-ioxide-does-h3 rather than restating the measurements, so there is one place for them to be wrong. Site panes regenerated from the samples, since they are generated rather than written. * chore: 0.7.210 across the packages * tests: stop the h3 client opening a control stream per request H3TestClient built a fresh HTTP/3 session inside Request() - ih3_client_new plus a new control, QPACK encoder and QPACK decoder stream - on every call. HTTP/3 permits exactly one control stream per peer, and RFC 9114 6.2.1 requires a second one to be treated as a connection error of type H3_STREAM_CREATION_ERROR. So from the second request onward this client was speaking invalid HTTP/3. The server was right and said so. nghttp3 returned H3_STREAM_CREATION_ERROR from read_stream on the duplicate stream (type 0x00 carrying SETTINGS, and later a duplicate QPACK decoder), the run loop set _protocolFailed, exited, and closed the connection - about a millisecond after the first response. Every nghttp3 run loop did it: buffered sync, buffered async, streaming, and streamed response. The pure-C# stack did not, because it never saw a second control stream: it only ever ran one request per connection in these tests. None of that was visible, for the worst possible reason. The requests already in flight kept being answered off state the transport had already dismantled - the CIDs unregistered, the connection out of _quicConnSet, PeerAddr freed and zeroed, the transport's reference dropped - so three more requests returned 200 apiece and every test stayed green. A test client that provokes a connection error and then passes anyway is worse than one that fails. Found while investigating why a migrated connection's pinned socket died instantly: the pin is released in QuicRemoveConnection, which this was triggering after the first response. Fixing this is a prerequisite for that work, and for any test that means to exercise more than one request over one connection. The session now stands up once per connection; only the bidi request stream and the response state are per request. 431 pass across E2E, Unit, Chaos, Http, Tls and File. Connections now live to the end of their test and exit clean (isClosed=True, protocolFailed=False) rather than on a protocol failure. * tests: give the h3 client loss recovery, so a lost datagram is not fatal H3TestClient never called iq_conn_handle_expiry. The shim has exported it, and iq_conn_expiry beside it, since the engine binding landed - the client simply never fired either. So it had no loss recovery whatsoever: a dropped datagram left its packet unacked forever, nothing retransmitted, the congestion window filled, and writev_stream began answering 0 with nothing consumed. Both ends then sat silent until the 10 s write deadline called it a stall. The diagnosis took a while because the symptom accuses the server. What settled it was that the client's own sent and received datagram counts were FROZEN across every spin - it was neither sending nor receiving, so it was not blocked on anything the peer had done. Flow control looks different and recovers: the streaming-upload test shows n=-208 (STREAM_DATA_BLOCKED) and comes back. Here it was n=0 with consumed=-1, which is ngtcp2 saying it has nothing to send at all. Nothing here ever lost a packet before, which is why an omission this size survived: every connection served one request over loopback and was gone. Once connections started living across several requests, and an address change started discarding datagrams in flight to the forwarder's old socket, the gap became a hang - and it presented as a server that stopped answering after a migration, which is exactly the bug it is not. Timers fire from PumpIn, which every wait loop already drives. Three consecutive full E2E runs green; the migration tests had been failing roughly two runs in three before this. * tests: re-mint a spec fixture when it stops meaning what it was minted to mean The mtls test "a client certificate that is not valid yet is refused" failed today, reporting that the server had served a not-yet-valid certificate. It had not. The fixture had gone stale. Spec fixtures are cached by their spec rather than by freshness, deliberately, because most of them are supposed to be invalid and re-minting an expired one every run would defeat the test. That is safe in one direction only. An expired certificate stays expired forever; one minted to be NOT YET VALID becomes valid the moment its notBefore arrives. This fixture is minted with notBefore = now + 1 day, so the copy left in /tmp yesterday became a perfectly good certificate overnight, the server accepted it exactly as it should, and the test called that a product bug. A test that fails once a day has passed is a bug in the test, and one that accuses the product of a security defect is worse than most. The cached file is now reused only while its actual validity state still matches the intent - not yet started if the spec asked for that, expired if it asked for that. Confirmed both ways: planting an already-valid certificate in the not-yet-valid fixture's place makes the cache re-mint it (CN=alice back to CN=future-alice, notBefore tomorrow), where before it was served as-is and the test failed. Unrelated to the QUIC work in flight; found because the date rolled over mid-session. * quic: claim a migrated client's address, so the forwarding stops after one hop Cross-reactor forwarding (#205) is correct but permanent. The kernel picks a socket by hashing the sender's address, and after a migration that address does not change back - so every one of that client's datagrams keeps landing on the wrong reactor and keeps paying a cross-thread hop for the life of the connection. A datagram socket bound to the same address as the others but connect()ed to one peer is a MORE SPECIFIC match than a wildcard bind, and the kernel's lookup takes the narrowest match before it ever reaches the reuseport hash. So the owning reactor opens one toward the peer's new address and the datagrams arrive there directly. Three things about that, each measured rather than assumed. connect() on a datagram socket puts nothing on the wire - it is a local declaration, and the peer is never told. Adding one does not re-scatter anybody else: eight established peers, none moved. And it cannot bootstrap itself, since the owning reactor only learns the new address from a datagram and the datagrams are going elsewhere - forwarding has to deliver the first one. The two are complements: forwarding makes the claim possible, the claim stops forwarding being forever. The claim is made from the reactor's sweep, NOT from the engine's path-change report, and that distinction is the whole difference between working and thrashing. ngtcp2 reports a path many times while it validates one, alternating between the old address for data and the new one for PATH_CHALLENGE probes. Claiming on each report tore down a working socket and built another, with datagrams already queued on the one it closed - losing packets in order to avoid a hop, which is the wrong way round. Measured: five claims for a single address change, flip-flopping between two ports. From the sweep it is one claim per address, because by then the path has settled. Only connections that actually moved are claimed. The sweep visits every connection every 250 ms, so a guard that asked merely "is this address claimed yet" would spend a descriptor and an armed receive on every connection on the server to change nothing - they are already being delivered here by the hash. PeerAddressMoved is the signal, set where the address is adopted. Bounded and best-effort throughout. A ceiling of 512 concurrent claims, because ngtcp2 adopts a new path BEFORE it finishes validating it, so a forged datagram can reach here; past the ceiling the connection simply keeps forwarding. A bind that fails is swallowed for the same reason - this runs under a callback whose exceptions fault the connection, and faulting a working connection to skip an optimisation is a bad trade. Ignored entirely under KernelFilter, where the kernel already routes by connection id. QuicPinsCreated and QuicPinsOpen expose it. Tests cover both directions: that the forwarding stops after a claim, that one address change produces one claim rather than one per path report, and that a fleet whose clients never move claims nothing and forwards nothing. Also here, because the migration tests could not be trusted until it was proven: both h3 stacks now assert that the SAME connection object serves before and after the address changes, and that the factory ran exactly once. Status codes cannot tell migration from a client quietly re-handshaking, and a reconnect would take the h3 session, the QPACK tables and any application state with it while the tests stayed green. Confirmed both ways - asserting a DIFFERENT connection makes both fail. 436 tests pass; E2E stable across three consecutive runs. Benchmarks against main: -0.4% Tls/OpenSsl, -0.3% Nghttp3Response, -1.1% ManagedBuffered, -0.1% ManagedStreamedBoth, and Nghttp3Buffered repeats at +2.5%/+1.9% against a baseline that measures +2.1% against itself - so no regression on either h3 stack, buffered or streamed. * chore: trim the comments on the QUIC routing work Roughly half the prose added across #205 and the claim work said the same thing twice, or explained the investigation rather than the code. Kept: why the packet moves instead of the connection, why the copy is not optional, why the claim runs from the sweep, and the classic-BPF listing, which is unreadable without it. One comment was also wrong - it said the claim runs inside ngtcp2's path-change callback, which stopped being true when it moved to the sweep. No behaviour change; 436 tests pass. * docs: fold learn/quic-h3 into how-ioxide-does-h3 Two pages covering one subject, with the routing material duplicated across both. Now one: deployment first (layers, TLS on QUIC, SNI, rotation, mTLS, migration, routing, the counters), then the packet-level walk - ingress, read surface, egress, the engines, timers, and the invariants. learn/quic-h3.html is deleted and every sidebar and inbound link repointed. No broken links.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #204.
Read the scope section before merging: #204's stated cause is fixed, and #204 itself scopes the multi-reactor case out to #205 — but ioxide defaults to one reactor per core, so on a default deployment a migrating client is still blackholed by #205. "Migration works" is not yet true end to end. A client that changed network — or, far more commonly, whose NAT recycled a UDP mapping while it was idle — was blackholed. The server kept answering the address the connection was accepted on, and the connection died at the idle sweep with no error to either side. That is the failure QUIC's connection ID exists to prevent.
The cause was one line hiding the input from the library that already solves this:
ngtcp2 implements path validation in full. It was simply never told an address had changed. So this does not implement migration — it stops concealing it, and lets ngtcp2 decide.
What the two reviews changed
I had this passing and benched before review. Two independent reviewers found five real problems, one of which was a regression this work introduced and one of which invalidated its own test result. That second commit is most of the value here.
The security claim was false. Six places — including the commit message — said the callback fires "only after PATH_CHALLENGE/PATH_RESPONSE succeeded." It does not:
conn_recv_non_probing_pkt_on_new_pathcopies the new DCID intodcid.currentbefore creating the path validation, and for a plain remote-address change ngtcp2'spref_addrguard is false, so adoption is unconditional. Adoption happens on the first decryptable non-probing packet; validation follows.What actually makes it safe is not the ordering I asserted: the packet must decrypt under 1-RTT keys, which an off-path spoofer cannot forge, and ngtcp2 caps an unvalidated path at 3× what it received. Both reviewers reached that independently. All six places now say so.
Both also proposed ngtcp2's
path_validationcallback as "the correct primitive". It isn't, for this job — ngtcp2 addresses its own output todcid.currentfrom the moment it adopts, so waiting for validation to movePeerAddrwould send ngtcp2's new-path packets to the old address. Followingget_path2is right; only the claim about it was wrong.A path change is not only visible from read. ngtcp2 changes
dcid.currentin four places and only three are insideread_pkt. The fourth isconn_on_path_validation_failed, reached from inside a write, restoring the previously validated path when a probe times out — and because ngtcp2 also rewrites ourremote_addron every write, the next read compared equal and never fired again. The caller stayed pinned to an address that had just failed validation, permanently: a connection that survived before this work died at the idle sweep after it. There is now oneiq_sync_path, called from read, write and expiry.writev_stream's path argument is an OUT parameter and was being discarded. ngtcp2 writes the destination it chose for that datagram into it — for aPATH_RESPONSE, the address the challenge arrived from, which RFC 9000 §8.2.2 requires. The GSO batch is flushed before the address moves, so datagrams queued for one peer are never readdressed to another.The callback table had no way to detect skew.
iq_callbacksgrew a ninth pointer; it is passed by value and mirrored by hand in eight managed declarations — a reviewer found five, the compiler found three more. The callee read 72 bytes where callers wrote 64. Latent only because those callers pass no remote address, but the first commit's "148 passed" was collected against a mismatched ABI. The struct now leads with its own size and the shim refuses a table it does not recognise, which is the only thing that can catch this.active_connection_id_limitwas ngtcp2's default of 2, the RFC minimum. A migrating client must use a fresh CID and ngtcp2 pops one from that pool; during a validation window 2 is current + fallback + nothing spare, so a second migration finds it empty, ngtcp2 swallows it, and the connection blackholes silently. Now 8.The test took three attempts
Worth recording, because two of them were green against a build with no migration support at all.
A reviewer then found the forwarder set
ReceiveTimeouton a stale local capture while receiving through the volatile field, so a swapped-in socket kept the default of block-forever and the pump wedged until disposal — a background thread outliving its test, which is how one suite starts perturbing another.My own regression
Found by running the suite, not by either review. Moving the path sync ahead of the error check, I replaced
return rvwithreturn 0and swallowed every ngtcp2 error. Twelve E2E tests failed, all "the peer was never told". Restored, andreported_addris seeded at accept so the first sync is a no-op rather than a path change every connection never had.Scope
This fixes a single-reactor server, which includes the common NAT-rebind case. #205 remains open: on a multi-reactor server the kernel's 4-tuple hash may not deliver to the reactor that owns the connection at all. Closing that needs a cross-reactor registry — which does not exist, each
Reactoris constructed independently and cannot reach its siblings — and it competes with eBPF reuseport steering that would make the forwarding path unnecessary. That is a design decision, not something to bolt on here.Verification
Migration re-verified red then green after every change, with a never-swapping control so the forwarder is not the variable. All suites: 430 passed, 0 failed, 19 pending.
Bench, against main, two runs each plus an untouched control:
Tcp/Raw(control)Http3/Nghttp3BufferedHttp3/ManagedBufferedThe per-datagram sync is an
addrlencompare plus a 16-bytesockaddr_eq; nothing measurable.One loose end: the Tls suite failed once, non-reproducibly, before these fixes and could not be identified; five subsequent runs and every run since were clean. The forwarder's blocking-pump bug is a plausible mechanism and is now fixed, but I cannot prove it was the cause.