Status after #209. #204 is fixed: the transport is now told the real arrival address and ngtcp2 migrates the connection. That makes this issue the only thing standing between ioxide and working migration — and it bites the default configuration, because ReactorCount defaults to one per core. A single-reactor server survives a rebind today; a default one does not.
Every reactor binds the QUIC port with SO_REUSEPORT and keeps its own _quicConns dictionary. There is no BPF program, no shard id encoded in the connection id, and no cross-reactor table — grep -rn "bpf" src/ is empty. Which reactor a datagram lands on is decided by the kernels 4-tuple hash.
That is fine while the 4-tuple never changes, which is why it has never bitten: a connection is accepted on whichever reactor the hash chose, and every later datagram from the same source hashes the same way.
It stops being fine the moment the client's address changes. A NAT rebind, a network switch, or a deliberate migration re-hashes to a different reactor, which has never heard of that connection id.
What happens then is worth being precise about, because it decides the fix. A migrating client is past its handshake, so its packets carry a short header - and those hit the early return before the factory:
if (_quicConns.TryGetValue(dcid, out conn)) { ...serve...; return; }
if (!longHeader) { return; } // a migrating client stops here: droppedSo the packets are discarded, not turned into a second connection. No duplicate is created and no client is handed a fresh empty session mid-flight - the only thing that creates a connection is a genuine long-header Initial.
The connection itself survives on the reactor that owns it, unreachable, and then dies cleanly: LastSeenMs is only stamped when a datagram arrives at that reactor, so QuicSweep evicts it after the idle timeout and OnEvicted runs Destroy, freeing the ngtcp2 conn, the picotls session and the GCHandle. Nothing leaks; the client simply stops being served.
It cannot be continued elsewhere either, and that is the architecture rather than a missing feature: the ngtcp2_conn, the picotls session, the open streams and their buffers are native memory owned by one reactor thread, and QuicConnection is documented "Reactor thread only" throughout. Moving a live connection between reactors is the one thing shared-nothing forbids - which is why the fix is routing the datagram to the right reactor, not moving state to the wrong one.
So on a multi-reactor server — the shape ioxide actually ships — address changes fail even after #204 gives the transport migration support, because the packet never reaches the connection that could migrate.
The usual fix is to encode the owning shard in the connection ids the server issues, which ioxide is well placed to do since it already controls LocalCidLength and mints its own ids; a BPF SO_ATTACH_REUSEPORT_CBPF steering program is the other route.
Worth noting this has never been executed by a test: every QUIC entry point in the harness pins ReactorCount = 1. A sharded QUIC TestServer is the prerequisite for covering it, in the way StartSharded gaining a per-shard OnStart was the prerequisite for the TLS fleet tests.
Every reactor binds the QUIC port with
SO_REUSEPORTand keeps its own_quicConnsdictionary. There is no BPF program, no shard id encoded in the connection id, and no cross-reactor table —grep -rn "bpf" src/is empty. Which reactor a datagram lands on is decided by the kernels 4-tuple hash.That is fine while the 4-tuple never changes, which is why it has never bitten: a connection is accepted on whichever reactor the hash chose, and every later datagram from the same source hashes the same way.
It stops being fine the moment the client's address changes. A NAT rebind, a network switch, or a deliberate migration re-hashes to a different reactor, which has never heard of that connection id.
What happens then is worth being precise about, because it decides the fix. A migrating client is past its handshake, so its packets carry a short header - and those hit the early return before the factory:
So the packets are discarded, not turned into a second connection. No duplicate is created and no client is handed a fresh empty session mid-flight - the only thing that creates a connection is a genuine long-header Initial.
The connection itself survives on the reactor that owns it, unreachable, and then dies cleanly:
LastSeenMsis only stamped when a datagram arrives at that reactor, soQuicSweepevicts it after the idle timeout andOnEvictedrunsDestroy, freeing the ngtcp2 conn, the picotls session and the GCHandle. Nothing leaks; the client simply stops being served.It cannot be continued elsewhere either, and that is the architecture rather than a missing feature: the
ngtcp2_conn, the picotls session, the open streams and their buffers are native memory owned by one reactor thread, andQuicConnectionis documented "Reactor thread only" throughout. Moving a live connection between reactors is the one thing shared-nothing forbids - which is why the fix is routing the datagram to the right reactor, not moving state to the wrong one.So on a multi-reactor server — the shape ioxide actually ships — address changes fail even after #204 gives the transport migration support, because the packet never reaches the connection that could migrate.
The usual fix is to encode the owning shard in the connection ids the server issues, which ioxide is well placed to do since it already controls
LocalCidLengthand mints its own ids; a BPFSO_ATTACH_REUSEPORT_CBPFsteering program is the other route.Worth noting this has never been executed by a test: every QUIC entry point in the harness pins
ReactorCount = 1. A sharded QUICTestServeris the prerequisite for covering it, in the wayStartShardedgaining a per-shardOnStartwas the prerequisite for the TLS fleet tests.