tokio-quiche: server handshake fails on NICs without GSO — acceptor Retry/version-negotiation reply always sets UDP_SEGMENT cmsg (EIO)

[zaczh]

Summary

On a host whose NIC does not support GSO, a tokio-quiche server cannot complete any QUIC handshake. It receives the client's Initial fine, but every reply the acceptor sends (stateless Retry / version negotiation) fails with EIO, so the handshake never progresses and no connection is ever established. This affects IPv4 and IPv6 equally.

The crate already avoids GSO on the client path for exactly this reason, but the server acceptor path has no such guard.

Root cause

ConnectionAcceptor::handshake_reply (used by stateless_retry and version negotiation) unconditionally calls gso::send_to(...) on Linux:

// tokio-quiche/src/quic/router/acceptor.rs — handshake_reply()
#[cfg(target_os = "linux")]
{
    let from = with_pktinfo.then_some(incoming.local_addr);
    let _ = crate::quic::io::gso::send_to(udp, to, from, send_buf, send_buf.len(), ...);
}

gso::send_to always attaches a UDP_SEGMENT cmsg (made unconditional in #2060), even for a single-segment packet:

// tokio-quiche/src/quic/io/gso.rs
cmsgs.push(ControlMessage::UdpGsoSegments(&segment_size_u16));

On a NIC that can't do GSO, sendmsg() with a UDP_SEGMENT cmsg returns EIO. So the Retry reply never leaves the box, the client keeps retransmitting its Initial, and nothing happens.

The crate is already aware that some NICs lack GSO and deliberately avoids it on the client path:

// tokio-quiche/src/quic/mod.rs
// Don't apply_max_capabilities(): some NICs don't support GSO

…but the server acceptor's handshake_reply has no equivalent guard. The has_gso capability only gates the io-worker's connection sends (worker.rs::flush_buffer_to_socket checks self.cfg.with_gso); it does not gate the acceptor's Retry/version-negotiation path. Consequently, even constructing the QuicListener with GSO disabled (default SocketCapabilities, has_gso = false) does not help — the acceptor still calls gso::send_to and still fails.

Environment / repro

• Linux 5.15, virtio_net.
• The vNIC reports scatter-gather: off [fixed] and tx-checksum-ip-generic: off [fixed], so generic-segmentation-offload can't be enabled (ethtool -K eth0 gso on → Could not change any device features). This feature set is negotiated by the hypervisor and is seen on some lightweight/budget cloud VPS instances.
• The capability probe is a false positive here: setsockopt(UDP_SEGMENT, …) succeeds (the kernel supports the UDP GSO API regardless of NIC), so apply_max_capabilities() reports has_gso = true. The failure only surfaces at sendmsg() time.

strace of the server while a client tries to connect — it receives:

recvmsg(9, {msg_name={sa_family=AF_INET6, ... "::ffff:<client-ipv4>" ...}, ...}) = 1200

…but every reply errors:

sendmsg(9, {..., msg_control=[{cmsg_len=18, cmsg_level=SOL_UDP, cmsg_type=UDP_SEGMENT}], msg_controllen=24, ...}, 0) = -1 EIO (Input/output error)

(Identical for IPv6 clients.)

Impact

A server on such a NIC is completely unreachable over QUIC while address validation (stateless Retry) is enabled — which is the default. Current workarounds are limited to setting disable_client_ip_validation and building the listener without GSO, or moving to a NIC that supports GSO.

Suggested fix

Make the acceptor's handshake_reply (Retry / version negotiation) respect GSO availability the same way the client path does — fall back to a plain send_to (no UDP_SEGMENT cmsg) when GSO isn't in use. Because the capability probe gives a false positive on these NICs, a fully robust fix would additionally detect EIO on a GSO send and disable GSO for the socket (or perform a real test-send during capability detection instead of relying on the setsockopt succeeding).

[zaczh]

Follow-up: this looks like a regression introduced by #2060.

Before that PR, handshake_reply called:

gso::send_to(udp, to, from, send_buf, 1, 1, None)   // segment_size = 1, num_pkts = 1

and the old gso::send_to only attached the UDP_SEGMENT cmsg when sending multiple segments. With num_pkts = 1, the single-packet Retry went out as a plain sendmsg with no cmsg — which works on any NIC.

#2060 removed that per-packet guard (the cmsg is now pushed unconditionally) and changed the call to:

gso::send_to(udp, to, from, send_buf, send_buf.len(), None)

so the single-packet Retry now always carries UDP_SEGMENT, which is what fails with EIO on NICs that can't segment.

#2060's goal — avoid a stale socket-level segment size truncating packets when the segment size is changed at runtime — is orthogonal to single-packet sends. So restoring a "skip the cmsg when sending a single segment" fast-path, and/or gating the acceptor's Retry/version-negotiation path on actual GSO capability (as the client path already does), would fix this without reintroducing that footgun.

[antoniovicente]

See comment on the PR

[zaczh]

See comment on the PR

Thanks. That PR doesn’t fix this issue and I’ve closed it.