There is no way to wait on a deadline from a reactor without leaving it.
Task.Delay resumes on the thread pool, which drags a connection off the reactor that owns its ring and buffers. Reactor.ScheduleOnReactor fixes affinity but is not a timer: something still has to decide when to post.
io_uring already has the right primitive. IORING_OP_TIMEOUT (or a linked IORING_OP_LINK_TIMEOUT) puts the deadline in the kernel and completes it on the ring, on the reactor thread, with no second thread anywhere and no per-timer allocation on the managed side.
What a consumer has to build without it
HttpArena's ioxide entry needs this for a benchmark profile that answers GET /delay/{ms} after the milliseconds named in the path, with 64,000 connections in flight — so 64,000 pending timers at all times. What I ended up writing:
- one dedicated process thread, ticking every 250 µs
- per-reactor pending queues in
[ThreadStatic] state, so the hot path takes no lock
- the tick thread posts one
ScheduleOnReactor drain per reactor per tick, and the reactor pops everything now due in one pass
The batching is what makes it viable. One post per expiry would have been ~4M cross-thread posts a second at that load; one post per reactor per tick is ~64.
It works — 1.02M req/s at 16,384 connections, 93% of the arithmetic ceiling, 15.8 ms measured against a 15 ms request — but it is a userspace tick standing in for a kernel timer:
- the 250 µs tick is a floor on accuracy, and it shows up as a systematic overshoot in a number being compared across frameworks
- it burns a thread spinning, because
Thread.Sleep(1) is coarser than the tick it has to keep
- every consumer that needs a timeout has to write it again
Proposal
Something along the lines of:
public Task DelayAsync(Reactor reactor, TimeSpan due, CancellationToken ct = default);
backed by IORING_OP_TIMEOUT, completing on the reactor thread like every other ring completion.
A lower-level "submit this timeout and complete this task" escape hatch would work equally well and would cover the cases I have not thought of — right now ioxide.pg gets ConnectAsync/SendAsync/RecvAsync and everyone else gets nothing, so anything that is not socket-shaped has no route onto the ring at all.
Context: MDA2AV/HttpArena#1341. Related: #211.
There is no way to wait on a deadline from a reactor without leaving it.
Task.Delayresumes on the thread pool, which drags a connection off the reactor that owns its ring and buffers.Reactor.ScheduleOnReactorfixes affinity but is not a timer: something still has to decide when to post.io_uring already has the right primitive.
IORING_OP_TIMEOUT(or a linkedIORING_OP_LINK_TIMEOUT) puts the deadline in the kernel and completes it on the ring, on the reactor thread, with no second thread anywhere and no per-timer allocation on the managed side.What a consumer has to build without it
HttpArena's
ioxideentry needs this for a benchmark profile that answersGET /delay/{ms}after the milliseconds named in the path, with 64,000 connections in flight — so 64,000 pending timers at all times. What I ended up writing:[ThreadStatic]state, so the hot path takes no lockScheduleOnReactordrain per reactor per tick, and the reactor pops everything now due in one passThe batching is what makes it viable. One post per expiry would have been ~4M cross-thread posts a second at that load; one post per reactor per tick is ~64.
It works — 1.02M req/s at 16,384 connections, 93% of the arithmetic ceiling, 15.8 ms measured against a 15 ms request — but it is a userspace tick standing in for a kernel timer:
Thread.Sleep(1)is coarser than the tick it has to keepProposal
Something along the lines of:
backed by
IORING_OP_TIMEOUT, completing on the reactor thread like every other ring completion.A lower-level "submit this timeout and complete this task" escape hatch would work equally well and would cover the cases I have not thought of — right now
ioxide.pggetsConnectAsync/SendAsync/RecvAsyncand everyone else gets nothing, so anything that is not socket-shaped has no route onto the ring at all.Context: MDA2AV/HttpArena#1341. Related: #211.