Conversation
The IO loop is one per process; threads only run spawned `!` work, so
the only way a lightweight route uses N cores is N processes on one
port. Without SO_REUSEPORT the second process gets Fail(Address
already in use). Lemah, a Bend API framework, measured what that
costs: one Bend process with 4 threads serves a light route at 88.3k
req/s, a second process on the same port cannot start at all, and Bun
with 4 processes on one port (reusePort) serves the same route at
118.9k on the same machine.
SO_REUSEADDR was already set unconditionally on the same socket;
SO_REUSEPORT joins it under #ifdef SO_REUSEPORT (the option is not
defined on every platform clang targets, even though every platform
Bend ships for has it). The JS lane sets the numeric equivalent through
the same io_sys() FFI tcp_listen.js already uses for SO_REUSEADDR
(2/0xf on Linux, 4/0x200 on mac); it talks to libc directly and never
goes through Node's or Bun's listen options.
Considered an opt-in argument on TCP.listen instead of default-on.
Rejected: TCP.listen takes only a port, so opt-in means a new
signature; SO_REUSEADDR is already default-on in the same function;
and no existing test asserts that a second TCP.listen on a bound port
fails (grepped every test that calls TCP.listen), so nothing regresses.
The one observable trade-off: two listeners on one port by mistake now
share its connections (the kernel hashes them across listeners) instead
of the second failing. Bun (`reusePort: true`) and Node 23.1+
(`net.listen({reusePort})`) expose the same option opt-in.
tests/io/tcp_listen_shared.bend listens twice on the same port in one
process; both must answer Done{} (the second answered Fail before this
change -- checked by reverting the two effs files and rerunning).
Checked, interpreted, JS-built and C-built locally (this repo's own
bend2/main.ts, not the installed toolchain, so the edited effs are the
ones exercised): all four lanes print "first" / "second"; tcp_connect_
black_hole, tcp_listen_close, tcp_loopback, tcp_poll, tcp_send_slow_
peer, tcp_short_recv, tcp_spawn and port_bound keep their existing
output in all four lanes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
rescuing this PR by commenting (github bug caused it to be hidden). |
| #ifdef SO_REUSEPORT | ||
| setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one)); | ||
| #endif |
There was a problem hiding this comment.
On macOS, SO_REUSEPORT permits duplicate listeners, but it doesn't appear to distribute incoming connections between them. In my testing with two processes (in both C and JS), all 200 connections were handled by a single process. The new test passes, but it doesn't exercise actual traffic acceptance, so it may not catch this behavior. It might be worth documenting this limitation and adding a test for connection distribution before we present this as enabling a process-per-core server on macOS.
There was a problem hiding this comment.
You are right, and the promise in the guide was too broad. Fixed in afa92fd.
On Linux the kernel does spread the load: two processes sharing one port took 100 and 100 of 200 sequential connections here (Linux 6.x, x86-64), so process-per-core holds there. I have no macOS to test on, and your result matches what the option means on that side -- BSD's SO_REUSEPORT permits the duplicate bind, and the balancing one is FreeBSD's separate SO_REUSEPORT_LB, which macOS does not have. So the guide and the comment at the call now say both: N processes share the port, Linux spreads accepted connections over them, macOS permits the bind but does not distribute.
On the test: I kept it to the duplicate bind, which is what the change guarantees on every platform, rather than adding a distribution test. Which listener accepts a given connection is the kernel's decision and differs by platform, so a cross-platform assertion would either be flaky on Linux or fail outright on macOS. If you would rather have it pinned per platform, I am happy to add it -- say the shape you want and I will write it.
| const one = new Int32Array([1]); | ||
| const level = sys.mac ? 0xffff : 1; | ||
| sys.setsockopt(fd, level, sys.mac ? 4 : 2, sys.ptr(one), 4); | ||
| sys.setsockopt(fd, level, sys.mac ? 0x200 : 15, sys.ptr(one), 4); |
There was a problem hiding this comment.
Both backends ignore the return value of the new setsockopt call. Injecting an option failure in JS makes the first listener report success, then the second fails with Address already in use; the original setup error disappears. Check the return value and close the socket before reporting the error, or explicitly document and test a fallback if sharing is optional
There was a problem hiding this comment.
Good catch, and it found a second bug. Fixed in afa92fd: the call now fails like the bind, listen and fcntl beside it, closing the socket first, so an option failure is reported where it happens instead of surfacing later as the second listener's EADDRINUSE.
Writing the check is what exposed the other one: my first version put it after the bind, and the C lane's test immediately printed 'Address already in use' instead of 'second' -- the option has to be set before the bind or the kernel has already refused the duplicate. The JS lane happened to be correct because || evaluates left to right. Both lanes now set it before the bind and check the return.
I left SO_REUSEADDR's return unchecked, since changing it is outside this PR, but the same argument applies to it if you want that in a follow-up.
… spreads the load Two things review found. The setsockopt return was ignored, so an option failure surfaced later as the second listener's EADDRINUSE with the real cause gone; it now fails the call like the bind, listen and fcntl beside it, closing the socket first. Writing that check also caught a bug in the first version of it: the call has to run before the bind, or the kernel has already refused the duplicate -- the C lane's test printed "Address already in use" until it moved up. And the promise was too broad. On Linux the kernel spreads accepted connections over the listeners: two processes sharing one port took 100 and 100 of 200 sequential connections here. On macOS the option permits the duplicate bind but does not distribute, so N processes share the port without sharing the load. The guide now says both, and so does the comment at the call. The test stays as it is: two listeners on one port is what the change guarantees everywhere, while which listener accepts a given connection is the kernel's to decide and differs by platform, so it is documented rather than pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Checked this locally on macOS (arm64) against the merge base. Concurrent runs of the TCP tests now collide. On macOS the first listener takes everything. With two listeners on Given that the benefit is Linux-only, I'd take you up on the opt-in shape (e.g. a separate For what it's worth, a single listener shows no throughput change: Note: this comment was written by an AI after it tested the PR for me and I reviewed the findings. If anything here is wrong, reply and I will review it myself. |
What changes
bend2/effs/tcp_listen.csetsSO_REUSEPORTon the listening socket,right after the
SO_REUSEADDRthat is already set unconditionallythere, guarded by
#ifdef SO_REUSEPORTfor portability (the option isnot defined on every platform clang can target, even though every
platform Bend ships for -- Linux and macOS -- has it). The JS lane
(
bend2/effs/tcp_listen.js) sets the numeric equivalent through thesame
io_sys()libc FFI the file already uses forSO_REUSEADDR(2 /0xf on Linux, 4 / 0x200 on mac).
TCP.listen's signature is unchanged.guide/GUIDE.mdgets one sentence next to the existing paragraph onhandles (
File,Socket,Window), sinceListeneris one:`TCP.listen` sets `SO_REUSEPORT`, so N processes can share one port.A new test,
tests/io/tcp_listen_shared.bend, callsTCP.listentwiceon the same port inside one process; both calls must answer
Done{}.Why
Bend's IO loop is one per process -- threads only run spawned
!work -- so the only way a lightweight route uses N cores is N
processes on the same port. Without
SO_REUSEPORTthe second processgets
Fail(Address already in use). Lemah, a Bend API framework,measured what that costs on one 4-core machine, same light route, same
client: one Bend process with 4 threads serves 88.3k req/s; a second
Bend process on the same port cannot start at all; Bun with 4
processes on one port (
reusePort: true) serves 118.9k. The gap isthe socket option, not the runtime.
Alternative considered
An opt-in argument on
TCP.listeninstead of default-on. Rejectedbecause
TCP.listentakes only a port, so opt-in means a newsignature;
SO_REUSEADDRis already default-on in the same function;and no existing test asserts that a second listen on an already-bound
port fails (checked every test that calls
TCP.listen:tests/io/port_bound.bend,tcp_listen_close.bend,tcp_loopback.bend,tcp_poll.bend,tcp_send_slow_peer.bend,tcp_short_recv.bend,tcp_spawn.bend), so nothing regresses. The one observable trade-offof default-on: two listeners on one port by mistake now share its
connections (the kernel hashes them across listeners) instead of the
second failing. Bun (
Bun.serve({reusePort: true})) and Node 23.1+(
net.listen({reusePort: true})) expose the same option as opt-in; ifyou would rather keep the failure, an opt-in shape is a small follow-up
and I am happy to do it.
Test output
tests/io/tcp_listen_shared.bendlistens on port 8730 twice in thesame process, prints
firstafter the firstDone{},secondafterthe second, then closes both listeners.
Checked (
bend tests/io/tcp_listen_shared.bend --check-only):Interpreted, JS-built and C-built (
bend tests/io/tcp_listen_shared.bend,-o x.jsrun under bun,-o xrun natively) all print:Negative check: reverted both effs files and reran the interpreted
lane -- the second
TCP.listen(8730)answersFail, printingAddress already in useinstead ofsecond, confirming the testcatches the regression this PR fixes.
The existing
tests/io/tcp_*tests andport_bound.bend(the othertest that calls
TCP.listen) were also run in all four lanes (check,interpreted, JS, C) against the changed effs and print the same output
as before the change:
tcp_connect_black_hole,tcp_listen_close,tcp_loopback,tcp_poll,tcp_send_slow_peer,tcp_short_recv,tcp_spawn,port_bound.These runs used this repo's own
bend2/main.ts(viabun), not aseparately installed
bendbinary, so the editedeffs/tcp_listen.cand
.jsare the ones actually exercised. The cluster-backedgates/test.ts/gates/repo.tsharness (mini cluster,ttok) was notavailable in this environment;
awk 'length > 80'was run on everychanged file instead to check the guide's column limit by hand.
Duplicate check
GH_CONFIG_DIR=$HOME/.config/gh-omarchy gh issue list -R bendlang/bend --search reuseport --state alland the same for
pr list, and again forSO_REUSEPORTandTCP.listen: no open or closed issue or PR in either search.