Skip to content

TCP.listen sets SO_REUSEPORT: N processes can share a port - #972

Open
aldeni wants to merge 2 commits into
bendlang:mainfrom
aldeni:tcp-listen-reuseport
Open

aldeni wants to merge 2 commits into
bendlang:mainfrom
aldeni:tcp-listen-reuseport

Conversation

@aldeni

@aldeni aldeni commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

What changes

bend2/effs/tcp_listen.c sets SO_REUSEPORT on the listening socket,
right after the SO_REUSEADDR that is already set unconditionally
there, guarded by #ifdef SO_REUSEPORT for portability (the option is
not defined on every platform clang can target, even though every
platform Bend ships for -- Linux and macOS -- has it). The JS lane
(bend2/effs/tcp_listen.js) sets the numeric equivalent through the
same io_sys() libc FFI the file already uses for SO_REUSEADDR (2 /
0xf on Linux, 4 / 0x200 on mac). TCP.listen's signature is unchanged.
guide/GUIDE.md gets one sentence next to the existing paragraph on
handles (File, Socket, Window), since Listener is one:
`TCP.listen` sets `SO_REUSEPORT`, so N processes can share one port.

A new test, tests/io/tcp_listen_shared.bend, calls TCP.listen twice
on the same port inside one process; both calls must answer Done{}.

Why

Bend's IO loop is one per process -- threads only run spawned !
work -- so the only way a lightweight route uses N cores is N
processes on the same port. Without SO_REUSEPORT the second process
gets Fail(Address already in use). Lemah, a Bend API framework,
measured what that costs on one 4-core machine, same light route, same
client: one Bend process with 4 threads serves 88.3k req/s; a second
Bend process on the same port cannot start at all; Bun with 4
processes on one port (reusePort: true) serves 118.9k. The gap is
the socket option, not the runtime.

Alternative considered

An opt-in argument on TCP.listen instead of default-on. Rejected
because TCP.listen takes only a port, so opt-in means a new
signature; SO_REUSEADDR is already default-on in the same function;
and no existing test asserts that a second listen on an already-bound
port fails (checked every test that calls TCP.listen:
tests/io/port_bound.bend, tcp_listen_close.bend, tcp_loopback.bend,
tcp_poll.bend, tcp_send_slow_peer.bend, tcp_short_recv.bend,
tcp_spawn.bend), so nothing regresses. The one observable trade-off
of default-on: two listeners on one port by mistake now share its
connections (the kernel hashes them across listeners) instead of the
second failing. Bun (Bun.serve({reusePort: true})) and Node 23.1+
(net.listen({reusePort: true})) expose the same option as opt-in; if
you would rather keep the failure, an opt-in shape is a small follow-up
and I am happy to do it.

Test output

tests/io/tcp_listen_shared.bend listens on port 8730 twice in the
same process, prints first after the first Done{}, second after
the second, then closes both listeners.

Checked (bend tests/io/tcp_listen_shared.bend --check-only):

All terms check.

Interpreted, JS-built and C-built (bend tests/io/tcp_listen_shared.bend,
-o x.js run under bun, -o x run natively) all print:

first
second

Negative check: reverted both effs files and reran the interpreted
lane -- the second TCP.listen(8730) answers Fail, printing
Address already in use instead of second, confirming the test
catches the regression this PR fixes.

The existing tests/io/tcp_* tests and port_bound.bend (the other
test that calls TCP.listen) were also run in all four lanes (check,
interpreted, JS, C) against the changed effs and print the same output
as before the change: tcp_connect_black_hole, tcp_listen_close,
tcp_loopback, tcp_poll, tcp_send_slow_peer, tcp_short_recv,
tcp_spawn, port_bound.

These runs used this repo's own bend2/main.ts (via bun), not a
separately installed bend binary, so the edited effs/tcp_listen.c
and .js are the ones actually exercised. The cluster-backed
gates/test.ts/gates/repo.ts harness (mini cluster, ttok) was not
available in this environment; awk 'length > 80' was run on every
changed file instead to check the guide's column limit by hand.

Duplicate check

GH_CONFIG_DIR=$HOME/.config/gh-omarchy gh issue list -R bendlang/bend --search reuseport --state all
and the same for pr list, and again for SO_REUSEPORT and
TCP.listen: no open or closed issue or PR in either search.

The IO loop is one per process; threads only run spawned `!` work, so
the only way a lightweight route uses N cores is N processes on one
port. Without SO_REUSEPORT the second process gets Fail(Address
already in use). Lemah, a Bend API framework, measured what that
costs: one Bend process with 4 threads serves a light route at 88.3k
req/s, a second process on the same port cannot start at all, and Bun
with 4 processes on one port (reusePort) serves the same route at
118.9k on the same machine.

SO_REUSEADDR was already set unconditionally on the same socket;
SO_REUSEPORT joins it under #ifdef SO_REUSEPORT (the option is not
defined on every platform clang targets, even though every platform
Bend ships for has it). The JS lane sets the numeric equivalent through
the same io_sys() FFI tcp_listen.js already uses for SO_REUSEADDR
(2/0xf on Linux, 4/0x200 on mac); it talks to libc directly and never
goes through Node's or Bun's listen options.

Considered an opt-in argument on TCP.listen instead of default-on.
Rejected: TCP.listen takes only a port, so opt-in means a new
signature; SO_REUSEADDR is already default-on in the same function;
and no existing test asserts that a second TCP.listen on a bound port
fails (grepped every test that calls TCP.listen), so nothing regresses.
The one observable trade-off: two listeners on one port by mistake now
share its connections (the kernel hashes them across listeners) instead
of the second failing. Bun (`reusePort: true`) and Node 23.1+
(`net.listen({reusePort})`) expose the same option opt-in.

tests/io/tcp_listen_shared.bend listens twice on the same port in one
process; both must answer Done{} (the second answered Fail before this
change -- checked by reverting the two effs files and rerunning).
Checked, interpreted, JS-built and C-built locally (this repo's own
bend2/main.ts, not the installed toolchain, so the edited effs are the
ones exercised): all four lanes print "first" / "second"; tcp_connect_
black_hole, tcp_listen_close, tcp_loopback, tcp_poll, tcp_send_slow_
peer, tcp_short_recv, tcp_spawn and port_bound keep their existing
output in all four lanes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@zxv

zxv commented Sep 22, 2026

Copy link
Copy Markdown

rescuing this PR by commenting (github bug caused it to be hidden).

Comment thread bend2/effs/tcp_listen.c Outdated
Comment on lines +11 to +13
#ifdef SO_REUSEPORT
setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one));
#endif

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On macOS, SO_REUSEPORT permits duplicate listeners, but it doesn't appear to distribute incoming connections between them. In my testing with two processes (in both C and JS), all 200 connections were handled by a single process. The new test passes, but it doesn't exercise actual traffic acceptance, so it may not catch this behavior. It might be worth documenting this limitation and adding a test for connection distribution before we present this as enabling a process-per-core server on macOS.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, and the promise in the guide was too broad. Fixed in afa92fd.

On Linux the kernel does spread the load: two processes sharing one port took 100 and 100 of 200 sequential connections here (Linux 6.x, x86-64), so process-per-core holds there. I have no macOS to test on, and your result matches what the option means on that side -- BSD's SO_REUSEPORT permits the duplicate bind, and the balancing one is FreeBSD's separate SO_REUSEPORT_LB, which macOS does not have. So the guide and the comment at the call now say both: N processes share the port, Linux spreads accepted connections over them, macOS permits the bind but does not distribute.

On the test: I kept it to the duplicate bind, which is what the change guarantees on every platform, rather than adding a distribution test. Which listener accepts a given connection is the kernel's decision and differs by platform, so a cross-platform assertion would either be flaky on Linux or fail outright on macOS. If you would rather have it pinned per platform, I am happy to add it -- say the shape you want and I will write it.

Comment thread bend2/effs/tcp_listen.js Outdated
const one = new Int32Array([1]);
const level = sys.mac ? 0xffff : 1;
sys.setsockopt(fd, level, sys.mac ? 4 : 2, sys.ptr(one), 4);
sys.setsockopt(fd, level, sys.mac ? 0x200 : 15, sys.ptr(one), 4);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both backends ignore the return value of the new setsockopt call. Injecting an option failure in JS makes the first listener report success, then the second fails with Address already in use; the original setup error disappears. Check the return value and close the socket before reporting the error, or explicitly document and test a fallback if sharing is optional

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and it found a second bug. Fixed in afa92fd: the call now fails like the bind, listen and fcntl beside it, closing the socket first, so an option failure is reported where it happens instead of surfacing later as the second listener's EADDRINUSE.

Writing the check is what exposed the other one: my first version put it after the bind, and the C lane's test immediately printed 'Address already in use' instead of 'second' -- the option has to be set before the bind or the kernel has already refused the duplicate. The JS lane happened to be correct because || evaluates left to right. Both lanes now set it before the bind and check the return.

I left SO_REUSEADDR's return unchecked, since changing it is outside this PR, but the same argument applies to it if you want that in a follow-up.

… spreads the load

Two things review found. The setsockopt return was ignored, so an option
failure surfaced later as the second listener's EADDRINUSE with the real
cause gone; it now fails the call like the bind, listen and fcntl beside
it, closing the socket first. Writing that check also caught a bug in the
first version of it: the call has to run before the bind, or the kernel
has already refused the duplicate -- the C lane's test printed "Address
already in use" until it moved up.

And the promise was too broad. On Linux the kernel spreads accepted
connections over the listeners: two processes sharing one port took 100
and 100 of 200 sequential connections here. On macOS the option permits
the duplicate bind but does not distribute, so N processes share the port
without sharing the load. The guide now says both, and so does the
comment at the call.

The test stays as it is: two listeners on one port is what the change
guarantees everywhere, while which listener accepts a given connection is
the kernel's to decide and differs by platform, so it is documented
rather than pinned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@nicolas-abril nicolas-abril self-assigned this Sep 23, 2026
@nicolas-abril

Copy link
Copy Markdown
Collaborator

Checked this locally on macOS (arm64) against the merge base.

Concurrent runs of the TCP tests now collide. tcp_loopback, tcp_poll, tcp_short_recv, tcp_spawn and tcp_send_slow_peer find a port with tcp_listen_any, which moves to port + 1 when TCP.listen fails. With SO_REUSEPORT always on, a port held by another Bend process no longer fails, so concurrent copies share it. Running 4 copies of each test at once: base 20/20 pass; with this PR, tcp_loopback, tcp_short_recv and tcp_spawn fail 4/4 and tcp_poll 3/4. Typically one copy hangs until killed because its own client's connection was accepted by another copy. The gate runs tests one at a time, so it won't show this, but parallel local runs and any user program that probes for a free port the same way will hit it.

On macOS the first listener takes everything. With two listeners on 0.0.0.0 and the same port, the first to bind got 200/200 connections in repeated runs. So on macOS, an accidental second instance now starts without error and never receives a connection, where it used to fail with EADDRINUSE.

Given that the benefit is Linux-only, I'd take you up on the opt-in shape (e.g. a separate TCP.listen_shared(port)), which also leaves tcp_listen_any working. If it stays default-on, the tests need another way to detect a taken port, and tcp_listen_shared should probably use tcp_listen_any too, not a fixed 8730.

For what it's worth, a single listener shows no throughput change: demos/io_http_server under ab gives about 47k req/s (C) and 44k req/s (JS) on both base and this branch.

Note: this comment was written by an AI after it tested the PR for me and I reviewed the findings. If anything here is wrong, reply and I will review it myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants