Improve websocket liveness checks and idle timeout handling - #5517
Open
aasoni wants to merge 4 commits into
Open
Improve websocket liveness checks and idle timeout handling#5517aasoni wants to merge 4 commits into
aasoni wants to merge 4 commits into
Conversation
aasoni
force-pushed
the
aasoni/ws-idle-timeout-rework
branch
from
July 10, 2026 14:30
e4fc66d to
234f6c9
Compare
jsdt
reviewed
Jul 10, 2026
Contributor
|
@aasoni If it is still to be merged, can you update this branch + resolve conflicts? |
aasoni
force-pushed
the
aasoni/ws-idle-timeout-rework
branch
2 times, most recently
from
August 12, 2026 09:54
d7eeca2 to
d893391
Compare
aasoni
force-pushed
the
aasoni/ws-idle-timeout-rework
branch
from
August 12, 2026 11:18
d893391 to
1af4d2b
Compare
kim
approved these changes
Aug 12, 2026
jsdt
approved these changes
Aug 12, 2026
aasoni
enabled auto-merge
August 12, 2026 19:54
auto-merge was automatically disabled
August 13, 2026 14:46
Pull Request is not mergeable
- Idle timeout no longer requires receive activity alone: the send loop extends the idle deadline whenever it makes write progress (rate-limited to once per second). - On idle timeout, initiate a proper close handshake carrying an "idle timeout" reason instead of abruptly tearing down the connection, so well-behaved clients can tell why they were disconnected. The connection is torn down if the handshake does not complete within a 10s grace period. - The "websocket connection aborted" warning and ws_clients_aborted metric now fire only when the client actor task is actually aborted or panics, not on every normal disconnect. - Fix the kick log message to report the channel's configured capacity instead of its remaining capacity (always 0 at that point).
Instead of passing around part of a watch to extend the timeout, this change adds a `last_activity` field to `ActorState`. It can be updated by calling `record_activity()` on the actor state, and you can get an idle timeout future by calling `idle_timer` on the actor state. IMO this is a bit more straightforward, and it reduces how many arguments we are passing around.
Per review feedback: completing a ws.feed only means the socket buffer (including any LB/proxy buffers) isn't full, not that the client is making progress, so a dead client could be kept alive for a long time by small periodic messages. Testing also showed this did not help the disconnection issue being investigated. Only data received from the client now counts as activity. Also update docs that still described the old behavior, and fix the unawaited tokio::time::advance calls in the last-activity test.
aasoni
force-pushed
the
aasoni/ws-idle-timeout-rework
branch
from
August 18, 2026 09:25
5613d72 to
e9ee86e
Compare
aasoni
enabled auto-merge
August 18, 2026 09:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
Idle timeout no longer requires a pong to restart: any data received from the client (including
Pongframes) counts as activity and extends the idle deadline. Per review feedback, send progress does not count as activity, completing a write only proves the socket (or LB) buffer isn't full, not that the client is alive, and testing showed counting it did not help the disconnection issue under investigation.On idle timeout, initiate a proper close handshake carrying an "idle timeout" reason instead of abruptly tearing down the connection, so well-behaved clients can tell why they were disconnected. The connection is torn down if the handshake does not complete within a 10s grace period.
Fix the kick log message to report the channel's configured capacity instead of its remaining capacity (always 0 at that point).
API and ABI breaking changes
No
Expected complexity level and risk
3
Testing
These changes were made on the BitCraft cluster and were tested by hundreds of players connecting and subscribing to large amounts of data. I am fairly confident the change doesn't break things. I am less confident it's the disconnection silver bullet, but it may help.