Skip to content

Conversation

@minosfuture
Copy link
Contributor

@minosfuture minosfuture commented Nov 15, 2025

Purpose

This is to fix the http.ReadError when concurrency (batch size) is high, by

  • allow unlimited connection
  • close connection asap

Also include minor update to the NixlConnector usage doc for DP deployment.

Test Plan

p/d disagg with both proxy script with concurrency = 4096.

Test Result

No more request failure.


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

@minosfuture minosfuture requested a review from ApostaC as a code owner November 15, 2025 17:58
@mergify
Copy link

mergify bot commented Nov 15, 2025

Documentation preview: https://vllm--28782.org.readthedocs.build/en/28782/

@mergify mergify bot added documentation Improvements or additions to documentation v1 kv-connector labels Nov 15, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses http.ReadError in proxy servers under high concurrency. The fix involves two main changes to the httpx.AsyncClient usage: allowing unlimited connections by adjusting limits, and ensuring connections are released promptly by consuming the response body with await response.aread(). These changes are consistently applied to both disagg_proxy_server.py and toy_proxy_server.py. Additionally, a helpful clarification has been added to the NixlConnector documentation regarding multi-host DP deployment. The changes are logical, well-targeted, and effectively resolve the stated issue. The code quality is good, and I have no further suggestions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation kv-connector v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant