Skip to content

feat(core): add sharded pinned memory pool via --pool-shards#167

Open
feifei-111 wants to merge 3 commits intomasterfrom
mofeite/feat/splited_buffer_alloc
Open

feat(core): add sharded pinned memory pool via --pool-shards#167
feifei-111 wants to merge 3 commits intomasterfrom
mofeite/feat/splited_buffer_alloc

Conversation

@feifei-111
Copy link
Contributor

Split the pinned memory pool into N independent sub-pools with round-robin allocation to reduce allocator lock contention under concurrent save workloads. Default is 1 (unchanged behavior).

feifei-111 and others added 3 commits March 24, 2026 16:14
Split the pinned memory pool into N independent sub-pools with
round-robin allocation to reduce allocator lock contention under
concurrent save workloads. Default is 1 (unchanged behavior).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lizers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@xiaguan
Copy link
Collaborator

xiaguan commented Mar 24, 2026

CI (internal H20): RESULT: PASS for PR head 5b985a7.
Image: vllm/vllm-openai:latest.
TTFT mean: cold 8116.74 ms -> warm 81.53 ms (99.55x).
Requests: 200/200 cold, 200/200 warm, 0 failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants