[Bug]: `self.executor_request_queue.get_canceled_req_ids_size()` grows unboundedly unless `enable_attention_dp:true` is set

[josephrocca]

This bug did not exist in v0.21.0. I think it may have been introduced in v1.0.0rc3. The server slows down and eventually basically stops responding to requests. It seems to occur more quickly if there are many client-aborted requests, and possibly high concurrency accelerates it too (so it could be a race condition). I noticed that the cause was _handle_canceled_requests:

I added some logging in _handle_canceled_requests and found that self.executor_request_queue.get_canceled_req_ids_size() accumulates - never decreases.

TensorRT-LLM/tensorrt_llm/_torch/pyexecutor/py_executor.py

Lines 1913 to 1936 in e2f69c5

	def _handle_canceled_requests(self):
	if self.executor_request_queue.get_canceled_req_ids_size() == 0:
	return
	# Remove cancel request in the waiting queue
	self.executor_request_queue.update_waiting_queue()
	for request in self.active_requests:
	req_id = request.py_request_id if not request.is_child else request.parent_request_id
	if req_id not in self.executor_request_queue.get_canceled_req_ids():
	continue
	is_cancelled = self._try_cancel_request(request)
	if is_cancelled:
	# Mark requests as finished, then, we reuse all existing code
	# to clean up the KV cache resources.
	request.finish_by_reason(FinishReason.CANCELLED)
	request.decoding_iter = request.py_decoding_iter
	if self.enable_attention_dp:
	# TODO: revisit the cancel logic of attention dp
	# When enable attention dp, each rank does not have full copy of requests
	# so we need to remove the cancel requests not in the local rank
	self.executor_request_queue.clear_canceled_req_ids()

Adding enable_attention_dp:true to the extra_llm_api_options yaml solves the issue for me.

So I guess the self.executor_request_queue.clear_canceled_req_ids() gating is incorrect, or maybe what I'm observing here is just a symptom of a different underlying issue.

This is what I'm running, using 4xB200 and nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc0:

cat >/root/data/trtllm-config.yml<<EOF
kv_cache_config:
  enable_block_reuse: true
  dtype: fp8
  free_gpu_memory_fraction: 0.88
  host_cache_size: 60000000000
EOF
trtllm-serve /root/data/nvidia/DeepSeek-R1-FP4 --backend pytorch --max_seq_len 9216 --max_batch_size 4096 --max_num_tokens 16384 --tp_size 4 --trust_remote_code --extra_llm_api_options /root/data/trtllm-config.yml

[josephrocca]

This issue definitely occurs much faster if many client requests are aborted before generation completes. And after a while it seems to cause a lot of these "storeContextBlocks: Can not find sequence for request" warnings (with request IDs that seem too high?):

INFO:     172.17.0.1:17708 - "POST /v1/completions HTTP/1.1" 200 OK
[10/03/2025-15:12:38] [TRT-LLM] [RANK 0] [W] Request of client_id 29633 is finished, cannot abort it.
INFO:     172.17.0.1:17598 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.17.0.1:33646 - "POST /v1/completions HTTP/1.1" 200 OK
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33730
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33730
[10/03/2025-15:12:39] [TRT-LLM] [RANK 0] [W] Request of client_id 29636 is finished, cannot abort it.
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33730
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33730
INFO:     172.17.0.1:33646 - "POST /v1/completions HTTP/1.1" 200 OK
[10/03/2025-15:12:39] [TRT-LLM] [RANK 0] [W] Request of client_id 29635 is finished, cannot abort it.
INFO:     172.17.0.1:17402 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.17.0.1:17712 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.17.0.1:17392 - "POST /v1/completions HTTP/1.1" 200 OK
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33732
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33732
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33732
[TensorRT-LLM][WARNING] [kv cache manager] storeContextBlocks: Can not find sequence for request 33732

Also, generated outputs start to become gibberish after a while, but this occurs even with enable_attention_dp:true, so I think the "cancelled request accumulation" issue may actually be a symtom of a deeper issue (maybe race condition?) with incorrect handling of request cancellation when the client aborts.

Note that simply handling asyncio.CancelledError correctly at the openai_server.py level (which, as of writing, is not merged) does not fix this issue, so I guess the actual issue is downstream of that promise.abort() call.

[karljang]

Hi @pcastonguay , Could you take a moment to check above comments? I believe it shares some common ground with what your PR #8114 addresses, hopefully~

[QiJune]

Hi @josephrocca , could you please try this fix #8207? Thanks

[josephrocca]

@QiJune That did solve the accumulation, but I had to change it to .remove(req_id) because the .erase method did not exist (IIUC, self.executor_request_queue.canceled_req_ids is a regular python list). Thanks!

But I'm wondering whether there is a deeper underlying bug here, since, I start to get "messed up" outputs after a while (even after applying your fix). I can open a separate issue about that if you're pretty sure this is unrelated my "messed up outputs" problem.

[QiJune]

@josephrocca thanks for the verification. Please open a seperate issue for the unreasonable outputs.

[josephrocca]

Okay, thanks!

[josephrocca]

@QiJune I've just discovered that the cancelled requests still accumulate, but just slower. I'm using nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc0. I'll reopen this if I can confirm the bug still exists in the latest docker image - I haven't tested that yet.