Skip to content

Commit d299b7a

Browse files
Fix auto deploy demo logprobs return issue.
Signed-off-by: Wangshanshan <[email protected]>
1 parent 4ec56fc commit d299b7a

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

tensorrt_llm/_torch/auto_deploy/shim/demollm.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -235,11 +235,11 @@ def _sample(
235235
logits_shape = logits.shape
236236
logits = logits.view(-1, logits_shape[-1]) # sampling_batch expects 2D logits
237237
if isinstance(sampling_params.top_k, int) and sampling_params.top_k > 1:
238-
idx_next, probs = top_k_sampling_batch(
238+
idx_next, probs, _ = top_k_sampling_batch(
239239
logits, top_k=sampling_params.top_k, temperature=1.0
240240
)
241241
else:
242-
idx_next, probs = greedy_search_sampling_batch(logits)
242+
idx_next, probs, _ = greedy_search_sampling_batch(logits)
243243
idx_next = idx_next.view(logits_shape[:-1])
244244
return idx_next, probs
245245

0 commit comments

Comments
 (0)