Speculative Decoding

[AakashKumarNain]

How would you like to use ModelOpt

I looked at the documentation for speculative decoding, and there are a few things that I couldn't find in the documentation.

1. The support matrix does not list any model from the Kimi family, yet I see a drafter for K2 trained by Nvidia. Are Kimi models like K2.5 supported or not? If they are supported, can we please update the documentation and usage accordingly?

2. The advanced usage section showcases that we can use a model served using vllm server for generating the dataset. But for hidden state extraction, only TRT-LLM is supported, right?

3. The training section for the draft model focuses on HuggingFace, and I don't think that works for Kimi. In most case we would either generate the dataset or extract the hidden state offline for it and then train unless we have more resources. Can you please clarify the usage with an example?

Who can help?

• ?

System information

• Container used (if applicable): nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc8
• OS (e.g., Ubuntu 22.04, CentOS 7, Windows 10): ? Ubuntu 24.04
• CPU architecture (x86_64, aarch64): x86_64
• GPU name (e.g. H100, A100, L40S): H200
• GPU memory size: 140G
• Number of GPUs: 8
• Library versions (if applicable):
  • Python: 3.12
  • ModelOpt version or commit hash: 0.37
  • CUDA: 13.1
  • PyTorch: 2.9.1
  • Transformers: 4.57.3
  • TensorRT-LLM: 1.3.0rc8
  • TensorRT: 10.14.1

[ChenhanYu]

Triage Analysis

Field	Value
Classification	Docs
Severity	Medium
Complexity	Moderate
Auto-fixable	No

Summary

The speculative decoding documentation lacks clarity on Kimi model support, hidden state extraction capabilities, and offline training workflows. The user cannot find guidance for training draft models with Kimi K2.5 or other unsupported models using offline hidden state extraction.

Root Cause / Approach

The documentation focuses on HuggingFace-compatible models (like Llama) and does not address: (1) explicit support matrix for Kimi models despite having Kimi-specific code (kimik2_eagle_default_config), (2) limitations of hidden state extraction methods, and (3) practical offline training workflows for models not natively supported by HuggingFace training pipeline.

Suggested Fix

1. Update support matrix in README to explicitly list Kimi models (K2, K2.5) with appropriate notes. 2. Clarify in Advanced Usage section that TRT-LLM supports hidden state extraction while HF serves dataset generation. 3. Add a new section or example documenting offline training workflow for non-HuggingFace models, including: generating dataset offline, extracting hidden states with TRT-LLM, and training draft model without the base model in memory. 4. Provide an example configuration for Kimi models using the eagle_decoder_type: kimik2 setting.

Relevant Files

• docs/source/guides/5_speculative_decoding.rst
• examples/speculative_decoding/README.md
• examples/speculative_decoding/main.py
• modelopt/torch/speculative/config.py
• modelopt/torch/speculative/eagle/conversion.py

---

Auto-triaged by pensieve /magic-triage