Triton runtime produces repetitive non-speech audio with a custom F5-TTS checkpoint (SWivid default works)

[spacearound404]

Environment

• GPU: NVIDIA GeForce RTX 4090 (24GB), CC 8.9
• OS/container: image built from Dockerfile.server in src/f5_tts/runtime/triton_trtllm
• Triton Server: 2.53.0
• TensorRT: 10.7.0
• TRT-LLM: 0.16.0
• PyTorch: 2.5.1 (per ONNX export log)
• Python: 3.12
• Additionally: installed cuda-python==12.6 (otherwise got ImportError: cannot import name 'cudart'; see reference)

Models

• Baseline model from HF: SWivid/F5-TTS — works correctly via Triton
  • https://huggingface.co/SWivid/F5-TTS/tree/main
• Custom model (Russian):
  • Checkpoint: Misha24-10/F5-TTS_RUSSIAN/F5TTS_v1_Base_v2/model_last.pt
  • https://huggingface.co/Misha24-10/F5-TTS_RUSSIAN/tree/main/F5TTS_v1_Base_v2

Expected behavior

• Same correct speech as local CLI/PyTorch backend, but via Triton runtime (TRT-LLM).

Actual behavior

• Triton server starts without errors and processes inference, but output is repetitive, identical sounds (no actual speech).
• Triton with the default SWivid model produces good audio.
• The same custom checkpoint works correctly outside Triton (local CLI).

Repro steps

1. Build image:

docker build . -f Dockerfile.server -t soar97/triton-f5-tts:24.12

2. Run container:

docker run --gpus all --rm -it --net host \
  -v /home/user/viksna/local-voice-ai-v2/f5-tts-assets:/assets \
  -v /home/user/viksna/local-voice-ai-v2/triton_models:/out \
  -v /home/user/viksna/F5-TTS:/F5-TTS \
  --shm-size=2g soar97/triton-f5-tts:24.12 bash

3. In container:

pip3 install cuda-python==12.6
cd /F5-TTS/src/f5_tts/runtime/triton_trtllm
# prepare and start:
bash run.sh 0 4 F5TTS_v1_Base_v2

4. Test request (HTTP):

python3 client_http.py \
  --reference-audio=/home/user/viksna/local-voice-ai-v2/f5-tts-assets/ref_max.wav \
  --reference-text="Солнце жёлтое. Море синее. Небо голубое. Космос чёрный. Кошка каричневая. Машина белая. Огонь оранжевый." \
  --target-text="Солнце жёлтое. Море синее. Небо голубое. Космос чёрный. Кошка каричневая. Машина белая. Огонь оранжевый." \
  --model-name="f5_tts"

Triton config highlights

• model_repo/f5_tts/config.pbtxt points to:
  • model_path: ./F5-TTS/F5TTS_v1_Base_v2/model_last.pt
  • vocab_file: ./F5-TTS/F5TTS_v1_Base/vocab.txt (I also tried replacing this with the exact Russian vocab.txt that works in my CLI; see below)
  • tllm_model_dir: ./f5_trt_llm_engine
  • vocoder: vocos
• Vocos engine builds and runs successfully.
• TRT engine builds without errors (many dtype warnings, but no failures).

What I tried

• Synchronized vocab.txt:
  • Removed defaults and used the exact same vocab.txt everywhere (Triton and local) that gives good results in the CLI.
  • Verified config.pbtxt actually points to that file.
• Increased diffusion steps nfe_steps to 32 — as expected, did not fix the “nonsense audio” (it affects quality/stability but not token correctness).
• Ensured model_path uses model_last.pt (EMA), not model_last_inference.safetensors.
• Used the same reference audio/text that works in the local CLI.

Logs/artifacts

• TRT-LLM build succeeds; many warnings like:
  • Parameter dtype is None, using default dtype: DataType.FLOAT ...
  • Provided but not required tensors: {... text_embed.* ...}
• Triton:
  • f5_tts and vocoder become READY, HTTP on 8000, gRPC on 8001, metrics 8002.
• Inference output: repetitive, identical sounds, no intelligible speech.

Why I suspect TRT-LLM conversion

• The exact same checkpoint + vocab + references works fine in local CLI/PyTorch, but produces non-speech via TRT.
• SWivid default model works via Triton (so request/IO pipeline is fine).
• Server-side uses the standard mel (Vocos: 24k, 100 mels, hop 256, win 1024) and tokenization (concat ref+target using vocab.txt). If the vocab is correct and CLI works with it, but only TRT produces nonsense, conversion or engine may be at fault.

Hypotheses

• Incompatibility or edge case in scripts/convert_checkpoint.py (name mapping, EMA handling, dtype, head scaling etc.) for this custom checkpoint, leading to an invalid DiT engine that outputs garbage mels.
• Differences in tokenizer/vocab assumptions for third-party fine-tuned checkpoints (e.g., symbol set/ordering), even though local PyTorch inference succeeds.