Why does TensorRT-LLM use output scaling for SmoothQuant while ModelOpt does not?

[junstar92]

I'm currently working on building a TensorRT-LLM engine from an LLM that was quantized using ModelOpt with SmoothQuant.

However, I've run into some difficulties because the SmoothQuant implementation in ModelOpt and TensorRT-LLM appears to differ slightly - especially regarding output scaling (scale_y).
While ModelOpt applies SmoothQuant without output scaling during inference, TensorRT-LLM expects it to be used as the input scale for SmoothQuantGemm plugin.

Because of this difference, the model quantized with ModelOpt cannot be directly used to build a TensorRT-LLM engine without additional modifications, leading to compatibility issues.

I would like to understand:

• Why does TensorRT-LLM require or benefit from applying output scaling, while ModelOpt does not?
• Is this mainly due to TensorRT-LLM's plugin kernel design, its need for tighter accuracy control, or other optimizations?
• What would be the recommended way to bridge this gap when converting a ModelOpt-quantized model for use with TensorRT-LLM?

Any technical explanation or best practice for addressing this difference would be greatly appreciated.

[juney-nvidia]

@yechank-nvidia Hi Yechan, can you help follow this ask from the community?

Thanks
June

[junstar92]

@juney-nvidia @yechank-nvidia
Thanks in advance, and thank you for all your great work on this project!
I was wondering if there have been any updates regarding this topic. I'm very interested in this feature and would appreciate any insights or progress, if available.

Thanks