Support GPTQ for LLMs INT4 quantization

[Yichangfa]

I am trying to quantize Qwen3-VL (a multimodal LLM) to INT4 using AIMET. Due to the hardware (Snapdragon 8295), I can only use per‑channel quantization. I first tried basic PTQ, but the accuracy was insufficient. Then I used SpinQuant, which improved accuracy, but it is still not enough. I read the SpinQuant paper, which states that after applying the rotation, GPTQ is used for quantization. However, AIMET's implementation does not seem to follow this approach.

[hitarthmehta]

Hi Yichangfa Thank you for reaching out. SpinQuant in AIMET targets activation quantization - the Hadamard rotations smooth activation outliers, not weights. For weight quantization at INT4, AIMET provides:

1. SeqMSE - minimizes per-layer output MSE by optimizing weight rounding
2. AdaScale - learns per-channel weight scales

The recommended approach is SpinQuant + AdaScale or SpinQuant + SeqMSE, which together cover what SpinQuant + GPTQ achieves in the paper. For Qwen3-VL, we'd suggest starting with SpinQuant + AdaScale. Please let us know if you have further questions.

[Yichangfa]

Thank you for the detailed explanation regarding SpinQuant's focus on activation outliers and the recommendation to combine it with AdaScale or SeqMSE for weight quantization.

I am currently using AIMET v2.18.0.

• The SpinQuant + SeqMSE combination, on its own, does not achieve the required INT4 accuracy for Qwen3-VL.
• When I attempt your recommended SpinQuant + AdaScale approach, the process fails with an error.

The error seems to be related to the fact that my model undergoes a model adaptation step before quantization. It appears that AdaScale cannot correctly recognize or handle the model's state after this adaptation.

Could you please advise:

1. Are there any known limitations or specific requirements for using AdaScale with models that have been adapted in a certain way?
2. Are there any changes or workarounds I could try to make AdaScale work on my adapted Qwen3-VL model within AIMET v2.18.0?
3. If the above is not feasible, what would be the recommended path forward to achieve acceptable INT4 accuracy for an adapted Qwen3-VL model using AIMET v2.18.0?

Thank you for your guidance.