[Qwen3 MOE] FP8 Per-Tensor Quantization Causes Dimension Mismatch in vLLM Inference

[wangwenmingaa]

The llmcompressor community doesn't provide dynamic FP8 per tensor quantization method. I added the following quantization scheme in the quant_scheme.py file of the compressor_tensors library:

FP8_TENSOR = dict(
    weights=QuantizationArgs(
        num_bits=8,
        type=QuantizationType.FLOAT,
        strategy=QuantizationStrategy.TENSOR,
        symmetric=True,
        dynamic=False,
    ),
    input_activations=QuantizationArgs(
        num_bits=8,
        type=QuantizationType.FLOAT,
        strategy=QuantizationStrategy.TOKEN,
        symmetric=True,
        dynamic=True,
        observer=None,
    ),
)

Using this quantization scheme, I was able to quantize the Qwen3 MOE model, but it failed during inference on vLLM. The key error message is as follows:

quant_config.per_out_ch_quant == (quant_config.w1_scale.size(1) == w1_q.size(1))
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

My questions are:

1. Is the quantization scheme I added for dynamic FP8 per tensor quantization correct?
2. Why can't the quantized MOE model run inference on the vLLM framework?

[HDCharles]

I don't think vllm has a kernel for that. Your issue makes it sound like this is an issue with Qwen3 MoE, does it work with other model architectures?

The ones that tend to be well supported can be found here: https://github.com/vllm-project/compressed-tensors/blob/main/src/compressed_tensors/quantization/quant_scheme.py#L288

as for whether what you have is correct, if you are asking if its doing what you think, i'd say yes, if you are asking whether it will result in a model with ops and kernels supported by vllm, i think no.

but as you mentioned there's per-tensor static fp8 and per-channel dynamic fp8 but not per-tensor dynamic fp8. The reason is likely because its kind of an unhappy medium, if you want fast, per-tensor static quant is the way to go, if you want more accuracy, per-channel or per-block dynamic quant make more sense.