[Bug]: When quantifying Qwe3-Next-80B-A3B using w8a8-int8, some parameters become random parameters

[dingjingzhen-bot]

⚙️ Your current environment

The output of python collect_env.py

### Environment Information ###
Python Version: `3.10`
llm-compressor Version: `0.8.1`
compressed-tensors Version: `0.12.2`
transformers Version: `4.57.1`
torch Version: `2.8.0`
CUDA Devices: `A100 GPU`
AMD Devices: `None`

🐛 Describe the bug

from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "./Qwen3-Next-80B-A3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto",trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID,trust_remote_code=True)
from datasets import load_dataset

NUM_CALIBRATION_SAMPLES=256
MAX_SEQUENCE_LENGTH=2048

# Load dataset.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)

# Preprocess the data into the format the model is trained with.
def preprocess(example):
    return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False,)}
ds = ds.map(preprocess)

# Tokenize the data (be careful with bos tokens - we need add_special_tokens=False since the chat_template already added it).
def tokenize(sample):
    return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
ds = ds.map(tokenize, remove_columns=ds.column_names)
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier

# Configure the quantization algorithms to run.
recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$","re:.*linear_attn.*"]),
]

# Apply quantization.
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save to disk compressed.
# SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-W8A8-Dynamic-Per-Token"
SAVE_DIR = "./Qwen3-Next-80B-A3B-Instruct-w8a8"

model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

Some parameters have become random parameters, such as input_layernorm, etc.
If I set the SmoothQuantModifier(smoothing_strength=0.8),
It seems that deleting it won't print the parameters of the quantized norm, but it's extremely slow

🛠️ Steps to reproduce

No response

[HDCharles]

what makes you think the parameter is random? is the output garbled?

[dingjingzhen-bot]

what makes you think the parameter is random? is the output garbled?

Yes, and I printed out the input_layernorm.weight in safetensor, which is different from the original model of bf16

[HDCharles]

You're applying smoothquant though, I believe by default that modifies those layernorms when it applies the equalization.

Do the same issues arise if you do only 1 of those 2 techniques?

i.e.

1. if you only apply smoothquant and no quantization, is the output garbled?
2. if you only apply quantization without smoothquant, does that work?

[dingjingzhen-bot]

You're applying smoothquant though, I believe by default that modifies those layernorms when it applies the equalization.

Do the same issues arise if you do only 1 of those 2 techniques?

i.e.

1. if you only apply smoothquant and no quantization, is the output garbled?
2. if you only apply quantization without smoothquant, does that work?

Do you mean that as long as smoothquant is specified, the parameters of Layernorm will be modified? Because I think that normal quantification will not change these parameters. The quantified model also yields incorrect results when used for reasoning. Regarding the specific experiment you mentioned for the first point, I'm currently working on the second one, but it's very slow and might take more than 10 hours.

[HDCharles]

smoothquant works by looking at activations and scaling them so quantization will be more accurate i.e.

X.W = (X*s).((1/s)*W)

normal linear -> linear with rescaling (the scales would cancel out)

so your quantized weight gets altered based on this scaling but for the other side where you modify the activation, you either add an additional rescale op into your model (which slows things down), or if there's a layernorm before the linear, then you can alter the layernorm weight so it handles the rescaling at the same time as the layernorm. So yeah its expected that the layernorms get rescaled during smoothquant.

In general i would recommend that you use easier/quicker-start techniques as a proof of concept before doing a 10 hour GPTQ run. You can just do round to nearest and see if that's at all reasonable and if that works with your model and if it looks ok but could be more accurate then do GPTQ.

For the two options i outlined, option 1 will work if you can run a non-quantized model. Ideally this would see if smoothquant is working properly with your model. option 2 will determine whether there's an issue with GPTQ but its possible that the issue is quantization in general or the specific GPTQ algorithm, doing rount to nearest would allow us to determine whether quantization (or some specific type) works with your model at all.

[dingjingzhen-bot]

@HDCharles

I've identified the issue. When I used the above script to generate a quantization model and performed validation on safetensor, I found that there were a large number of scale=0 or Nans, which led to the situation where hidden_state had Nans during inference, resulting in incorrect outcomes.

import torch
from safetensors import safe_open
import sys
import argparse

def check_safetensors(file_path, check_zero=True):
    print(f"Scanning file: {file_path} ...")
    
    # Statistics
    nan_keys = []
    inf_keys = []
    zero_keys = []
    
    # Read tensors one by one using safe_open to avoid OOM by loading everything at once
    try:
        with safe_open(file_path, framework="pt", device="cpu") as f:
            for key in f.keys():
                tensor = f.get_tensor(key)
                
                # 1. Check for NaN
                if torch.isnan(tensor).any():
                    print(f"[NaN Found] Key: {key} | Shape: {tensor.shape}")
                    nan_keys.append(key)
                    continue # If NaN is found, usually no need to check others
                
                # 2. Check for Inf (Infinity)
                if torch.isinf(tensor).any():
                    print(f"[Inf Found] Key: {key} | Shape: {tensor.shape}")
                    inf_keys.append(key)
                    continue

                # 3. Check for 0 values (Zero)
                if check_zero:
                    if (tensor == 0).any():
                        # For Scale tensors, containing 0 is fatal
                        # For Weight tensors, containing 0 is normal (sparsity)
                        # We distinguish the printing here
                        if "scale" in key or "inv" in key:
                            print(f"[ALERT: Zero in Scale] Key: {key} | Zero count: {(tensor==0).sum().item()}")
                            zero_keys.append(key)
                        else:
                            # If all elements are 0, there is definitely a problem
                            if (tensor == 0).all():
                                print(f"[All Zeros Found] Key: {key} | Shape: {tensor.shape}")
                                zero_keys.append(key)
                            # If only partially 0, it might be too noisy for normal weights, 
                            # so we can choose not to print or only print statistics.
                            # For debugging purposes, you can uncomment the line below:
                            # print(f"[Zeros Found] Key: {key}") 
                            pass

    except FileNotFoundError:
        print(f"Error: File not found {file_path}")
        return

    print("-" * 50)
    print("Scan Summary:")
    print(f"Number of keys with NaN: {len(nan_keys)}")
    print(f"Number of keys with Inf: {len(inf_keys)}")
    if check_zero:
        print(f"Number of keys with Zeros (Scale/All-Zeros): {len(zero_keys)}")

    if len(nan_keys) > 0:
        print("\nDetailed NaN List:")
        for k in nan_keys: print(f"  - {k}")

if __name__ == "__main__":
    # Usage: python check_model.py path/to/model.safetensors
    if len(sys.argv) < 2:
        print("Usage: python check_model.py <path_to_safetensors_file>")
    else:
        check_safetensors(sys.argv[1])