AWQ with custom model

[eghouti]

Does anyone knows how to run AWQ with a custom model. my model works perfectly in vllm but I could not apply AWQ on it

[HDCharles]

you should be able to follow the documentation and examples in:

https://github.com/vllm-project/llm-compressor/tree/99e231e16d7ef45e2fab67c4c77178900eb00f33/examples/awq

to get it working with an arbitrary model.

[eghouti]

thank you @HDCharles for your answer: I just realised that my model is so big and I cannot do the quantization on 1 GPU, I need to do it on multiple GPUs. Is there a way to do it (using for example pipline parallelism)?

[HDCharles]

We're actively working on a bunch of issues with AWQ memory and performance, especially for large models. I believe by default it tries to do model offloading to cpu. If you can provide a repro we'd be happy to take a look.

[eghouti]

I cannot really share the model. but it is a simple model that can be called using from_pretrained from huging-face. So I was wondering if I can run oneshot function from llmcompressor on multiple GPUs

[eghouti]

@HDCharles sorry for disturbing you again. I did some research and looks like the oneshot from: from llmcompressor import oneshot, does not support multi GPUs runing. Is there anotherway to run it avoiding cuda out of memory without doing it on CPU? Thanks for your help.

[HDCharles]

yeah multi-gpu isn't supported but you can see #7 here so its not like your model is expected to entirely fit in gpu before quantization.

for AWQ there's also:

There's

llm-compressor/src/llmcompressor/modifiers/awq/base.py

Line 110 in b77175d

:param offload_device: offload cached args to this device, which reduces memory

which can help.

Having said that if this was me I would first:

A) verify performance. You don't need an accurate model to verify performance for your use case. I would try either RTN or model-free and make sure everything is working there before going forward.

B) once it looks like the quantized model's performance looks good i would probably try GPTQ first, that tends to do better than AWQ and has lower memory usage. There is also a similar offloading option on top of the GPTQ reduced memory.

let me know if that helps

[eghouti]

yeah multi-gpu isn't supported but you can see #7 here so its not like your model is expected to entirely fit in gpu before quantization.

for AWQ there's also:

There's

llm-compressor/src/llmcompressor/modifiers/awq/base.py

Line 110 in b77175d

 :param offload_device: offload cached args to this device, which reduces memory 

which can help.

Having said that if this was me I would first:

A) verify performance. You don't need an accurate model to verify performance for your use case. I would try either RTN or model-free and make sure everything is working there before going forward.

B) once it looks like the quantized model's performance looks good i would probably try GPTQ first, that tends to do better than AWQ and has lower memory usage. There is also a similar offloading option on top of the GPTQ reduced memory.

let me know if that helps

thank you for your answer. I already tried the PTQ methods (like fp8 and W4A16). I want now to get better performance using on 4bits using GPTQ or AWQ.

[eghouti]

@HDCharles I added the offload as suggested as follows:

recipe_full = [
    AWQModifier(
        offload_device=torch.device("cpu"),
        ignore=["lm_head"],
        config_groups={
            "group_0": QuantizationScheme(
                targets=["Linear"],
                weights=QuantizationArgs(
                    num_bits=num_bits,
                    group_size=128,
                    symmetric=True,
                ),
            )
        }
    )
]

but still have cuda out of memory

[HDCharles]

thank you for your answer. I already tried the PTQ methods (like fp8 and W4A16). I want now to get better performance using on 4bits using GPTQ or AWQ.

GPTQ and AWQ wont change performance, it will only change accuracy.

Does GPTQ run into the same issue?

[eghouti]

thank you for your answer. I already tried the PTQ methods (like fp8 and W4A16). I want now to get better performance using on 4bits using GPTQ or AWQ.

GPTQ and AWQ wont change performance, it will only change accuracy.

Does GPTQ run into the same issue?

Yes GPTQ run into the same issue. Yes I want to improve accuray.