CI: Add gptqmodel to the CI

[BenjaminBossan]

This issue is to track the TODO from this comment. Once optimum 1.24.0 and transformers 4.49.0 are released, we should enable gptqmodel in the CI (and remove auto-gptq).

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[BenjaminBossan]

Not stale. New optimum and transformers releases are out, we should be good to proceed.

[Qubitium]

@BenjaminBossan Affirmative. To be clear, should we clear out all autogptq specific base code or just remove autogptq ci tests and replace with gptqmodel tests only?

[BenjaminBossan]

Let's go with the latter.

[BenjaminBossan]

@Qubitium @jiqing-feng We forgot a bit about this, I wonder if it's safe now to remove gptq from PEFT completely and fully rely on gptqmodel. What is your opinion?

Also, since ModelCloud/GPTQModel#1703 is merged, I wonder if we can remove auto-awq and also use gptqmodel instead.

[jiqing-feng]

I am ok with it. How about you @Qubitium ?

[Qubitium]

@BenjaminBossan @jiqing-feng Yes. I am in agreement with both AutoGPTQ and AutoAWQ deprecation. On the AutoAWQ front, not only have we merged AWQ support but we have sinced added two new AWQ kernels: AwqTorch (100% torch) and AwqTorchFused which is based on the work by @jiqing-feng for optimized cpu ops using Intel avx/xmx/amx instructions (we have since validated the kernel on AMD as well).

There may be Peft compat issues due to latest GPT-QModel refractors but I will work with Peft and @jiqing-feng to fix any issues regarding Peft regardes to kernel selection, training, etc so we have a stable pinned release Peft can count on.

[BenjaminBossan]

Fantastic, thanks you two. Let us know if we can help with anything.

[Qubitium]

@BenjaminBossan @jiqing-feng Work has begun on the autogptq and autoawq deprecation.

#2917
huggingface/transformers#41567

I will ping when the PRs is ready for review and CI are passing.