[init]support gptq grpo in colocate mode #5569

ItGirls · 2025-08-27T13:08:56Z

PR type

Bug Fix
New Feature
Document Updates
More Models or Datasets Support

PR information

support gtpq model in GRPO training mentioned by #3949 , #5372

using vllm colocate mode.

Experiment results

Paste your experiment result here(if needed).

gemini-code-assist

Summary of Changes

Hello @ItGirls, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement by enabling the training of GPTQ quantized models with GRPO using vLLM in colocate mode. The core problem addressed is the incompatibility between GPTQ models and LoRA merging within the vLLM acceleration framework. The solution involves a revised approach to managing LoRA parameters, where they are saved locally and then dynamically provided to the vLLM engine during inference via a new AdapterRequest mechanism. This change allows for efficient training of quantized models without the previous merging limitations, improving the overall flexibility and capability of the training pipeline.

Highlights

GPTQ GRPO Training Support: This PR enables the training of GPTQ quantized models using GRPO (Generative Reinforcement Learning with Policy Optimization) within vLLM's colocate mode. This resolves a critical limitation where GPTQ models previously could not be merged with LoRA when vLLM acceleration was active, leading to training errors.
Enhanced LoRA Parameter Handling: A new mechanism has been implemented to handle LoRA adapter parameters. Instead of attempting to merge LoRA directly into the quantized model, the adapter weights are now saved locally and then passed to the vLLM engine via an AdapterRequest during inference. This allows for dynamic loading of LoRA adapters without requiring a full model merge.
New Configuration Option: local_adapter_path: A new command-line argument, local_adapter_path, has been introduced. This argument specifies a temporary directory where LoRA adapter weights are stored, facilitating their transfer and use by the vLLM engine in colocate mode.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for training GPTQ-quantized models with GRPO in colocate mode. The core idea is to save LoRA adapters to a local path instead of merging them, which is a good solution for quantized models. The implementation looks solid. My review includes suggestions to fix a typo in the documentation, remove unused code, improve code robustness by using os.makedirs, and fix some typos in comments and log messages.

gemini-code-assist · 2025-08-27T13:11:18Z

docs/source/BestPractices/GPTQ量化模型GRPO训练.md

@@ -0,0 +1,88 @@
+# 采用Colocate模式进行GTPQ量化模型的GRPO训练


There's a typo in the title. GTPQ should be GPTQ.

Suggested change

# 采用Colocate模式进行GTPQ量化模型的GRPO训练

# 采用Colocate模式进行GPTQ量化模型的GRPO训练

Thanks for suggestion, I already changed the file.

gemini-code-assist · 2025-08-27T13:11:19Z

swift/trainers/rlhf_trainer/grpo_trainer.py

@@ -107,6 +110,9 @@ def __init__(self,
        from swift.trainers.rlhf_arguments import GRPOConfig
        args: GRPOConfig = kwargs['args']
        self.args = args
+        self.local_adapter_path = getattr(args, 'local_adapter_path', None)
+        self.enable_lora = True if self.local_adapter_path else False
+        self.update_adapter_count = 0


The update_adapter_count attribute is initialized here, incremented in _move_model_to_vllm (line 588), but it's not used anywhere. The code that would use it is commented out in _engine_infer. This unused variable and its related logic should be removed to avoid dead code.

Thanks for suggestion. I removed the unused variable.

gemini-code-assist · 2025-08-27T13:11:19Z

swift/trainers/rlhf_trainer/grpo_trainer.py

+                                #　delete exists files
+                                shutil.rmtree(self.local_adapter_path)
+                                logger.info(f"step:{self.state.global_step}，deleta previous lora")
+
+                            os.mkdir(self.local_adapter_path)
+                            self.update_adapter_count+=1


There are a couple of issues in this block:

Typos in the comment on line 583 and the log message on line 585.

os.mkdir on line 587 is not as robust as os.makedirs, which can create parent directories if they don't exist.

update_adapter_count is incremented on line 588 but is unused and should be removed.

Suggested change

#　delete exists files

shutil.rmtree(self.local_adapter_path)

logger.info(f"step:{self.state.global_step}，deleta previous lora")

os.mkdir(self.local_adapter_path)

self.update_adapter_count+=1

# delete existing files

shutil.rmtree(self.local_adapter_path)

logger.info(f"step:{self.state.global_step}, deleted previous lora")

os.makedirs(self.local_adapter_path)

Thanks a lot. I fixed the misspelled word and replaced 'mkdir' with 'makedirs', also deleted the unused variable.

gemini-code-assist · 2025-08-27T13:11:19Z

swift/trainers/rlhf_trainer/grpo_trainer.py

-                res = self.engine.infer(infer_requests, request_config, use_tqdm=use_tqdm)
+                # use adapter_request path lora to vllm engine
+                if self.local_adapter_path:
+                    assert os.path.exists(self.local_adapter_path)


Using assert for path existence checks can be risky as assertions can be disabled. It's better to raise a more descriptive error if the path does not exist, for example by replacing the assert with if not os.path.exists(self.local_adapter_path): raise FileNotFoundError(...).

Thanks, I followed the modification instructions.

…lmin colocate mode

hjh0119 · 2025-08-29T06:43:29Z

thanks for the contributing! Is this PR ready for review?

ItGirls · 2025-08-29T10:40:30Z

Yes, of course. 发自我的iPhone

…

------------------ Original ------------------ From: jinghanhu ***@***.***> Date: Fri,Aug 29,2025 2:43 PM To: modelscope/ms-swift ***@***.***> Cc: ItGirls ***@***.***>, Mention ***@***.***> Subject: Re: [modelscope/ms-swift] [init]support gptq grpo in colocate mode(PR #5569) hjh0119 left a comment (modelscope/ms-swift#5569) thanks for the contributing! Is this PR ready for review? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

ItGirls · 2025-08-29T10:41:30Z

thanks for the contributing! Is this PR ready for review?

yes, of course.It's ready for review.

[init]support gptq grpo in colocate mode

08ed4b5

gemini-code-assist bot reviewed Aug 27, 2025

View reviewed changes

[fix]enable the training of a GPTQ quantized model with GRPO using vl…

4dee43e

…lmin colocate mode

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[init]support gptq grpo in colocate mode #5569

[init]support gptq grpo in colocate mode #5569

Uh oh!

ItGirls commented Aug 27, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Aug 27, 2025

Uh oh!

ItGirls Aug 27, 2025

Uh oh!

gemini-code-assist bot Aug 27, 2025

Uh oh!

ItGirls Aug 27, 2025

Uh oh!

gemini-code-assist bot Aug 27, 2025

Uh oh!

ItGirls Aug 27, 2025

Uh oh!

gemini-code-assist bot Aug 27, 2025

Uh oh!

ItGirls Aug 27, 2025

Uh oh!

hjh0119 commented Aug 29, 2025

Uh oh!

ItGirls commented Aug 29, 2025 via email

Uh oh!

ItGirls commented Aug 29, 2025

Uh oh!

Uh oh!

		@@ -0,0 +1,88 @@
		# 采用Colocate模式进行GTPQ量化模型的GRPO训练

	# 采用Colocate模式进行GTPQ量化模型的GRPO训练
	# 采用Colocate模式进行GPTQ量化模型的GRPO训练

[init]support gptq grpo in colocate mode #5569

Are you sure you want to change the base?

[init]support gptq grpo in colocate mode #5569

Uh oh!

Conversation

ItGirls commented Aug 27, 2025

PR type

PR information

Experiment results

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

ItGirls Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

ItGirls Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

ItGirls Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

ItGirls Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

hjh0119 commented Aug 29, 2025

Uh oh!

ItGirls commented Aug 29, 2025 via email

Uh oh!

ItGirls commented Aug 29, 2025

Uh oh!

Uh oh!