Skip to content

Conversation

@RuBing-Yang
Copy link
Collaborator

@RuBing-Yang RuBing-Yang commented Nov 11, 2025

TODOs:

  • 重构Trainer/DatasetBuilder/TargetWrapper仓库,根据online/offline LLM/VLM创建
  • Qwen3-VL HF Online训练
  • Qwen3-VL HF Offline训练
  • 支持mrope

AI Summary:

This pull request introduces support for Vision-Language Model (VLM) data handling and refactors trainer and data collator imports for better modularity. The main changes include adding new VLM-specific data collators, updating chat template types to support VLM, introducing new model configuration files, and cleaning up unused imports related to trainers and data collators.

Vision-Language Model (VLM) Support:

  • Added VLMDataCollatorWithPadding and supporting tensor padding functions to data_utils.py for handling VLM-specific batch data, including images and videos. [1] [2] [3]
  • Updated chat template handling to include a new QWEN3_VL type and its mapping, enabling VLM-specific system prompts and headers. [1] [2]
  • Added new model configuration files for VLM models: qwen3-vl-4b-eagle3.json and qwen3-vl-4b-eagle3-mrope.json. [1] [2]

Trainer and Data Collator Refactoring:

  • Removed unused imports and references to OfflineEagle3Trainer, OnlineEagle3Trainer, and DataCollatorWithPadding in __init__.py files, replacing them with Eagle3TrainerFactory for cleaner and more maintainable code. [1] [2] [3] [4] [5]

Dataset Builder Modularization:

  • Added a new dataset_builder module with factory and builder classes for both online and offline LLM/VLM datasets, improving extensibility and code organization.

@RuBing-Yang RuBing-Yang reopened this Nov 24, 2025
@liusong1222 liusong1222 self-requested a review November 24, 2025 06:33
liusong1222
liusong1222 previously approved these changes Nov 24, 2025
@RuBing-Yang RuBing-Yang changed the title [WIP] Support VLM Eagle3 Training Support VLM Eagle3 Training Nov 24, 2025
@RuBing-Yang RuBing-Yang merged commit 7382a63 into Tencent:main Nov 24, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants