-
Notifications
You must be signed in to change notification settings - Fork 390
[ci] init e2e test struct #1269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
| repo_org: | ||
| required: false | ||
| description: 'Tested repository organization name. Default is InternLM' | ||
| type: string | ||
| default: 'InternLM/xtuner' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why this arguments is required?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for more flexible specification of test code
| - HF_DATASETS_OFFLINE=1 | ||
| - HF_HUB_OFFLINE=1 | ||
|
|
||
| case: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The hierarchy here seems unusual. From the original semantics, the xtuner repository should have the following test levels:
- Tasking: sft & pretrain & rl
- Training config: config path
- Resource: test resource
Xtuner's configuration files can determine the first two, while the last one is determined by clusterx's configuration. What do you think about adjusting the hierarchy like this:
case:
- task: <rl | sft | pretrain>
config: <config path>
assert_info: <...>
resources: [<resource_name, including env>]Then resources should contain test resources like gpu, npu, etc., with different names. In each case item, you can select which platforms and resource specifications to test on.
Co-authored-by: Mashiro <[email protected]>
Co-authored-by: Mashiro <[email protected]>
update preaction
export run_id
add clean and log
push config
Initializing the testing framework allows for quick addition of sft、rl and pretrain related test cases through configuration.
On the quality side, focus on the code development of the validation and regression points themselves(train、infer、eval and so on).