Skip to content

Merge branch 'dev' into fix_mfsdp_random_hang #3237

Merge branch 'dev' into fix_mfsdp_random_hang

Merge branch 'dev' into fix_mfsdp_random_hang #3237

Triggered via push November 20, 2025 03:37
Status Success
Total duration 41m 12s
Artifacts 19

cicd-main.yml

on: push
is-not-external-contributor
9s
is-not-external-contributor
pre-flight  /  pre-flight
13s
pre-flight / pre-flight
cicd-wait-in-queue
2s
cicd-wait-in-queue
cicd-container-build
3m 33s
cicd-container-build
cicd-parse-unit-tests
7s
cicd-parse-unit-tests
Matrix: cicd-unit-tests-latest
cicd-parse-integration-tests
9s
cicd-parse-integration-tests
Matrix: cicd-integration-tests-latest
Nemo_CICD_Test
6s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
ko3n1g
approved Nov 20, 2025
test

Automatically approved by queue manager

Annotations

37 errors and 13 warnings
cicd-container-build
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_model_configs.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/data/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/data/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/pipeline_parallel/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/pipeline_parallel/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_fp8_param.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_fp8_param.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/moe/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/moe/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/distributed/fsdp/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/distributed/fsdp/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8 - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8 - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/data/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/pipeline_parallel/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/test_fp8_param.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/moe/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/distributed/fsdp/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.

Artifacts

Produced during runtime
Name Size Digest
coverage-unit-test-19524740229-01771440-4e3b-4885-a988-f1518b795397
122 KB
sha256:346d7ccfbe4a7f1b0100bd9c6a7cf93a7c80283b311a4117a0de3703b27e6111
coverage-unit-test-19524740229-133c8111-1498-41a0-b363-233d249d8365
119 KB
sha256:ba9548c616b0fbac103fd0c957676bafd92e6a383fd3e56a14f88324f53bceb0
coverage-unit-test-19524740229-23d38da0-86aa-4bbf-b9aa-2a806ea33046
116 KB
sha256:31c6636442a276a9ae3ab79c739be297dfbe62b73e038c150ddf76b002847991
coverage-unit-test-19524740229-4e8de499-3009-4c3a-abb4-7cdffd082976
121 KB
sha256:44cc01198f2e8fbb522ac7cfe000ec6640f44099945fc2c34fa4aad029d1ba64
coverage-unit-test-19524740229-579e2ffe-9bf8-44bb-91a4-7c011790da82
129 KB
sha256:9d2f3dfe1052f9002776b24e3d3d6d35dadbcaad33fa82ea8ae5de0569f55a46
coverage-unit-test-19524740229-5d2121db-e47a-4dc3-809e-9838de0a3bde
120 KB
sha256:7838790fbda6d5473f55853df5b08e76975b9efe106a650d4158f2a42abe56e8
coverage-unit-test-19524740229-5f7a4e7e-989b-4e59-ae21-738919d5c8b6
120 KB
sha256:af122d93604008e1900340ed76e4abf60555c44d948cbb47c8a702f1b067e2c1
coverage-unit-test-19524740229-678c5217-d16c-423e-8378-b18c6f37ba2c
121 KB
sha256:24117fe678d50beca661671875894f750cbc474af2cee7decdd7295594e7731f
coverage-unit-test-19524740229-92ca8ba2-153d-4751-b464-33195caff05b
119 KB
sha256:d70c5c3ac8ce83322916cc0e54fbdf546af8c95f925676cb2ba3d8065f72e953
coverage-unit-test-19524740229-a15cd824-eec0-422f-b9cf-10b117c17379
121 KB
sha256:71ad4f2cb17419b408be727a130baaecd90a875064df0f1fe0f689bc53dd329e
coverage-unit-test-19524740229-c9eecfec-4aa7-404b-a6ab-f79cb6f97e42
118 KB
sha256:5f3674cc9d73b9bc37f4e32b64c065973c9681a2834209ed3f5194edd34a8e8c
coverage-unit-test-19524740229-cfb9cb0b-9ef7-4366-a835-3fb3fb251c3c
119 KB
sha256:603ea44dd4dabdceb6c20e286b6090592cdeed966d214f5361189fde495ceeef
coverage-unit-test-19524740229-dc5342d8-bbd0-492c-92a2-0a6d40a3942a
124 KB
sha256:f18e58154abe05e44ff1eeeb9dc2736dc6469a93c5d4a2cd9cb9ab3f8060ec34
coverage-unit-test-aggregated
28 KB
sha256:38ba6d8890908937dc01aa1219028e0bbcb5386fedecd178b5f5cc01d187b02c
logs-gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer-19524740229-54549c6f-7661-42e1-a945-296bc7f8e3e8
41.2 KB
sha256:a54c0a1518afd06019ea8099a05e902cdded1e0d23e0a865fb28818d9048bdad
logs-gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed-19524740229-e59eb9b1-7b50-4b7d-9d2b-85103bdc3fe1
35.4 KB
sha256:1ba0d6320fcbe42771c28c5903d0b7dc8cc1ee5a3e9048c0c29fd16cd479e92a
logs-gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8-19524740229-f46cfc39-565b-490d-a643-209f94c805c6
35.4 KB
sha256:4bb7140a678d24a6eb7ab63a39e358fd72ca821f853b16e7a651394cc9b51a83
logs-gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather-19524740229-74c3c0cd-82e5-40c5-b9ce-f1be94812c63
82 KB
sha256:ff2a4cb203fac6d7f93d513ad25d9bca5801a44231bbc41ae17fc78b0cbc823e
logs-gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer-19524740229-4f91ba49-e5dc-4887-835a-eff1ca8a2562
83 KB
sha256:65aaed2bcc2fc0c227ffc6455a9654a6f4bf2c96d934d61973a49114b6e03aa7