Skip to content

CICD Megatron-LM

CICD Megatron-LM #3225

Triggered via schedule November 20, 2025 00:03
Status Success
Total duration 20m 52s
Artifacts 18

cicd-main.yml

on: schedule
is-not-external-contributor
6s
is-not-external-contributor
pre-flight  /  pre-flight
9s
pre-flight / pre-flight
linting
9s
linting
cicd-wait-in-queue
0s
cicd-wait-in-queue
cicd-container-build
45s
cicd-container-build
cicd-parse-unit-tests
6s
cicd-parse-unit-tests
Matrix: cicd-unit-tests-latest
cicd-parse-integration-tests
10s
cicd-parse-integration-tests
Matrix: cicd-integration-tests-latest
Nemo_CICD_Test
9s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
Fit to window
Zoom out
Zoom in

Annotations

35 errors and 13 warnings
cicd-container-build
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_model_configs.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/data/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/data/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/pipeline_parallel/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/pipeline_parallel/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_fp8_param.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_fp8_param.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/distributed/fsdp/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/distributed/fsdp/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/moe/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/moe/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/data/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/pipeline_parallel/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/test_fp8_param.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/distributed/fsdp/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/moe/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.

Artifacts

Produced during runtime
Name Size Digest
coverage-unit-test-19520453108-11e25294-8337-429c-bd57-eff40f42cb39
112 KB
sha256:759722a0c1007347726b3e6ff15f77ee70243594ed4ced6bbce58d24ec52396b
coverage-unit-test-19520453108-20d058ac-5bbc-485e-a3db-77592745a3af
116 KB
sha256:148e067abe47fbead569d3e641c06ed4c9a501a5f5242f6cc219b05a3cf165e4
coverage-unit-test-19520453108-245ac957-fae5-47e9-b3ef-c83c31b9125f
119 KB
sha256:8d64bee9b065caf06304a92f4d41464dfa4e2723a6fabfec1f16bd9437fa200a
coverage-unit-test-19520453108-3c635c42-43f6-4ec2-bfd9-2c5d179670d1
115 KB
sha256:fc1cd4a0132bb8c829adc97699bbebb7c5e56bd05f046e214c9f00a7ec2b7aa0
coverage-unit-test-19520453108-4274412f-dbc5-4276-9fb3-d22e791ead23
118 KB
sha256:5f0a5645dd7d247b64ae92aad80792f17faa930179ec72c133a148b66a704d09
coverage-unit-test-19520453108-8171ab04-99ef-4359-9b6f-b7dcbdb03a8f
114 KB
sha256:11169e52fc2ec7f52a50f35708513a7d58f034db4041ff02fc4542cf813820b3
coverage-unit-test-19520453108-82b3c878-00c7-43b0-843c-771f833e3abb
117 KB
sha256:529c7353d88761e89178b40ccdffa65ed4dc6e6e036d7db45a8e0866d0b51ac3
coverage-unit-test-19520453108-94725c5b-5b73-4f4c-847c-3126a2a6516d
115 KB
sha256:4ccbda599bd20190740b6c8de1686de57803c79f4c453adbb2611b8ba9bcf86a
coverage-unit-test-19520453108-9b363f3f-6fe8-4682-9d1b-194ac1843754
117 KB
sha256:7c022602e9f5c8818c9cc8daa3e03cb313f8f93ae6875fc6cbf1ceee83318c98
coverage-unit-test-19520453108-b9569a6a-1685-4a63-945b-674018d6c4e5
115 KB
sha256:f2c1394812810ae45375498e6bca04850e64593e18f40219208c9fe0637581b5
coverage-unit-test-19520453108-d03d66ae-05a3-40ef-bcf7-54b820747b38
117 KB
sha256:1fa4e95f67e84ab404d83f6ed716a942b859cdb487bb1f2bd8a914f5977d6b3d
coverage-unit-test-19520453108-e3883535-4080-4965-af09-6964926ca31c
116 KB
sha256:7fa74fc4a0568e14ebb1b0537ce0b9de8265d98d2f30bad02e4304a3f369dca6
coverage-unit-test-19520453108-e65a1d00-e052-4c76-9bd0-392f2aaf10eb
125 KB
sha256:208d0cfaa237f1653d33bbfd88601ff58b44cbd228673d9c5343a7307610741d
coverage-unit-test-aggregated
27.2 KB
sha256:4df27e0ee9f427e0265a595ee58ddacd13781c47fc1dca0bfdd2159e67e0b318
logs-gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer-19520453108-87e0983f-5a5f-4d20-b023-8f3219602f4a
39.5 KB
sha256:b0919a98eae00ae43da0bdc981e7baeb2d1fc43d06b37c16bd9254777ee3d4fb
logs-gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed-19520453108-fd163753-1385-445b-a040-c52146560973
35.9 KB
sha256:52e4eac8807e950e832491e952831304fb04d073a0cadd5810b3a8eb16bd092b
logs-gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather-19520453108-14204c4e-d800-461c-b496-a45c26ba89e2
79.8 KB
sha256:5f71b80f2f9db65b95950e2e8f4e252e8b8de2deadf45fcf59bd64e5aa195ef2
logs-gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer-19520453108-db6d24bf-2b1d-4d95-b230-49895a0b4fcd
80.7 KB
sha256:2b9dc48da9d59221cfeb7a1c1ae6bcc87d27477f6e70a6c4d011515345baf480