Skip to content

CI

CI #5050

Triggered via schedule November 19, 2025 09:34
Status Failure
Total duration 2h 42m 53s
Artifacts 52

ci.yaml

on: schedule
metadata
5s
metadata
bump-manifest
16s
bump-manifest
Matrix: amd64 / test-distribution
Matrix: arm64 / test-distribution
amd64  /  ...  /  build-base
2m 34s
amd64 / build-base / build-base
arm64  /  ...  /  build-base
3m 30s
arm64 / build-base / build-base
amd64  /  ...  /  build-mpi-operator-compatible-base
2m 38s
amd64 / test-nccl / build-mpi-operator-compatible-base
amd64  /  ...  /  build-nccl-gke
2m 43s
amd64 / test-nccl / nccl-test-gke / build-nccl-gke
arm64  /  ...  /  build-mpi-operator-compatible-base
arm64 / test-nccl / build-mpi-operator-compatible-base
arm64  /  ...  /  build-nccl-gke
arm64 / test-nccl / nccl-test-gke / build-nccl-gke
Matrix: amd64 / test-jax-cutlass-h100 / jax-cutlass-test-h100
Matrix: amd64 / test-jax / run-unit-test
Matrix: amd64 / test-te-a100 / run-unit-test
Matrix: amd64 / test-te-h100 / te-test-h100
amd64  /  build-maxtext
9m 8s
amd64 / build-maxtext
amd64  /  build-torchax
7m 13s
amd64 / build-torchax
amd64  /  ...  /  launch-slurm-runner
1h 42m
amd64 / test-jax / runner / launch-slurm-runner
amd64  /  test-nsys-jax-eks
4m 11s
amd64 / test-nsys-jax-eks
amd64  /  ...  /  launch-slurm-runner
29m 47s
amd64 / test-te-a100 / runner / launch-slurm-runner
amd64  /  build-upstream-t5x
7m 4s
amd64 / build-upstream-t5x
amd64  /  build-axlearn
2m 6s
amd64 / build-axlearn
Matrix: amd64 / test-nsys-jax / run-unit-test
amd64  /  ...  /  launch-slurm-runner
1h 29m
amd64 / test-nsys-jax / runner / launch-slurm-runner
Matrix: amd64 / test-nccl / nccl-test
Matrix: amd64 / test-nccl / nccl-test-gke / nccl-gke
Matrix: arm64 / test-jax-cutlass-h100 / jax-cutlass-test-h100
Waiting for pending jobs
Matrix: arm64 / test-jax / run-unit-test
Waiting for pending jobs
Matrix: arm64 / test-te-a100 / run-unit-test
Waiting for pending jobs
Matrix: arm64 / test-te-h100 / te-test-h100
Waiting for pending jobs
arm64  /  build-torchax
7m 56s
arm64 / build-torchax
arm64  /  test-nsys-jax-eks
arm64 / test-nsys-jax-eks
arm64  /  ...  /  launch-slurm-runner
arm64 / test-jax / runner / launch-slurm-runner
arm64  /  ...  /  launch-slurm-runner
arm64 / test-te-a100 / runner / launch-slurm-runner
arm64  /  build-upstream-t5x
9m 46s
arm64 / build-upstream-t5x
arm64  /  build-axlearn
2m 3s
arm64 / build-axlearn
Matrix: arm64 / test-nsys-jax / run-unit-test
Waiting for pending jobs
arm64  /  ...  /  launch-slurm-runner
arm64 / test-nsys-jax / runner / launch-slurm-runner
Matrix: arm64 / test-nccl / nccl-test
Waiting for pending jobs
Matrix: arm64 / test-nccl / nccl-test-gke / nccl-gke
Waiting for pending jobs
amd64  /  ...  /  maxtext-gke-xpk
9m 35s
amd64 / test-maxtext-gke / maxtext-gke-xpk
Matrix: amd64 / test-maxtext / maxtext-multinode
Matrix: amd64 / test-maxtext / single-process-multi-device
amd64  /  ...  /  build-rosetta
14m 39s
amd64 / build-rosetta-t5x / build-rosetta
amd64  /  test-axlearn-eks
0s
amd64 / test-axlearn-eks
amd64  /  test-axlearn-fuji-models-eks
0s
amd64 / test-axlearn-fuji-models-eks
Matrix: amd64 / test-nsys-jax-archive
arm64  /  ...  /  maxtext-gke-xpk
arm64 / test-maxtext-gke / maxtext-gke-xpk
Matrix: arm64 / test-maxtext / maxtext-multinode
Waiting for pending jobs
Matrix: arm64 / test-maxtext / single-process-multi-device
Waiting for pending jobs
arm64  /  ...  /  build-rosetta
15m 42s
arm64 / build-rosetta-t5x / build-rosetta
arm64  /  test-axlearn-eks
arm64 / test-axlearn-eks
arm64  /  test-axlearn-fuji-models-eks
arm64 / test-axlearn-fuji-models-eks
Matrix: arm64 / test-nsys-jax-archive
amd64  /  ...  /  test-maxtext-metrics
21s
amd64 / test-maxtext / test-maxtext-metrics
amd64  /  collect-docker-tags
4s
amd64 / collect-docker-tags
Matrix: amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node
arm64  /  ...  /  test-maxtext-metrics
arm64 / test-maxtext / test-maxtext-metrics
arm64  /  collect-docker-tags
4s
arm64 / collect-docker-tags
Matrix: arm64 / test-rosetta-t5x / vit-multi-gpu-multi-node
Waiting for pending jobs
amd64  /  ...  /  sitrep
30s
amd64 / test-maxtext / test-maxtext-sitrep / sitrep
amd64  /  ...  /  test-t5x-rosetta-summary
3s
amd64 / test-rosetta-t5x / test-t5x-rosetta-summary
amd64  /  ...  /  test-t5x-rosetta-metrics
17s
amd64 / test-rosetta-t5x / test-t5x-rosetta-metrics
arm64  /  ...  /  sitrep
arm64 / test-maxtext / test-maxtext-sitrep / sitrep
arm64  /  ...  /  test-t5x-rosetta-summary
arm64 / test-rosetta-t5x / test-t5x-rosetta-summary
arm64  /  ...  /  test-t5x-rosetta-metrics
arm64 / test-rosetta-t5x / test-t5x-rosetta-metrics
amd64  /  ...  /  test-maxtext-outcome
2s
amd64 / test-maxtext / test-maxtext-outcome
amd64  /  ...  /  sitrep
14s
amd64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep
arm64  /  ...  /  test-maxtext-outcome
arm64 / test-maxtext / test-maxtext-outcome
arm64  /  ...  /  sitrep
arm64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep
amd64  /  ...  /  test-t5x-rosetta-outcome
3s
amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome
arm64  /  ...  /  test-t5x-rosetta-outcome
arm64 / test-rosetta-t5x / test-t5x-rosetta-outcome
make-publish-configs
5s
make-publish-configs
merge-new-manifest
12s
merge-new-manifest
Matrix: publish-containers
finalize  /  workflow-badge
5s
finalize / workflow-badge
finalize  /  report
20s
finalize / report
finalize  /  upload-badge
28s
finalize / upload-badge
finalize  /  publish-badge
5s
finalize / publish-badge
Fit to window
Zoom out
Zoom in

Annotations

8 errors and 2 warnings
amd64 / build-axlearn
buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c <<\"EOF\" bash -exu\n git config --global user.email \"${GIT_USER_EMAIL}\"\n git config --global user.name \"${GIT_USER_NAME}\"\n git-clone.sh \"${URLREF_AXLEARN}\" \"${SRC_PATH_AXLEARN}\"\n ${DEST_MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${DEST_MANIFEST_DIR}/manifest.yaml \\\n --package axlearn\nEOF" did not complete successfully: exit code: 1
amd64 / test-te-h100 / te-test-h100 (unittest, 8)
Process completed with exit code 1.
arm64 / build-axlearn
buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c <<\"EOF\" bash -exu\n git config --global user.email \"${GIT_USER_EMAIL}\"\n git config --global user.name \"${GIT_USER_NAME}\"\n git-clone.sh \"${URLREF_AXLEARN}\" \"${SRC_PATH_AXLEARN}\"\n ${DEST_MANIFEST_DIR}/create-distribution.sh \\\n --manifest ${DEST_MANIFEST_DIR}/manifest.yaml \\\n --package axlearn\nEOF" did not complete successfully: exit code: 1
amd64 / test-te-a100 / te-A100-unit-test
The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
amd64 / test-nsys-jax / nsys-jax-A100-unit-test
Process completed with exit code 1.
amd64 / test-maxtext / test-maxtext-outcome
Process completed with exit code 1.
amd64 / test-rosetta-t5x / test-t5x-rosetta-metrics
Process completed with exit code 1.
amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome
Process completed with exit code 1.
merge-new-manifest
Unexpected input(s) 'owner_and_repo', valid inputs are ['route', 'mediaType']
merge-new-manifest
Unexpected input(s) 'owner_and_repo', 'head', 'base', 'body', 'title', 'draft', valid inputs are ['route', 'mediaType']

Artifacts

Produced during runtime
Name Size Digest
artifact-axlearn-build-amd64
472 Bytes
sha256:8aa65b1efe68cc89095e892ac283f365fde223ccc0553ea0ffb0a67789cf1c95
artifact-axlearn-build-arm64
470 Bytes
sha256:f6627193ffbbb6d2acb9c0406545b6538ad0ad8c76031f35c5ee67ac3ae8fbaa
artifact-base-build-amd64
570 Bytes
sha256:23b7deccaf49fd319379b07b7bc4c89c930e004f43944dfead9bd3f7c5117087
artifact-base-build-arm64
567 Bytes
sha256:158ee88aedf2d444689788fab9994493e69569239c061df59d36d6e15c9eb6db
artifact-equinox-build-amd64
569 Bytes
sha256:0ae75827869152c7af2181e19fcaf9b06b702c2d3d8b1d6d004b4d9e09f2a337
artifact-equinox-build-arm64
569 Bytes
sha256:e4b843add831dd768b5bddcbcf2c10c3dc200b0a15cb83a3724bb01a58fd89d4
artifact-final-report
3.93 KB
sha256:d13c728d6fbe2032aa159ec6bafe543c9ccd2508ac79b7bed9d43319bee421db
artifact-jax-build-amd64
553 Bytes
sha256:0e0c4d3d6dc58041a09e9b54b207193a20095eade4f1b151659831f7a5a13579
artifact-jax-build-arm64
553 Bytes
sha256:a01592dd4a8351fbbbf293237dc0557e290db95e2fb87456e9e7709dc4138608
artifact-maxtext-build-amd64
567 Bytes
sha256:fa5efa0ef808491b43506df88bdac95ddaa795a142eb2c63e6a23947e83cf2a8
artifact-maxtext-build-arm64
568 Bytes
sha256:321589403cf4bde7de15c933d2897b966232939ba024de411244b12844933196
artifact-maxtext-test
1.48 KB
sha256:ec338f07b1cda3c9c6216b43bd71613636cf02beb7e00587cf56d1002d205b8f
artifact-mpi-operator-compatible-base-build-amd64
638 Bytes
sha256:dc9acee982d0c6eac90c80859b009d46c9aaa04de22c66b9f6c5e5851f090aeb
artifact-nccl-gke-build-amd64
570 Bytes
sha256:adc09fad7529eadb8c84a72b606d16d6562f0f6e6cdc308ae661025fdd01b9a1
artifact-rosetta-build-t5x-amd64
584 Bytes
sha256:d3f6c8152e8f1eb1ded28de9ff138d178d85a7864defd1de6d0b82136394dc70
artifact-rosetta-build-t5x-arm64
584 Bytes
sha256:f048012d2e3b1b6cc8906bec450be9a994d3bd45de44775810a9b0a754fe7ce9
artifact-rosetta-t5x-mgmn-test
624 Bytes
sha256:f62d4d1cb5e49c42c4541723b0342c975b1492481e38512ed7d57c9d7093dd29
artifact-t5x-build-amd64
570 Bytes
sha256:9bafb067e95c2a9a98f3e056083e2aafee52592cbc787641f65a0f38d8decc4a
artifact-t5x-build-arm64
570 Bytes
sha256:bbbfe37e29134cd98f5218b46f90dc0c7ae6f8078029071768a7abe56f2cbe82
artifact-torchax-build-amd64
568 Bytes
sha256:e66d0d225ad1acc35d37a7b948e05f2b6bcacd89ca88b59446da74f9ec09ec20
artifact-torchax-build-arm64
568 Bytes
sha256:6eb7553033b5cdc078f4830e80c507f0bf26e09e08ddb8aab99245edec7e1341
artifact-workflow-metadata
277 Bytes
sha256:1147a1aaedafbe67dbdaa642f1e1d321604fb05690911d3070e228d646059119
bumped-manifest
51.6 KB
sha256:897bc7f1397c761834e1b325b01fc3e14b5b99417a9e4e81bd33771638ff1345
final-base
249 Bytes
sha256:2c354d253bc63fb54fd726c39f648af0102ae70406dd089311b15a4c9b2a48a0
final-equinox
258 Bytes
sha256:557edc649bf8265f3e41a0a4db3162f82d6c0dbd076bc3b7c23eb9cdad931b51
final-jax
246 Bytes
sha256:354500a5619fd4e6df9be2a97d1d78a9908989bece6d587f1d6c8393a17215dc
final-maxtext
258 Bytes
sha256:b4190937d283f398c579b53892ead298959dfc09a2377a901c560ea93048da60
final-t5x
246 Bytes
sha256:c79a3287478221e5a89573510ec8cc7791ddfc8d678c335b8f62b75c35796b6a
final-upstream-t5x
273 Bytes
sha256:ee7d097a538067fb39ac6e924d1111fdd47249ca46a1d084a3bc4cabb13551cc
gke-maxtext-train
369 MB
sha256:afe444820da97c8c8c8f8ff9e91c48c0ac277d055b5b4c1893bed2e41e3703df
gke-maxtext-train-sitrep
228 Bytes
sha256:f3cffc8c3fb845d13eea86512d5bbea2fbe0122dd756324aa092fc89282d83ec
jax-cutlass-test-H100
1.24 KB
sha256:9ce6700411483e92b36c2a1da134f132f3e34a5e8a9a9a68421d9f1396a51e9c
jax-unit-test-A100
22.5 KB
sha256:6ac2e467ddaca88ad9625f4f74e129e0b31f1fa88a36c7b2b5c60f68bae941d4
mealkit-equinox
269 Bytes
sha256:d3a7277fd4ca87f682b9ffbb04628353af421ed8e5da091f9bf4386472c33eaa
mealkit-jax
256 Bytes
sha256:af297b53090b3e7743bacfbf03130d641c96b55208f623683a71689883e00ef9
mealkit-maxtext
269 Bytes
sha256:bba8981f79d32fc79f4295a2de182e90a27e3f7b4896e8974c0bb5b3764f2189
mealkit-t5x
258 Bytes
sha256:2f23d000bcdf7f3bdcfb89ecd4ca6bdc7ff2938915fed81c2a55ded01eae007a
mealkit-upstream-t5x
282 Bytes
sha256:e9e1623f4796fdba5c4378baf1141d92a5f31facf095d54891905134b01b2697
nccl-gke-all-gather
15.4 KB
sha256:606779723269ad28bb91bd51cea11f21038d2d4ad0ff2c663b3919bf024719a1
nccl-gke-all-gather-sitrep
231 Bytes
sha256:5b2e7ebb1532d8fefb7d8adbd4854e19e3d0612ea4d9b8e8220b7df70e10b22e
nccl-gke-all-reduce
15.6 KB
sha256:5b45274ca9f1b230ec3a8eccb3a9d453aafe8dea4aee5570250226d2bd7c021f
nccl-gke-all-reduce-sitrep
231 Bytes
sha256:0996b43a153f89e8a1ac01a1238d92ddd9cb253f19035bf1b8ba7e856e16e999
nccl-gke-broadcast
15.2 KB
sha256:99c33122e5db5ca2997068be6749d0d2073f12a37434f5922ca8035b19526d80
nccl-gke-broadcast-sitrep
229 Bytes
sha256:76f8219a9c2306eff44eb01de347060a517c85552f407d00432a087fb6e15b67
nccl-gke-reduce-scatter
15.5 KB
sha256:e94daa15e93ed630bc22cbf1717b342cf869faae2d3a4f5c5396b4b4f7dcbb3d
nccl-gke-reduce-scatter-sitrep
234 Bytes
sha256:1b0b52bc6d2985092d2dff2fbe48da19e32f06b35b44a2b1a7be99dce4d4f722
nsys-jax-unit-test-A100
137 MB
sha256:a0119137d020880969926ab03bd330e51aa7dc9846abe5b6eb203edb80b47e06
rosetta-t5x-vit-19496592076-VIT8G1N
15.8 KB
sha256:824493409137f917295b09cef95b8a0b7dc4177604d51f51f84494d90fad3581
te-unit-test-H100
4.48 MB
sha256:6f5324e8ea1c3081c8acacf9421f37c984722c2e82cc48fe9f85ad4b11af029c
upstream-maxtext-19496592076-1DP2FSDP4TP1PP_single_process
27.9 KB
sha256:0e997f02bbfaa8267b1722a045334a62a2b49aff4703a25300291760cfbdc1b3
upstream-maxtext-19496592076-2DP2FSDP2TP1PP
56 KB
sha256:0b3023190f436c9d3bf7247e2e86af27a88cb5451ea0c821a30afeaac16b4b43
upstream-maxtext-metrics-test-log
2.53 KB
sha256:b2a66db63e55404ac6dce85e6406770ba6cf8bfb20a8ddc697d748d8258276ff