Skip to content

CI

CI #5048

Triggered via schedule November 17, 2025 09:36
Status Failure
Total duration 5h 27m 40s
Artifacts 47

ci.yaml

on: schedule
metadata
4s
metadata
bump-manifest
14s
bump-manifest
Matrix: amd64 / test-distribution
Matrix: arm64 / test-distribution
amd64  /  ...  /  build-base
2m 27s
amd64 / build-base / build-base
arm64  /  ...  /  build-base
3m 0s
arm64 / build-base / build-base
amd64  /  ...  /  build-mpi-operator-compatible-base
2m 14s
amd64 / test-nccl / build-mpi-operator-compatible-base
amd64  /  ...  /  build-nccl-gke
1m 54s
amd64 / test-nccl / nccl-test-gke / build-nccl-gke
arm64  /  ...  /  build-mpi-operator-compatible-base
arm64 / test-nccl / build-mpi-operator-compatible-base
arm64  /  ...  /  build-nccl-gke
arm64 / test-nccl / nccl-test-gke / build-nccl-gke
Matrix: amd64 / test-jax-cutlass-h100 / jax-cutlass-test-h100
Matrix: amd64 / test-jax / run-unit-test
Matrix: amd64 / test-te-a100 / run-unit-test
Matrix: amd64 / test-te-h100 / te-test-h100
amd64  /  build-torchax
6m 37s
amd64 / build-torchax
amd64  /  ...  /  launch-slurm-runner
1h 41m
amd64 / test-jax / runner / launch-slurm-runner
amd64  /  test-nsys-jax-eks
4m 7s
amd64 / test-nsys-jax-eks
amd64  /  ...  /  launch-slurm-runner
1h 12m
amd64 / test-te-a100 / runner / launch-slurm-runner
amd64  /  build-upstream-t5x
10m 3s
amd64 / build-upstream-t5x
Matrix: amd64 / test-nsys-jax / run-unit-test
amd64  /  build-equinox
6m 8s
amd64 / build-equinox
amd64  /  ...  /  launch-slurm-runner
1h 1m
amd64 / test-nsys-jax / runner / launch-slurm-runner
Matrix: amd64 / test-nccl / nccl-test
Matrix: amd64 / test-nccl / nccl-test-gke / nccl-gke
Matrix: arm64 / test-jax-cutlass-h100 / jax-cutlass-test-h100
Waiting for pending jobs
Matrix: arm64 / test-jax / run-unit-test
Waiting for pending jobs
Matrix: arm64 / test-te-a100 / run-unit-test
Waiting for pending jobs
Matrix: arm64 / test-te-h100 / te-test-h100
Waiting for pending jobs
arm64  /  build-maxtext
5m 0s
arm64 / build-maxtext
arm64  /  build-torchax
7m 29s
arm64 / build-torchax
arm64  /  test-nsys-jax-eks
0s
arm64 / test-nsys-jax-eks
arm64  /  ...  /  launch-slurm-runner
arm64 / test-jax / runner / launch-slurm-runner
arm64  /  ...  /  launch-slurm-runner
arm64 / test-te-a100 / runner / launch-slurm-runner
arm64  /  build-upstream-t5x
10m 0s
arm64 / build-upstream-t5x
Matrix: arm64 / test-nsys-jax / run-unit-test
Waiting for pending jobs
arm64  /  ...  /  launch-slurm-runner
arm64 / test-nsys-jax / runner / launch-slurm-runner
Matrix: arm64 / test-nccl / nccl-test
Waiting for pending jobs
Matrix: arm64 / test-nccl / nccl-test-gke / nccl-gke
Waiting for pending jobs
amd64  /  ...  /  maxtext-gke-xpk
amd64 / test-maxtext-gke / maxtext-gke-xpk
Matrix: amd64 / test-maxtext / maxtext-multinode
Waiting for pending jobs
Matrix: amd64 / test-maxtext / single-process-multi-device
Waiting for pending jobs
amd64  /  ...  /  build-rosetta
13m 15s
amd64 / build-rosetta-t5x / build-rosetta
amd64  /  test-axlearn-eks
12m 26s
amd64 / test-axlearn-eks
amd64  /  test-axlearn-fuji-models-eks
17m 58s
amd64 / test-axlearn-fuji-models-eks
Matrix: amd64 / test-nsys-jax-archive
arm64  /  ...  /  maxtext-gke-xpk
arm64 / test-maxtext-gke / maxtext-gke-xpk
Matrix: arm64 / test-maxtext / maxtext-multinode
Waiting for pending jobs
Matrix: arm64 / test-maxtext / single-process-multi-device
Waiting for pending jobs
arm64  /  ...  /  build-rosetta
18m 18s
arm64 / build-rosetta-t5x / build-rosetta
arm64  /  test-axlearn-eks
0s
arm64 / test-axlearn-eks
arm64  /  test-axlearn-fuji-models-eks
0s
arm64 / test-axlearn-fuji-models-eks
Matrix: arm64 / test-nsys-jax-archive
amd64  /  ...  /  test-maxtext-metrics
amd64 / test-maxtext / test-maxtext-metrics
amd64  /  collect-docker-tags
3s
amd64 / collect-docker-tags
Matrix: amd64 / test-rosetta-t5x / vit-multi-gpu-multi-node
arm64  /  ...  /  test-maxtext-metrics
arm64 / test-maxtext / test-maxtext-metrics
arm64  /  collect-docker-tags
9s
arm64 / collect-docker-tags
Matrix: arm64 / test-rosetta-t5x / vit-multi-gpu-multi-node
Waiting for pending jobs
amd64  /  ...  /  sitrep
amd64 / test-maxtext / test-maxtext-sitrep / sitrep
amd64  /  ...  /  test-t5x-rosetta-summary
4s
amd64 / test-rosetta-t5x / test-t5x-rosetta-summary
amd64  /  ...  /  test-t5x-rosetta-metrics
18s
amd64 / test-rosetta-t5x / test-t5x-rosetta-metrics
arm64  /  ...  /  sitrep
arm64 / test-maxtext / test-maxtext-sitrep / sitrep
arm64  /  ...  /  test-t5x-rosetta-summary
arm64 / test-rosetta-t5x / test-t5x-rosetta-summary
arm64  /  ...  /  test-t5x-rosetta-metrics
arm64 / test-rosetta-t5x / test-t5x-rosetta-metrics
amd64  /  ...  /  test-maxtext-outcome
amd64 / test-maxtext / test-maxtext-outcome
amd64  /  ...  /  sitrep
8s
amd64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep
arm64  /  ...  /  test-maxtext-outcome
arm64 / test-maxtext / test-maxtext-outcome
arm64  /  ...  /  sitrep
arm64 / test-rosetta-t5x / test-t5x-rosetta-sitrep / sitrep
amd64  /  ...  /  test-t5x-rosetta-outcome
3s
amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome
arm64  /  ...  /  test-t5x-rosetta-outcome
arm64 / test-rosetta-t5x / test-t5x-rosetta-outcome
make-publish-configs
4s
make-publish-configs
merge-new-manifest
12s
merge-new-manifest
Matrix: publish-containers
finalize  /  workflow-badge
5s
finalize / workflow-badge
finalize  /  report
15s
finalize / report
finalize  /  upload-badge
15s
finalize / upload-badge
finalize  /  publish-badge
5s
finalize / publish-badge
Fit to window
Zoom out
Zoom in

Annotations

7 errors and 2 warnings
amd64 / test-te-h100 / te-test-h100 (unittest, 8)
Process completed with exit code 1.
amd64 / build-maxtext
buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c pip-finalize.sh" did not complete successfully: exit code: 1
arm64 / build-maxtext
buildx failed with: ERROR: failed to build: failed to solve: process "/bin/sh -c pip-finalize.sh" did not complete successfully: exit code: 1
amd64 / test-te-a100 / te-A100-unit-test
The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
amd64 / test-nsys-jax / nsys-jax-A100-unit-test
Process completed with exit code 1.
amd64 / test-rosetta-t5x / test-t5x-rosetta-metrics
Process completed with exit code 1.
amd64 / test-rosetta-t5x / test-t5x-rosetta-outcome
Process completed with exit code 1.
merge-new-manifest
Unexpected input(s) 'owner_and_repo', valid inputs are ['route', 'mediaType']
merge-new-manifest
Unexpected input(s) 'owner_and_repo', 'head', 'base', 'body', 'title', 'draft', valid inputs are ['route', 'mediaType']

Artifacts

Produced during runtime
Name Size Digest
artifact-axlearn-build-amd64
567 Bytes
sha256:573c9e475c9e59d34ce5cae4025437d462fc1aa25cc035df6233a5656850a37a
artifact-axlearn-build-arm64
566 Bytes
sha256:275f4ac79938b9a194a2953f0a828316312f7bb07afd341349bb4b318d05d805
artifact-axlearn-test
179 KB
sha256:0ca9e92f4eb493fd5b7fbb59a90c09da3880cf960dbdf5a638a6aca68e295106
artifact-base-build-amd64
566 Bytes
sha256:8db4a77cb46dea13147b3c3e2187cebc5da65a97f9af40ab97f5bb5d7a8659ee
artifact-base-build-arm64
566 Bytes
sha256:7ee95fcb8af73c1b80cac4412b8e60fc4367bd9ea20fcc12d0f7a9cbd59ba90a
artifact-equinox-build-amd64
569 Bytes
sha256:234526c1d4a4f2004ac7c7fafa913d080b4aba0ed2a3254933bd6d722094ae93
artifact-equinox-build-arm64
569 Bytes
sha256:978a73f88c3a097296f40110ae5145694f128eb92d3aa3632123feb8c78df85c
artifact-final-report
3.26 KB
sha256:b857ac3dd3e4db94ddf3431b95990470209c976790411e76e9ad58843a56012f
artifact-jax-build-amd64
553 Bytes
sha256:1a2297ed205bb05f46a111887e685b94ba942768798b1e98e212d1557a9aa2c1
artifact-jax-build-arm64
554 Bytes
sha256:cab00e9ed7baff04c916455e95e1d1380ffe666ef783f59700ffe921b36bb453
artifact-maxtext-build-amd64
515 Bytes
sha256:32a5d42c1a422c796f011d121ab9c187c81eec624209b2ad88b6797e973883a6
artifact-maxtext-build-arm64
515 Bytes
sha256:76f325159c9a3587751015855769f22e5901edee071fd4face0fea19c5b69fbd
artifact-mpi-operator-compatible-base-build-amd64
639 Bytes
sha256:eb3296614afe0ee0d0df1e79586fbd2fab86d3c8c764ad67af7fbf97eb997eee
artifact-nccl-gke-build-amd64
571 Bytes
sha256:8ca4807594ae1bf484665aaa871eb120d430a321808d8f4a8ceaaf3d1354f2fb
artifact-rosetta-build-t5x-amd64
583 Bytes
sha256:a2b74498f8eb512b989f0911b226bebae6a624dbc8800727a83bbef25e49e2a3
artifact-rosetta-build-t5x-arm64
585 Bytes
sha256:cadc26c16b11068c07d6b43b8b599a50cd0b58e6230dc5aea0f03bf98e84010a
artifact-rosetta-t5x-mgmn-test
624 Bytes
sha256:99104a0ae25b045440a85f8bf1187d10240e322a8477b8f4d91e1d42cf965750
artifact-t5x-build-amd64
568 Bytes
sha256:9865b8b179e16ae56dfcaa254c971225f4d4e6bc6c7c30ffdb266da1523b6858
artifact-t5x-build-arm64
568 Bytes
sha256:755c44cd0c996af40e852910de183ef11ad90dc7d8b6a7a04c12eac71cf114af
artifact-torchax-build-amd64
567 Bytes
sha256:4cfb76b012a190b3233bae20aa2d1a62195221dbb8d31c162d5c2c4e3d546049
artifact-torchax-build-arm64
567 Bytes
sha256:9e8936b10ffdb264ff8c9080ffdb6df047d7629f129335abd0ec954030033dea
artifact-workflow-metadata
277 Bytes
sha256:9ea701da64e6f7647ef12c2440b7e78c302c1c35549f254f6b7fec834665fcbf
bumped-manifest
51.6 KB
sha256:050cdcdda111f279178baad907d68c11edf8643f38f15186510ca951912cffb8
final-axlearn
258 Bytes
sha256:9038d2f3304a4dbe06384c2dd1523266a1d764da177ee379235393e97b3e9fdc
final-base
249 Bytes
sha256:27589fc1b48ce66627911ef4de1e2663847cf4f287c1eb22ac5c753b70400225
final-equinox
258 Bytes
sha256:d2821a4e909d09103eee798263ff54d1511eddad047d7e7bf36ff3b329fd8e57
final-jax
246 Bytes
sha256:04bfd553b5da8f130c16a8f2adc2174d09ac30def2f4669c171cc0db9752a3a4
final-t5x
246 Bytes
sha256:7cfa3df8c957d4c1ce09df6d37aa976dc41784fa55568325e340ee7f5c5c979c
final-upstream-t5x
273 Bytes
sha256:c389bf9ea6a9625ac2e9de365cbc699635b85d7b825ca02ecf04288bf43c5935
jax-cutlass-test-H100
1.24 KB
sha256:137ad5544da5911bc0b1288e04a2f97cee64b82baf3c80f06a73a5cba88570ac
jax-unit-test-A100
22.5 KB
sha256:9a28d904fda5b0d6df32d1345966153f7fd0a7c46b2c46d06a60a4f8164136b6
mealkit-axlearn
269 Bytes
sha256:28ac356e43fdf9344cfe7907ba6ac1f1fb0585fb1832299221e85fa3e86c9538
mealkit-equinox
270 Bytes
sha256:5ba2116375232b2b6eaa4e076ae4752e6fe6ccd4c7516831c520e415c3bd0720
mealkit-jax
256 Bytes
sha256:fe7c8721c519c10c82b9fd20512133482f0ca4a3808af40a40860bda07aa1223
mealkit-t5x
258 Bytes
sha256:8437b0b625f1c0e47729ac8c1c208e4702b261c3eb4dbde8c8d8fefb0e2c523e
mealkit-upstream-t5x
283 Bytes
sha256:b537e7670d4b5116f92428906d7c77318d91d3be6f54f742cf319830cba54f6a
nccl-gke-all-gather
15.4 KB
sha256:4fe070d276c9ab45a559cf7b502d97027c5125d4268e55d63a699ce836c28664
nccl-gke-all-gather-sitrep
231 Bytes
sha256:8c8096aaed2db392078d97506ab46980ec6ae1dfe2c60a2d45713ba4e23c956e
nccl-gke-all-reduce
15.6 KB
sha256:8cd26daa9b2f418cee09d602ff3410ec70ae7679532f9b69510403660c6f912d
nccl-gke-all-reduce-sitrep
231 Bytes
sha256:b16615a3f860c95dfc02935bccc86957aa8adc28047ce5deaa1b256d0cb1c06a
nccl-gke-broadcast
15.2 KB
sha256:753015130ae49fd56e840f5254eb048c1d7051a3f0e62ec5c8f1ca3b816cecea
nccl-gke-broadcast-sitrep
229 Bytes
sha256:f509512d4bb88a03f4689c19780ba0d20ff5ffc5d15cb73279da37a73990dba9
nccl-gke-reduce-scatter
15.5 KB
sha256:df5dc76a4e77f6afe968504b6970a65a88c3bc6e2520d4aa213fd3c27606c3d4
nccl-gke-reduce-scatter-sitrep
234 Bytes
sha256:2b1aa33bd2c0d849f0bc07cf9bc5688645d7c301c5a296cfbcebaa15ef5ab500
nsys-jax-unit-test-A100
140 MB
sha256:8cd6c0323b0726fd6990c94df7dc49b3568dd5751ef957d404a454c6cd8eed8d
rosetta-t5x-vit-19424945828-VIT8G1N
15.5 KB
sha256:1003384e854ed152e4755ed8833989ef7edd852f01c48813dfbc4d3351af0a0b
te-unit-test-H100
2.09 MB
sha256:470d32b9c2179ac5c122c2bedc7aa0524e070db30351a257cc7f965b5d0530dc