Skip to content

Allow test failures for build uploads when not on release branch or tags #420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Allow test failures for build uploads when not on release branch or tags #420

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
9c8fd9e
Upload after build without waiting for tests
ericphanson Jan 24, 2025
92a6532
Revert "Upload after build without waiting for tests"
ericphanson Jan 24, 2025
04bfe0c
try to do conditional allowed failures
ericphanson Jan 25, 2025
020a601
sign treehashes
staticfloat Jan 30, 2025
1497415
add `ALLOW_TEST_FAILURES` in second place
ericphanson Jan 30, 2025
6c9d34f
add comments
ericphanson Feb 1, 2025
7996b54
Merge branch 'main' into pr/420
IanButterworth Jul 31, 2025
1ed1a42
sign treehashes
IanButterworth Jul 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified pipelines/main/launch_signed_jobs.yml.signature
View file
Open in desktop
Binary file not shown.
9 changes: 9 additions & 0 deletions pipelines/main/launch_upload_jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ steps:
export BUILDKITE_PLUGIN_CRYPTIC_BASE64_SIGNED_JOB_ID_SECRET
export ALLOW_FAIL="false"

# allow test-failures for non-release/tags so that nightly & PR builds
# can be uploaded to s3 even if tests are not passing
# (allowing for usage with juliaup)
export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")
Copy link
Member

@DilumAluthge DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't actually verified myself that this pattern matching is correct, so ideally someone would just do some quick tests in Bash locally before we merge.

Or alternatively, we can have a quick GitHub Actions CI job to check miscellaneous things such as this. E.g. just a quick Bash-based CI job that puts in various values of BUILDKITE_BRANCH and BUILDKITE_TAG, and tests that the output ALLOW_TEST_FAILURES is correct.

The latter is probably easier, because we can do it in CI instead of relying on people to manually test things locally.

Copy link
Member

@DilumAluthge DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, can we rename this something else? In Buildkite, "allow fail" (or "soft fail") means something different - it means failures show up as a red warning sign (not a red X), and failures don't block Test from being green.

Here, we have a different meaning, so can we make the variable name more unambiguous?

Suggested change
export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")
export UPLOAD_EVEN_IF_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")

##### Launch `upload_*` jobs to store tarballs into S3 once tests are done
### Linux:
# Regular Linux upload jobs:
Expand Down Expand Up @@ -84,6 +88,11 @@ steps:
export BUILDKITE_PLUGIN_CRYPTIC_BASE64_SIGNED_JOB_ID_SECRET

export ALLOW_FAIL="true"

# allow test-failures for non-release/tags so that nightly & PR builds
# can be uploaded to s3 even if tests are not passing
# (allowing for usage with juliaup)
export ALLOW_TEST_FAILURES=$(([[ "${BUILDKITE_BRANCH?}" == release-* ]] || [[ "${BUILDKITE_TAG:-}" == v* ]]) && echo "false" || echo "true")
Copy link
Member

@DilumAluthge DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above - would be nice to quickly test in CI (in a small GitHub Actions CI job).


# Launch Linux allowed-to-fail upload jobs
GROUP="Allow Fail (Upload)" \
Expand Down
3 changes: 2 additions & 1 deletion pipelines/main/launch_upload_jobs.yml.signature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
Salted__�B�<F���~g�F�����r���!�A�\S 0�(NF��Cb�FaR5�q�9�/�Vn��2�G�� Q��!x��ٔ����b�1�
Salted__��H i��ey}k#�ab47�9����� \œצ۹m���mdFU����3Ԟ\*\����U.��d*Xѥ"y���
�.���s,,K��
3 changes: 2 additions & 1 deletion pipelines/main/platforms/upload_freebsd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ steps:
# Wait for the builder to finish
- "build_${TRIPLET?}"
# Wait for the tester to finish
- "test_${TRIPLET?}"
- step: "test_${TRIPLET?}"
allow_failure: "${ALLOW_TEST_FAILURES?}"
Copy link
Member

@DilumAluthge DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
allow_failure: "${ALLOW_TEST_FAILURES?}"
allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

# Prevent multiple pipelines from uploading to S3 simultaneously
# It is okay for two different triplets to upload simultaneously
concurrency: 1
Expand Down
3 changes: 2 additions & 1 deletion pipelines/main/platforms/upload_linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ steps:
# Wait for the builder to finish
- "build_${TRIPLET?}"
# Wait for the tester to finish
- "test_${TRIPLET?}"
- step: "test_${TRIPLET?}"
allow_failure: "${ALLOW_TEST_FAILURES?}"
Copy link
Member

@DilumAluthge DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
allow_failure: "${ALLOW_TEST_FAILURES?}"
allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

# Prevent multiple pipelines from uploading to S3 simultaneously
# It is okay for two different triplets to upload simultaneously
concurrency: 1
Expand Down
3 changes: 2 additions & 1 deletion pipelines/main/platforms/upload_macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ steps:
# Wait for the builder to finish
- "build_${TRIPLET?}"
# Wait for the tester to finish
- "test_${TRIPLET?}"
- step: "test_${TRIPLET?}"
allow_failure: "${ALLOW_TEST_FAILURES?}"
Copy link
Member

@DilumAluthge DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
allow_failure: "${ALLOW_TEST_FAILURES?}"
allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

# Prevent multiple pipelines from uploading to S3 simultaneously
# It is okay for two different triplets to upload simultaneously
concurrency: 1
Expand Down
3 changes: 2 additions & 1 deletion pipelines/main/platforms/upload_windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ steps:
# Wait for the builder to finish
- "build_${TRIPLET?}"
# Wait for the tester to finish
- "test_${TRIPLET?}"
- step: "test_${TRIPLET?}"
allow_failure: "${ALLOW_TEST_FAILURES?}"
Copy link
Member

@DilumAluthge DilumAluthge Jul 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
allow_failure: "${ALLOW_TEST_FAILURES?}"
allow_failure: "${UPLOAD_EVEN_IF_TEST_FAILURES?}"

# Prevent multiple pipelines from uploading to S3 simultaneously
# It is okay for two different triplets to upload simultaneously
concurrency: 1
Expand Down
2 changes: 1 addition & 1 deletion pipelines/scheduled/coverage/coverage.yml.signature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Salted__���T���zF�>`/U���.��m�|Q�sϓ� ��|��������C ����>z�T/�t�����������U.��
Salted__�^���I��8�jz� :T3�ɶT3'c�W���Eny�-����e�:^^��'�򈍵464��q�If'����έ�SvY�d-�jˈ��
3 changes: 1 addition & 2 deletions pipelines/scheduled/launch_signed_jobs.yml.signature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
Salted__�j�);��xn����J�m,��7��s��������-0�j� �����x
Y��E�'%��Z+Z���1�ĎTNv���F]Ҽ
Salted__��6G�� ���γ��(�@=g_F7E�9Ԯ�l�u�N���!��6E�}�n��#��҆�I>3`Z4lj����õ��k��}�
Expand Down
4 changes: 2 additions & 2 deletions pipelines/scheduled/launch_upload_jobs.yml.signature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
Salted__�x @������o�C9���5�����*
f��B�1j:��*1 dk����d��!��Uޭ�Q�s�D�&U0�_�����f�^�
Salted__'��>~��|�%����*ci�(w8vi��kk�
��Uv��3Q�4�#��6S��$�|�p�$҃�g)%�,a��r�u!��-�WU