tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

Enselic · 2025-06-25T06:10:21Z

Normally a run-fail ui test shall not be terminated by a signal like SIGABRT. So begin having that as a hard requirement.

Some of our current tests do terminate by a signal however. Introduce and use ~~run-fail-without-exit-code~~ run-crash for those tests.

This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as SIGABRT in tests/ui/panics/panic-main.rs (shown as Aborted (core dumped) in the logs attached to that issue, and I have also been able to reproduce this locally).

TODO

Q: what about on Windows?
A: we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix.
test all permutations of actual vs expected
Done: See tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002 (comment).

Zulip discussion

See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235

rustbot · 2025-06-25T06:10:25Z

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-06-25T06:10:29Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

workingjubilee · 2025-06-25T08:18:58Z

why not simply run-abort

also would accept run-crash

jieyouxu · 2025-06-25T10:10:37Z

@bors2 delegate=try

rust-bors · 2025-06-25T10:10:40Z

@Enselic can now perform try builds on this pull request

Enselic · 2025-06-25T11:37:25Z

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T11:37:29Z

⌛ Trying commit 17be091 with merge 873ecba…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] what about on Windows? - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

rust-bors · 2025-06-25T13:20:46Z

💔 Test failed

CI ❌

Enselic · 2025-06-25T13:34:03Z

An only-windows test that was run-fail now must be run-crash. Fixed now. Trying again:

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T13:34:07Z

⌛ Trying commit ad4e082 with merge e685982…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code - [ ] test all permutations of actual vs expected ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

rust-bors · 2025-06-25T16:09:16Z

💔 Test failed

CI ❌

rustbot · 2025-06-25T16:29:56Z

The rustc-dev-guide subtree was changed. If this PR only touches the dev guide consider submitting a PR directly to rust-lang/rustc-dev-guide otherwise thank you for updating the dev guide with your changes.

cc @BoxyUwU, @jieyouxu, @Kobzol, @tshepang

Enselic · 2025-06-25T16:31:28Z

The failing test in the second windows run was tests/run-make/short-ice/rmake.rs which is //@ ignore-windows so it's strange that it ran on Image: windows-2025. Some kind of flakiness? Let's see if we see that failure again.

Now testing all permutations of expected vs actual (i.e. CI should fail now).

Kobzol · 2025-06-25T17:01:00Z

Try to update your local branch on top of latest master, I think that test has been re-enabled on Windows a few days ago.

…t ok) Normally a `run-fail` ui test shall not be terminated by a signal like SIGABRT. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-crash` for those tests.

Enselic · 2025-06-25T18:17:09Z

For future reference: With this commit I tested what run-pass, run-fail and run-crash failures look like. The below can be confirmed here.

When run-pass fails with an exit code it looks like this:

error: test run failed! code=Some(101)
status: exit status: 101

When run-pass fails with a crash it looks like this:

error: test run failed! code=None
status: signal: 5 (SIGTRAP) (core dumped)

when run-fail fails with a crash it looks like this:

error: test did not exit with failure! code=None
status: signal: 6 (SIGABRT) (core dumped)

when run-fail fails by exiting with success it looks like this:

error: test did not exit with failure! code=Some(0)
status: exit status: 0

when run-crash fails by exiting with success it looks like this:

error: test did not crash! code=Some(0)
status: exit status: 0

when run-crash fails by exiting with failure it looks like this:

error: test did not crash! code=Some(101)
status: exit status: 101

Enselic · 2025-06-25T18:22:11Z

Hopefully we don't get the strange failure on Windows now after I rebased on latest master.

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T18:22:15Z

⌛ Trying commit 80e8db8 with merge 40ea28e…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See #143002 (comment). ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

rust-bors · 2025-06-25T20:58:31Z

☀️ Try build successful (CI)
Build commit: 40ea28e (40ea28e11de07e6605ea7913a16a2bb4802d5c4c, parent: 8cf5fad73d4e8f41863ecc3bcfa114eabc951faa)

Enselic · 2025-06-25T21:11:16Z

Now that tests pass on Windows I'd also like to make sure that failures also work on Windows by doing this on Windows.

… run-crash

Enselic · 2025-06-25T21:12:55Z

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T21:12:59Z

⌛ Trying commit 385df9c with merge 88154f8…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See #143002 (comment). ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

rust-log-analyzer · 2025-06-25T21:58:10Z

The job aarch64-gnu-llvm-19-1 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

failures:

---- [ui] tests/ui/deref.rs stdout ----

error: test run failed! code=Some(101)
status: exit status: 101
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/deref" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/deref/a"
stdout: none
--- stderr -------------------------------

---


---- [ui] tests/ui/inlined-main.rs stdout ----

error: test run failed! code=None
status: signal: 5 (SIGTRAP) (core dumped)
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/inlined-main" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/inlined-main/a"
stdout: none
stderr: none


---- [ui] tests/ui/mir/alignment/borrow_misaligned_field_projection.rs stdout ----

error: test did not crash! code=Some(0)
status: exit status: 0
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/mir/alignment/borrow_misaligned_field_projection" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/mir/alignment/borrow_misaligned_field_projection/a"
stdout: none
stderr: none


---- [ui] tests/ui/mir/alignment/misaligned_borrow.rs stdout ----

error: test did not crash! code=Some(101)
status: exit status: 101
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/mir/alignment/misaligned_borrow" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/mir/alignment/misaligned_borrow/a"
stdout: none
--- stderr -------------------------------

---


---- [ui] tests/ui/panics/main-panic.rs stdout ----

error: test did not exit with failure! code=None
status: signal: 6 (SIGABRT) (core dumped)
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/main-panic" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/main-panic/a"
stdout: none
--- stderr -------------------------------

thread 'main' panicked at /checkout/tests/ui/panics/main-panic.rs:8:9:
null pointer dereference occurred
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread caused non-unwinding panic. aborting.
------------------------------------------


---- [ui] tests/ui/panics/panic-main.rs stdout ----

error: test did not exit with failure! code=Some(0)
status: exit status: 0
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main" && RUSTC="/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc" RUST_TEST_THREADS="4" "/checkout/obj/build/aarch64-unknown-linux-gnu/test/ui/panics/panic-main/a"
stdout: none
stderr: none

rust-bors · 2025-06-25T22:51:07Z

💔 Test failed

CI ❌

bors · 2025-06-26T03:46:06Z

☔ The latest upstream changes (presumably #143026) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot assigned petrochenkov Jun 25, 2025

Enselic added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 25, 2025

Enselic mentioned this pull request Jun 25, 2025

RUST_BACKTRACE=full loop with -Cpanic=abort on aarch64-unknown-linux-gnu #123733

Open

rustbot added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label Jun 25, 2025

This comment has been minimized.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 6ad1973 to 17be091 Compare June 25, 2025 10:12

This comment was marked as outdated.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 17be091 to ad4e082 Compare June 25, 2025 13:30

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from ad4e082 to 2499dac Compare June 25, 2025 16:29

rustbot added the A-rustc-dev-guide Area: rustc-dev-guide label Jun 25, 2025

This comment has been minimized.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch 2 times, most recently from 2c00e05 to 80e8db8 Compare June 25, 2025 18:07

TEMP: crash and fail run-pass, crash and pass run-fail, fail and pass…

385df9c

… run-crash

tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002

Are you sure you want to change the base?

tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002

Conversation

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Zulip discussion

Uh oh!

rustbot commented Jun 25, 2025

Uh oh!

rustbot commented Jun 25, 2025

Uh oh!

This comment has been minimized.

workingjubilee commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieyouxu commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Enselic commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

rustbot commented Jun 25, 2025

Uh oh!

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Jun 25, 2025

Uh oh!

This comment has been minimized.

Enselic commented Jun 25, 2025

Uh oh!

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Enselic commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

rust-log-analyzer commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

bors commented Jun 26, 2025

Uh oh!

Uh oh!

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

Enselic commented Jun 25, 2025 •

edited

Loading

workingjubilee commented Jun 25, 2025 •

edited

Loading

Enselic commented Jun 25, 2025 •

edited

Loading

Enselic commented Jun 25, 2025 •

edited

Loading

Enselic commented Jun 25, 2025 •

edited

Loading

Enselic commented Jun 25, 2025 •

edited

Loading