tests: Require run-fail ui tests to have an exit code (SIGABRT not ok)

[Enselic]

And introduce two new directives for ui tests:

• run-crash
• run-fail-or-crash

Normally a run-fail ui test like tests that panic shall not be terminated by a signal like SIGABRT. So begin having that as a hard requirement.

Some of our current tests do terminate by a signal/crash however. Introduce and use run-crash for those tests. Note that Windows crashes are not handled by signals but by certain high bits set on the process exit code. Example exit code for crash on Windows: 0xc000001d (STATUS_ILLEGAL_INSTRUCTION). Because of this, we define "crash" on all platforms as "not exit with success and not exit with a regular failure code in the range 1..=127".

Some tests behave differently on different targets:

• Targets without unwind support will abort (crash) instead of exit with failure code 101 after panicking. As a special case, allow crashes for run-fail tests for such targets.
• Different sanitizer implementations handle detected memory problems differently. Some abort (crash) the process while others exit with failure code 1. Introduce and use run-fail-or-crash for such tests.

This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as SIGABRT in tests/ui/panics/panic-main.rs (shown as Aborted (core dumped) in the logs attached to that issue, and I have also been able to reproduce this locally).

TODO

• Q: what about on Windows?
  A: we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix.
• test all permutations of actual vs expected
  Done: See tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002 (comment).
• Handle targets without unwind support
• Add run-fail-or-crash for some sanitizer tests
• remote-test-client. See remote-test-client: Exit code 128 + <signal-number> instead of 3 #143448

Zulip discussion

See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235

try-job: aarch64-apple
try-job: x86_64-msvc-1
try-job: x86_64-gnu
try-job: dist-i586-gnu-i586-i686-musl
try-job: test-various
try-job: armhf-gnu

[rustbot]

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

Some changes occurred in src/tools/compiletest

cc @jieyouxu

[workingjubilee]

why not simply run-abort

also would accept run-crash

[jieyouxu]

@bors2 delegate=try

[rust-bors]

@Enselic can now perform try builds on this pull request

[Enselic]

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

[rust-bors]

⌛ Trying commit 17be091 with merge 873ecba…

To cancel the try build, run the command @bors2 try cancel.

[Enselic]

An only-windows test that was run-fail now must be run-crash. Fixed now. Trying again:

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

[rust-bors]

⌛ Trying commit ad4e082 with merge e685982…

To cancel the try build, run the command @bors2 try cancel.

[Enselic]

I think the most elegant solution to the remote-test-client problem is to make remote-test-client exit with code 128 + <signal-number> instead of always 3. The just finished try-job confirmed that it works. That way we don't need any special case in compiletest since it then follows the already established rules of:

• run-fail code = 1..=127
• run-crash code is outside of 0..=127 (or completely absent).

I opened a separate PR for that so we can land that beforehand and in isolation since the impact might be high (but maybe it isn't, not sure. If all of CI passes it's fine I guess): #143448

I opted for a random reviewer on that PR but maybe someone from here wants to approve that PR.

[bors]

☔ The latest upstream changes (presumably #143667) made this pull request unmergeable. Please resolve the merge conflicts.

[Enselic]

I have now rebased on #143448 and the rest of the changes on master, so the changes from the last approved commit 6950daf are basically just cosmetic (see diff of diff below). I also had to change from run-fail to run-crash for three new tests.

So I will r=@petrochenkov this again after PR CI passes.

Click to expand diff of diff (yes it's a bit confusing but illustrates that the changes are minor)

diff -u <(git show 6950daf) <(git show f120fbe)

--- /proc/self/fd/15	2025-07-16 21:26:23.161561664 +0200
+++ /proc/self/fd/20	2025-07-16 21:26:23.161561664 +0200
@@ -1,4 +1,4 @@
-commit 6950daf354e1f9718583578e71cdad3cf16360cb (origin/tests-ui-run-fail-exit-vs-signal-20)
+commit f120fbe5c1b42dd08233ecf64eb45bc937319412 (HEAD -> tests-ui-run-fail-exit-vs-signal, origin/tests-ui-run-fail-exit-vs-signal-21, origin/tests-ui-run-fail-exit-vs-signal)
 Author: Martin Nordholts <martin.nordholts@codetale.se>
 Date:   Wed Jun 25 07:56:40 2025 +0200
 
@@ -45,7 +45,7 @@
  | `dont-check-failure-status` | Don't check exact failure status (i.e. `1`) | `ui`, `incremental`                       | N/A             |
  | `failure-status`            | Check                                       | `ui`, `crashes`                           | Any `u16`       |
 diff --git a/src/doc/rustc-dev-guide/src/tests/ui.md b/src/doc/rustc-dev-guide/src/tests/ui.md
-index f7e62e1eccf..7353e941e8e 100644
+index 4fce5838b6e..9bfc60e08a6 100644
 --- a/src/doc/rustc-dev-guide/src/tests/ui.md
 +++ b/src/doc/rustc-dev-guide/src/tests/ui.md
 @@ -448,7 +448,7 @@ even run the resulting program. Just add one of the following
@@ -58,8 +58,8 @@
    - `//@ check-fail` — compilation should fail (the codegen phase is skipped).
      This is the default for UI tests.
 @@ -457,10 +457,20 @@ even run the resulting program. Just add one of the following
-     without the codegen phase, then a second time the full compile should
-     fail.
+     - First time is to ensure that the compile succeeds without the codegen phase
+     - Second time is to ensure that the full compile fails
    - `//@ run-fail` — compilation should succeed, but running the resulting
 -    binary should fail.
 -
@@ -83,10 +83,10 @@
  If you want to check the output of running the program, include the
  `check-run-results` directive. This will check for a `.run.stderr` and
 diff --git a/src/tools/compiletest/src/common.rs b/src/tools/compiletest/src/common.rs
-index cdce5d941d0..21b5217d49b 100644
+index 33da1a25db1..c83070aaba7 100644
 --- a/src/tools/compiletest/src/common.rs
 +++ b/src/tools/compiletest/src/common.rs
-@@ -111,11 +111,37 @@ pub enum PassMode {
+@@ -88,11 +88,37 @@ pub enum PassMode {
      }
  }
  
@@ -125,34 +125,20 @@
  }
  
  string_enum! {
-diff --git a/src/tools/compiletest/src/directive-list.rs b/src/tools/compiletest/src/directive-list.rs
-index adf2a7bffef..84d162043fe 100644
---- a/src/tools/compiletest/src/directive-list.rs
-+++ b/src/tools/compiletest/src/directive-list.rs
-@@ -241,7 +241,9 @@
-     "regex-error-pattern",
-     "remap-src-base",
-     "revisions",
-+    "run-crash",
-     "run-fail",
-+    "run-fail-or-crash",
-     "run-flags",
-     "run-pass",
-     "run-rustfix",
 diff --git a/src/tools/compiletest/src/directives.rs b/src/tools/compiletest/src/directives.rs
-index a6242cf0c22..e10eb8ab39e 100644
+index 93133ea0bfd..513716357f4 100644
 --- a/src/tools/compiletest/src/directives.rs
 +++ b/src/tools/compiletest/src/directives.rs
 @@ -9,7 +9,7 @@
  use semver::Version;
  use tracing::*;
  
--use crate::common::{Config, Debugger, FailMode, Mode, PassMode};
-+use crate::common::{Config, Debugger, FailMode, Mode, PassMode, RunFailMode};
+-use crate::common::{Config, Debugger, FailMode, PassMode, TestMode};
++use crate::common::{Config, Debugger, FailMode, PassMode, RunFailMode, TestMode};
  use crate::debuggers::{extract_cdb_version, extract_gdb_version};
  use crate::directives::auxiliary::{AuxProps, parse_and_update_aux};
  use crate::directives::needs::CachedNeedsConditions;
-@@ -656,7 +656,13 @@ fn update_fail_mode(&mut self, ln: &str, config: &Config) {
+@@ -654,7 +654,13 @@ fn update_fail_mode(&mut self, ln: &str, config: &Config) {
              Some(FailMode::Build)
          } else if config.parse_name_directive(ln, "run-fail") {
              check_ui("run");
@@ -167,38 +153,41 @@
          } else {
              None
          };
+@@ -1007,7 +1013,9 @@ fn line_directive<'line>(
+     "regex-error-pattern",
+     "remap-src-base",
+     "revisions",
++    "run-crash",
+     "run-fail",
++    "run-fail-or-crash",
+     "run-flags",
+     "run-pass",
+     "run-rustfix",
 diff --git a/src/tools/compiletest/src/runtest.rs b/src/tools/compiletest/src/runtest.rs
-index f8bf4ee3022..6833b27253a 100644
+index cb8f593c9df..f66d4f98f1f 100644
 --- a/src/tools/compiletest/src/runtest.rs
 +++ b/src/tools/compiletest/src/runtest.rs
-@@ -17,10 +17,10 @@
+@@ -16,8 +16,8 @@
+ use tracing::*;
  
  use crate::common::{
-     Assembly, Codegen, CodegenUnits, CompareMode, Config, CoverageMap, CoverageRun, Crashes,
--    DebugInfo, Debugger, FailMode, Incremental, MirOpt, PassMode, Pretty, RunMake, Rustdoc,
--    RustdocJs, RustdocJson, TestPaths, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR, UI_RUN_STDOUT,
--    UI_STDERR, UI_STDOUT, UI_SVG, UI_WINDOWS_SVG, Ui, expected_output_path, incremental_dir,
--    output_base_dir, output_base_name, output_testname_unique,
-+    DebugInfo, Debugger, FailMode, Incremental, MirOpt, PassMode, Pretty, RunFailMode, RunMake,
-+    RunResult, Rustdoc, RustdocJs, RustdocJson, TestPaths, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR,
-+    UI_RUN_STDOUT, UI_STDERR, UI_STDOUT, UI_SVG, UI_WINDOWS_SVG, Ui, expected_output_path,
-+    incremental_dir, output_base_dir, output_base_name, output_testname_unique,
+-    CompareMode, Config, Debugger, FailMode, PassMode, TestMode, TestPaths, TestSuite,
+-    UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR, UI_RUN_STDOUT, UI_STDERR, UI_STDOUT, UI_SVG,
++    CompareMode, Config, Debugger, FailMode, PassMode, RunFailMode, RunResult, TestMode, TestPaths,
++    TestSuite, UI_EXTENSIONS, UI_FIXED, UI_RUN_STDERR, UI_RUN_STDOUT, UI_STDERR, UI_STDOUT, UI_SVG,
+     UI_WINDOWS_SVG, expected_output_path, incremental_dir, output_base_dir, output_base_name,
+     output_testname_unique,
  };
- use crate::compute_diff::{DiffLine, make_diff, write_diff, write_filtered_diff};
- use crate::directives::TestProps;
-@@ -277,7 +277,11 @@ fn pass_mode(&self) -> Option<PassMode> {
- 
+@@ -282,7 +282,8 @@ fn pass_mode(&self) -> Option<PassMode> {
      fn should_run(&self, pm: Option<PassMode>) -> WillExecute {
          let test_should_run = match self.config.mode {
--            Ui if pm == Some(PassMode::Run) || self.props.fail_mode == Some(FailMode::Run) => true,
-+            Ui if pm == Some(PassMode::Run)
-+                || matches!(self.props.fail_mode, Some(FailMode::Run(_))) =>
-+            {
-+                true
-+            }
-             MirOpt if pm == Some(PassMode::Run) => true,
-             Ui | MirOpt => false,
-             mode => panic!("unimplemented for mode {:?}", mode),
+             TestMode::Ui
+-                if pm == Some(PassMode::Run) || self.props.fail_mode == Some(FailMode::Run) =>
++                if pm == Some(PassMode::Run)
++                    || matches!(self.props.fail_mode, Some(FailMode::Run(_))) =>
+             {
+                 true
+             }
 diff --git a/src/tools/compiletest/src/runtest/ui.rs b/src/tools/compiletest/src/runtest/ui.rs
 index f6bc85cd051..0507c2600ae 100644
 --- a/src/tools/compiletest/src/runtest/ui.rs
@@ -489,20 +478,20 @@
  //@ ignore-i686-pc-windows-msvc: #112480
  //@ compile-flags: -C debug-assertions
  //@ error-pattern: misaligned pointer dereference: address must be a multiple of 0x4 but is
-diff --git a/tests/ui/mir/enum/convert_non_enum_break.rs b/tests/ui/mir/enum/convert_non_enum_break.rs
-index de062c39907..366f48dd77c 100644
---- a/tests/ui/mir/enum/convert_non_enum_break.rs
-+++ b/tests/ui/mir/enum/convert_non_enum_break.rs
+diff --git a/tests/ui/mir/enum/convert_non_integer_break.rs b/tests/ui/mir/enum/convert_non_integer_break.rs
+index 29795190bf6..b0778e2024f 100644
+--- a/tests/ui/mir/enum/convert_non_integer_break.rs
++++ b/tests/ui/mir/enum/convert_non_integer_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
  //@ compile-flags: -C debug-assertions
- //@ error-pattern: trying to construct an enum from an invalid value 0x10000
+ //@ error-pattern: trying to construct an enum from an invalid value
  
-diff --git a/tests/ui/mir/enum/convert_non_enum_niche_break.rs b/tests/ui/mir/enum/convert_non_enum_niche_break.rs
+diff --git a/tests/ui/mir/enum/convert_non_integer_niche_break.rs b/tests/ui/mir/enum/convert_non_integer_niche_break.rs
 index 9ff4849c5b1..d26a3aeb506 100644
---- a/tests/ui/mir/enum/convert_non_enum_niche_break.rs
-+++ b/tests/ui/mir/enum/convert_non_enum_niche_break.rs
+--- a/tests/ui/mir/enum/convert_non_integer_niche_break.rs
++++ b/tests/ui/mir/enum/convert_non_integer_niche_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
@@ -520,14 +509,14 @@
  //@ error-pattern: trying to construct an enum from an invalid value 0xfd
  
 diff --git a/tests/ui/mir/enum/niche_option_tuple_break.rs b/tests/ui/mir/enum/niche_option_tuple_break.rs
-index 43eef3a4cc5..70d4de1f3d0 100644
+index affdc4784a3..0a933afa153 100644
 --- a/tests/ui/mir/enum/niche_option_tuple_break.rs
 +++ b/tests/ui/mir/enum/niche_option_tuple_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
  //@ compile-flags: -C debug-assertions
- //@ error-pattern: trying to construct an enum from an invalid value 0x3
+ //@ error-pattern: trying to construct an enum from an invalid value
  
 diff --git a/tests/ui/mir/enum/numbered_variants_break.rs b/tests/ui/mir/enum/numbered_variants_break.rs
 index e3e71dc8aec..fbe7d6627a3 100644
@@ -570,14 +559,14 @@
  //@ error-pattern: trying to construct an enum from an invalid value 0x1
  
 diff --git a/tests/ui/mir/enum/with_niche_int_break.rs b/tests/ui/mir/enum/with_niche_int_break.rs
-index 0ec60a33564..9747bb8cfaf 100644
+index 6a97eaa8f4f..d363dc7568a 100644
 --- a/tests/ui/mir/enum/with_niche_int_break.rs
 +++ b/tests/ui/mir/enum/with_niche_int_break.rs
 @@ -1,4 +1,4 @@
 -//@ run-fail
 +//@ run-crash
  //@ compile-flags: -C debug-assertions
- //@ error-pattern: trying to construct an enum from an invalid value 0x4
+ //@ error-pattern: trying to construct an enum from an invalid value
  
 diff --git a/tests/ui/mir/enum/wrap_break.rs b/tests/ui/mir/enum/wrap_break.rs
 index 4491394ca5a..5c410afa511 100644
@@ -931,6 +920,36 @@
  //@ compile-flags: -Copt-level=3 -Cdebug-assertions=no -Zub-checks=yes
  //@ error-pattern: unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached
  
+diff --git a/tests/ui/precondition-checks/vec-from-parts.rs b/tests/ui/precondition-checks/vec-from-parts.rs
+index 0bafb5aa715..ace90770360 100644
+--- a/tests/ui/precondition-checks/vec-from-parts.rs
++++ b/tests/ui/precondition-checks/vec-from-parts.rs
+@@ -1,4 +1,4 @@
+-//@ run-fail
++//@ run-crash
+ //@ compile-flags: -Cdebug-assertions=yes
+ //@ error-pattern: unsafe precondition(s) violated: Vec::from_parts_in requires that length <= capacity
+ #![feature(allocator_api)]
+diff --git a/tests/ui/precondition-checks/vec-from-raw-parts.rs b/tests/ui/precondition-checks/vec-from-raw-parts.rs
+index 884d34c0a56..1bc8e6ada10 100644
+--- a/tests/ui/precondition-checks/vec-from-raw-parts.rs
++++ b/tests/ui/precondition-checks/vec-from-raw-parts.rs
+@@ -1,4 +1,4 @@
+-//@ run-fail
++//@ run-crash
+ //@ compile-flags: -Cdebug-assertions=yes
+ //@ error-pattern: unsafe precondition(s) violated: Vec::from_raw_parts_in requires that length <= capacity
+ //@ revisions: vec_from_raw_parts vec_from_raw_parts_in string_from_raw_parts
+diff --git a/tests/ui/precondition-checks/vec-set-len.rs b/tests/ui/precondition-checks/vec-set-len.rs
+index 0987e7fe028..c6bdee7dc67 100644
+--- a/tests/ui/precondition-checks/vec-set-len.rs
++++ b/tests/ui/precondition-checks/vec-set-len.rs
+@@ -1,4 +1,4 @@
+-//@ run-fail
++//@ run-crash
+ //@ compile-flags: -Cdebug-assertions=yes
+ //@ error-pattern: unsafe precondition(s) violated: Vec::set_len requires that new_len <= capacity()
+ 
 diff --git a/tests/ui/precondition-checks/write_volatile.rs b/tests/ui/precondition-checks/write_volatile.rs
 index 0d5ecb014b3..25107871c39 100644
 --- a/tests/ui/precondition-checks/write_volatile.rs

[Enselic]

(See above)

@bors r=@petrochenkov

[bors]

📌 Commit f120fbe has been approved by petrochenkov

It is now in the queue for this repository.

[fmease]

I've got a CI failure in a rollup: #144067 (comment). Not sure if it's caused by this PR alone or together with other ones conglomerated. Let's rerun CI and preemptively kick this one out of the queue for now.

@bors r-

[fmease]

@rustbot author

[Enselic]

Rebased on #142304 and #144073 and the rest of master. The CI failure was because this PR was combined with a new run-fail test that should have used run-crash (but it didn't know), namely the above test. So we just had some bad luck.

I'll r this again after PR CI is green.

click to expand diff of diff between last approved commit and the latest commit

diff -u <(git show f120fbe) <(git show e1d4f2a0c29)

--- /proc/self/fd/15    2025-07-19 18:56:28.381374122 +0200
+++ /proc/self/fd/18    2025-07-19 18:56:28.381374122 +0200
@@ -1,4 +1,4 @@
-commit f120fbe5c1b42dd08233ecf64eb45bc937319412 (origin/tests-ui-run-fail-exit-vs-signal-21, origin/tests-ui-run-fail-exit-vs-signal)
+commit e1d4f2a0c297690ddfc24815de57539f532f2471 (HEAD -> tests-ui-run-fail-exit-vs-signal-23, origin/tests-ui-run-fail-exit-vs-signal-23)
 Author: Martin Nordholts <martin.nordholts@codetale.se>
 Date:   Wed Jun 25 07:56:40 2025 +0200

@@ -28,7 +28,7 @@
       failure code 1. Introduce and use `run-fail-or-crash` for such tests.

 diff --git a/src/doc/rustc-dev-guide/src/tests/directives.md b/src/doc/rustc-dev-guide/src/tests/directives.md
-index 839076b809d..7c27e6a2052 100644
+index 63aa08c389c..6685add461e 100644
 --- a/src/doc/rustc-dev-guide/src/tests/directives.md
 +++ b/src/doc/rustc-dev-guide/src/tests/directives.md
 @@ -75,8 +75,10 @@ expectations](ui.md#controlling-passfail-expectations).
@@ -670,6 +670,49 @@
  //@ exec-env:RUST_BACKTRACE=0
  //@ check-run-results
  //@ error-pattern: panicked while processing panic
+diff --git a/tests/ui/panics/panic-main.rs b/tests/ui/panics/panic-main.rs
+index 2395f68241f..2009f69e19e 100644
+--- a/tests/ui/panics/panic-main.rs
++++ b/tests/ui/panics/panic-main.rs
+@@ -1,30 +1,37 @@
+ //@ revisions: default abort-zero abort-one abort-full unwind-zero unwind-one unwind-full
+
++//@[default] run-fail
++
+ //@[abort-zero] compile-flags: -Cpanic=abort
+ //@[abort-zero] no-prefer-dynamic
+ //@[abort-zero] exec-env:RUST_BACKTRACE=0
++//@[abort-zero] run-crash
+
+ //@[abort-one] compile-flags: -Cpanic=abort
+ //@[abort-one] no-prefer-dynamic
+ //@[abort-one] exec-env:RUST_BACKTRACE=1
++//@[abort-one] run-crash
+
+ //@[abort-full] compile-flags: -Cpanic=abort
+ //@[abort-full] no-prefer-dynamic
+ //@[abort-full] exec-env:RUST_BACKTRACE=full
++//@[abort-full] run-crash
+
+ //@[unwind-zero] compile-flags: -Cpanic=unwind
+ //@[unwind-zero] exec-env:RUST_BACKTRACE=0
+ //@[unwind-zero] needs-unwind
++//@[unwind-zero] run-fail
+
+ //@[unwind-one] compile-flags: -Cpanic=unwind
+ //@[unwind-one] exec-env:RUST_BACKTRACE=1
+ //@[unwind-one] needs-unwind
++//@[unwind-one] run-fail
+
+ //@[unwind-full] compile-flags: -Cpanic=unwind
+ //@[unwind-full] exec-env:RUST_BACKTRACE=full
+ //@[unwind-full] needs-unwind
++//@[unwind-full] run-fail
+
+-//@ run-fail
+ //@ error-pattern:moop
+ //@ needs-subprocess
+
 diff --git a/tests/ui/precondition-checks/alignment.rs b/tests/ui/precondition-checks/alignment.rs
 index 92400528fa0..038a625bed7 100644
 --- a/tests/ui/precondition-checks/alignment.rs

[Enselic]

Hmm the spelling mistake caught by the Spellcheck CI job is not introduced by me as far as I can tell. Probably a temporary problem during a transition period. Hopefully bors ignores that job for now.

[fmease]

Thanks! @bors r=petrochenkov

Hmm the spelling mistake caught by the Spellcheck CI job is not introduced by me as far as I can tell. Probably a temporary problem during a transition period. Hopefully bors ignores that job for now.

Yeah, see #t-infra > Spellcheck workflow now fails on all PRs (tree bad?) @ 💬. Indeed, this doesn't prevent bors merges / won't fail bors's auto builds, so we can ignore it.

[bors]

📌 Commit e1d4f2a has been approved by petrochenkov

It is now in the queue for this repository.

[bors]

⌛ Testing commit e1d4f2a with merge 6707bf0...

[bors]

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing 6707bf0 to master...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing ee3a078 (parent) -> 6707bf0 (this PR)

Test differences

Show 2 test diffs

2 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 6707bf0f59485cf054ac1095725df43220e4be20 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. pr-check-2: 2126.3s -> 3276.3s (54.1%)
2. x86_64-apple-2: 3681.7s -> 5300.6s (44.0%)
3. dist-x86_64-solaris: 5066.8s -> 6080.1s (20.0%)
4. dist-apple-various: 6246.5s -> 7299.6s (16.9%)
5. x86_64-rust-for-linux: 2529.5s -> 2936.6s (16.1%)
6. x86_64-apple-1: 8162.5s -> 9374.9s (14.9%)
7. pr-check-1: 1516.3s -> 1696.7s (11.9%)
8. i686-gnu-1: 7159.4s -> 7988.9s (11.6%)
9. x86_64-gnu-tools: 3317.2s -> 3699.9s (11.5%)
10. aarch64-gnu-debug: 3851.7s -> 4263.9s (10.7%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (6707bf0): comparison URL.

Overall result: ❌ regressions - please read the text below

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

• If the regression was expected or you think it can be justified,
  please write a comment with sufficient written justification, and add
  @rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
• If you think that you know of a way to resolve the regression, try to create
  a new PR with a fix for the regression.
• If you do not understand the regression or you think that it is just noise,
  you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
  were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.3%, 0.3%]	1
Regressions ❌ (secondary)	0.5%	[0.4%, 0.8%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[0.3%, 0.3%]	1

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results (secondary -3.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.0%	[-3.0%, -3.0%]	1
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 464.521s -> 465.018s (0.11%)
Artifact size: 374.65 MiB -> 374.58 MiB (-0.02%)

[rylev]

Looks like noise as the regression corrected itself on the next run.

@rustbot label: +perf-regression-triaged