Skip to content

Allow ./x test to run tests without doc tests and without benchmarks#153143

Merged
rust-bors[bot] merged 3 commits intorust-lang:mainfrom
ferrocene:jh/bootstrap-test-targets
Mar 11, 2026
Merged

Allow ./x test to run tests without doc tests and without benchmarks#153143
rust-bors[bot] merged 3 commits intorust-lang:mainfrom
ferrocene:jh/bootstrap-test-targets

Conversation

@Urhengulas
Copy link
Contributor

@Urhengulas Urhengulas commented Feb 26, 2026

View all comments

Problem

For Ferrocene we would like to run only the coretests and alloctests test suites when measuring code coverage. Running corebenches and allocbenches would alter the numbers, which is not compliant with the certification.

This is currently not possible in bootstrap. By default ./x test runs unit, integration and doc tests. ./x test --doc only runs doc tests. So far, so good.

The problem is that while ./x test --no-doc stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like cargo test --all-targets.

Solution

This PR renames the existing --no-doc flag to --all-targets and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from cargo --bins --examples --tests --benches to cargo --all-targets because it will fail e.g. ./x test library/alloc --all-targets with an error like "use of unstable library feature test".

Additionally, this PR add a ./x test --tests flag (equivalent to cargo test --tests) that only executes unit and integration tests.

Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them.

Note that this PR does not change the behaviour of either ./x test or ./x test --doc.

Note on test = true

While this change enables bootstrap to run tests without doc tests and without benchmarks, executing ./x test library/core library/alloc --tests will still build and execute corebenches and allocbenches.

What?! 😱 Why all of this effort to enable it then?

Good question! The reason they are still executed is, that they are marked with test = true in their respective Cargo.toml (corebenches, allocbenches).

@bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no ./x --all-targets is passed. This is perfectly possible with this PR. Benchmarks marked with test = true will be executed when calling either of ./x test, ./x test --tests or ./x --all-targets. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene.

Open questions

Error message

I added one commit that adds an error message if a user passes --no-doc and points them to --all-targets and --tests. I think it might be nice to have, but if you think it is not necessary, I can remove it.

How to test this change

You can see the change in action by running ./x test library/alloc --tests and ./x test library/alloc --all-targets. The first will execute alloctests and allocbenches (which is marked with test = true), while the second will additionally run benches/vec_deque_append.rs (which is not marked with test = true).

@rustbot
Copy link
Collaborator

rustbot commented Feb 26, 2026

The rustc-dev-guide subtree was changed. If this PR only touches the dev guide consider submitting a PR directly to rust-lang/rustc-dev-guide otherwise thank you for updating the dev guide with your changes.

cc @BoxyUwU, @jieyouxu, @Kobzol, @tshepang

@rustbot rustbot added A-rustc-dev-guide Area: rustc-dev-guide A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Feb 26, 2026
@rustbot
Copy link
Collaborator

rustbot commented Feb 26, 2026

r? @jieyouxu

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

  • Owners of files modified in this PR: bootstrap
  • bootstrap expanded to 6 candidates
  • Random selection from Mark-Simulacrum, clubby789, jieyouxu

@Urhengulas Urhengulas force-pushed the jh/bootstrap-test-targets branch from 772d7dc to 52327a7 Compare February 26, 2026 16:57
@Urhengulas
Copy link
Contributor Author

Urhengulas commented Feb 27, 2026

I am pretty sure that following workaround will also become unnecessary when running ./x library/alloc --tests:

if cfg!(miri) {
// Don't benchmark Miri...
// (Due to bootstrap quirks, this gets picked up by `x.py miri library/alloc --no-doc`.)
return;
}

@jieyouxu
Copy link
Member

jieyouxu commented Mar 3, 2026

@rustbot reroll

@rustbot rustbot assigned Mark-Simulacrum and unassigned jieyouxu Mar 3, 2026
@Mark-Simulacrum
Copy link
Member

r=me with commits squashed a bit (or fully)

Copy link
Member

@Kobzol Kobzol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add a change tracker entry to change_tracker.rs, telling people that there is a new flag and --doc is deprecated? Thanks!

View changes since this review

@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 9, 2026
@rustbot
Copy link
Collaborator

rustbot commented Mar 9, 2026

Reminder, once the PR becomes ready for a review, use @rustbot ready.

@Urhengulas Urhengulas force-pushed the jh/bootstrap-test-targets branch from 5f8d29e to e6f3e44 Compare March 10, 2026 11:02
@Kobzol
Copy link
Member

Kobzol commented Mar 10, 2026

Sorry for the late notice, but not that I re-read the changed if conditions, I think that TestTarget::DocOnly would be a much clearer name, because that is what it actually means; doctests are otherwise also executed in the normal, "flagless" invocation of Cargo test. What do you think?

@Urhengulas
Copy link
Contributor Author

Sorry for the late notice, but not that I re-read the changed if conditions, I think that TestTarget::DocOnly would be a much clearer name, because that is what it actually means; doctests are otherwise also executed in the normal, "flagless" invocation of Cargo test. What do you think?

@Kobzol Sounds good to me! Will rename it right now :)

@Urhengulas
Copy link
Contributor Author

@Mark-Simulacrum I also squashed the commits into two

@Urhengulas Urhengulas force-pushed the jh/bootstrap-test-targets branch from e6f3e44 to 38d5071 Compare March 10, 2026 11:08
@Urhengulas
Copy link
Contributor Author

Urhengulas commented Mar 10, 2026

@Kobzol I force pushed the rename (e6f3e447..38d5071b)

@Kobzol
Copy link
Member

Kobzol commented Mar 10, 2026

Since it was only a CLI flag and not a config option, I guess I'm fine with removing the flag; not sure what's the general policy on keeping backwards compatibility with flags vs config options. I'll ask on Zulip though.

- `./x run generate-completions`
- Search and replace `--no-doc` with `--all-targets`
  This is to keep the behaviour the same.
- Document `./x test --tests` in `rustc-dev-guide`
- Add change info
JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 10, 2026
…s, r=Mark-Simulacrum,kobzol

Allow `./x test` to run tests without doc tests and without benchmarks

# Problem

For Ferrocene we would like to run only the `coretests` and `alloctests` test suites when measuring code coverage. Running `corebenches` and `allocbenches` would alter the numbers, which is not compliant with the certification.

This is currently not possible in bootstrap. By default `./x test` runs unit, integration and doc tests. `./x test --doc` only runs doc tests. So far, so good.

The problem is that while `./x test --no-doc` stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like `cargo test --all-targets`.

# Solution

This PR renames the existing `--no-doc` flag to `--all-targets` and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from `cargo --bins --examples --tests --benches` to `cargo --all-targets` because it will fail e.g. `./x test library/alloc --all-targets` with an error like "use of unstable library feature `test`".

Additionally, this PR add a `./x test --tests` flag (equivalent to `cargo test --tests`) that only executes unit and integration tests.

Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them.

Note that this PR does _not_ change the behaviour of either `./x test` or `./x test --doc`.

## Note on `test = true`

While this change enables bootstrap to run tests without doc tests and without benchmarks, executing `./x test library/core library/alloc --tests` will still build and execute `corebenches` and `allocbenches`.

What?! 😱 Why all of this effort to enable it then?

Good question! The reason they are still executed is, that they are marked with `test = true` in their respective `Cargo.toml` ([corebenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/coretests/Cargo.toml#L24), [allocbenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/alloctests/Cargo.toml#L32)).

@bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no `./x --all-targets` is passed. This is perfectly possible with this PR. Benchmarks marked with `test = true` will be executed when calling either of `./x test`, `./x test --tests` or `./x --all-targets`. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene.

## Open questions

### Error message

I added one commit that adds an error message if a user passes `--no-doc` and points them to `--all-targets` and `--tests`. I think it might be nice to have, but if you think it is not necessary, I can remove it.

# How to test this change

You can see the change in action by running `./x test library/alloc --tests` and `./x test library/alloc --all-targets`. The first will execute `alloctests` and `allocbenches` (which is marked with `test = true`), while the second will additionally run `benches/vec_deque_append.rs` (which is _not_ marked with `test = true`).
rust-bors bot pushed a commit that referenced this pull request Mar 10, 2026
…uwer

Rollup of 13 pull requests

Successful merges:

 - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`)
 - #152457 (Pass -pg to linker when using -Zinstrument-mcount)
 - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks)
 - #153471 (Refactor `ActiveJobGuard`)
 - #153595 (`QueryLatch` cleanups)
 - #153653 (scalable vector: type renames and simple checks)
 - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself)
 - #153479 (Add rationale for intentional potential_query_instability allows)
 - #153600 (add test for proc-macros with custom panic payloads)
 - #153641 (Move `Spanned`.)
 - #153643 (Avoid projection-only suggestions for inherent assoc types)
 - #153657 (triagebot: remove myself from some mention groups)
 - #153659 (Mark an unreachable match arm as such)
rust-bors bot pushed a commit that referenced this pull request Mar 10, 2026
…uwer

Rollup of 13 pull requests

Successful merges:

 - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`)
 - #152457 (Pass -pg to linker when using -Zinstrument-mcount)
 - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks)
 - #153471 (Refactor `ActiveJobGuard`)
 - #153595 (`QueryLatch` cleanups)
 - #153653 (scalable vector: type renames and simple checks)
 - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself)
 - #153479 (Add rationale for intentional potential_query_instability allows)
 - #153600 (add test for proc-macros with custom panic payloads)
 - #153641 (Move `Spanned`.)
 - #153643 (Avoid projection-only suggestions for inherent assoc types)
 - #153657 (triagebot: remove myself from some mention groups)
 - #153659 (Mark an unreachable match arm as such)
JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 10, 2026
…s, r=Mark-Simulacrum,kobzol

Allow `./x test` to run tests without doc tests and without benchmarks

# Problem

For Ferrocene we would like to run only the `coretests` and `alloctests` test suites when measuring code coverage. Running `corebenches` and `allocbenches` would alter the numbers, which is not compliant with the certification.

This is currently not possible in bootstrap. By default `./x test` runs unit, integration and doc tests. `./x test --doc` only runs doc tests. So far, so good.

The problem is that while `./x test --no-doc` stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like `cargo test --all-targets`.

# Solution

This PR renames the existing `--no-doc` flag to `--all-targets` and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from `cargo --bins --examples --tests --benches` to `cargo --all-targets` because it will fail e.g. `./x test library/alloc --all-targets` with an error like "use of unstable library feature `test`".

Additionally, this PR add a `./x test --tests` flag (equivalent to `cargo test --tests`) that only executes unit and integration tests.

Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them.

Note that this PR does _not_ change the behaviour of either `./x test` or `./x test --doc`.

## Note on `test = true`

While this change enables bootstrap to run tests without doc tests and without benchmarks, executing `./x test library/core library/alloc --tests` will still build and execute `corebenches` and `allocbenches`.

What?! 😱 Why all of this effort to enable it then?

Good question! The reason they are still executed is, that they are marked with `test = true` in their respective `Cargo.toml` ([corebenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/coretests/Cargo.toml#L24), [allocbenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/alloctests/Cargo.toml#L32)).

@bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no `./x --all-targets` is passed. This is perfectly possible with this PR. Benchmarks marked with `test = true` will be executed when calling either of `./x test`, `./x test --tests` or `./x --all-targets`. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene.

## Open questions

### Error message

I added one commit that adds an error message if a user passes `--no-doc` and points them to `--all-targets` and `--tests`. I think it might be nice to have, but if you think it is not necessary, I can remove it.

# How to test this change

You can see the change in action by running `./x test library/alloc --tests` and `./x test library/alloc --all-targets`. The first will execute `alloctests` and `allocbenches` (which is marked with `test = true`), while the second will additionally run `benches/vec_deque_append.rs` (which is _not_ marked with `test = true`).
rust-bors bot pushed a commit that referenced this pull request Mar 10, 2026
…uwer

Rollup of 14 pull requests

Successful merges:

 - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`)
 - #152457 (Pass -pg to linker when using -Zinstrument-mcount)
 - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks)
 - #153471 (Refactor `ActiveJobGuard`)
 - #153595 (`QueryLatch` cleanups)
 - #153653 (scalable vector: type renames and simple checks)
 - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself)
 - #153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization)
 - #153479 (Add rationale for intentional potential_query_instability allows)
 - #153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous)
 - #153600 (add test for proc-macros with custom panic payloads)
 - #153643 (Avoid projection-only suggestions for inherent assoc types)
 - #153657 (triagebot: remove myself from some mention groups)
 - #153659 (Mark an unreachable match arm as such)
rust-bors bot pushed a commit that referenced this pull request Mar 10, 2026
…uwer

Rollup of 13 pull requests

Successful merges:

 - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`)
 - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks)
 - #153471 (Refactor `ActiveJobGuard`)
 - #153595 (`QueryLatch` cleanups)
 - #153653 (scalable vector: type renames and simple checks)
 - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself)
 - #153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization)
 - #153479 (Add rationale for intentional potential_query_instability allows)
 - #153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous)
 - #153600 (add test for proc-macros with custom panic payloads)
 - #153643 (Avoid projection-only suggestions for inherent assoc types)
 - #153657 (triagebot: remove myself from some mention groups)
 - #153659 (Mark an unreachable match arm as such)
rust-bors bot pushed a commit that referenced this pull request Mar 10, 2026
…uwer

Rollup of 13 pull requests

Successful merges:

 - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`)
 - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks)
 - #153471 (Refactor `ActiveJobGuard`)
 - #153595 (`QueryLatch` cleanups)
 - #153653 (scalable vector: type renames and simple checks)
 - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself)
 - #153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization)
 - #153479 (Add rationale for intentional potential_query_instability allows)
 - #153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous)
 - #153600 (add test for proc-macros with custom panic payloads)
 - #153643 (Avoid projection-only suggestions for inherent assoc types)
 - #153657 (triagebot: remove myself from some mention groups)
 - #153659 (Mark an unreachable match arm as such)
@Zalathar
Copy link
Member

This change seems extremely confusing. From reading the description and the changes, I have no idea what flag to pass when I just want to run unit tests.

@rust-bors rust-bors bot merged commit 4c6755c into rust-lang:main Mar 11, 2026
11 checks passed
@rustbot rustbot added this to the 1.96.0 milestone Mar 11, 2026
rust-timer added a commit that referenced this pull request Mar 11, 2026
Rollup merge of #153143 - ferrocene:jh/bootstrap-test-targets, r=Mark-Simulacrum,kobzol

Allow `./x test` to run tests without doc tests and without benchmarks

# Problem

For Ferrocene we would like to run only the `coretests` and `alloctests` test suites when measuring code coverage. Running `corebenches` and `allocbenches` would alter the numbers, which is not compliant with the certification.

This is currently not possible in bootstrap. By default `./x test` runs unit, integration and doc tests. `./x test --doc` only runs doc tests. So far, so good.

The problem is that while `./x test --no-doc` stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like `cargo test --all-targets`.

# Solution

This PR renames the existing `--no-doc` flag to `--all-targets` and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from `cargo --bins --examples --tests --benches` to `cargo --all-targets` because it will fail e.g. `./x test library/alloc --all-targets` with an error like "use of unstable library feature `test`".

Additionally, this PR add a `./x test --tests` flag (equivalent to `cargo test --tests`) that only executes unit and integration tests.

Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them.

Note that this PR does _not_ change the behaviour of either `./x test` or `./x test --doc`.

## Note on `test = true`

While this change enables bootstrap to run tests without doc tests and without benchmarks, executing `./x test library/core library/alloc --tests` will still build and execute `corebenches` and `allocbenches`.

What?! 😱 Why all of this effort to enable it then?

Good question! The reason they are still executed is, that they are marked with `test = true` in their respective `Cargo.toml` ([corebenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/coretests/Cargo.toml#L24), [allocbenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/alloctests/Cargo.toml#L32)).

@bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no `./x --all-targets` is passed. This is perfectly possible with this PR. Benchmarks marked with `test = true` will be executed when calling either of `./x test`, `./x test --tests` or `./x --all-targets`. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene.

## Open questions

### Error message

I added one commit that adds an error message if a user passes `--no-doc` and points them to `--all-targets` and `--tests`. I think it might be nice to have, but if you think it is not necessary, I can remove it.

# How to test this change

You can see the change in action by running `./x test library/alloc --tests` and `./x test library/alloc --all-targets`. The first will execute `alloctests` and `allocbenches` (which is marked with `test = true`), while the second will additionally run `benches/vec_deque_append.rs` (which is _not_ marked with `test = true`).
@Kobzol
Copy link
Member

Kobzol commented Mar 11, 2026

Part of the confusion is sadly inherited from Cargo. For unit tests only, that would be x test --tests (note that this wasn't possible before this PR).

@tshepang tshepang deleted the jh/bootstrap-test-targets branch March 11, 2026 07:46
Delta17920 pushed a commit to Delta17920/rust that referenced this pull request Mar 11, 2026
…nathanBrouwer

Rollup of 13 pull requests

Successful merges:

 - rust-lang#149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`)
 - rust-lang#153143 (Allow `./x test` to run tests without doc tests and without benchmarks)
 - rust-lang#153471 (Refactor `ActiveJobGuard`)
 - rust-lang#153595 (`QueryLatch` cleanups)
 - rust-lang#153653 (scalable vector: type renames and simple checks)
 - rust-lang#152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself)
 - rust-lang#153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization)
 - rust-lang#153479 (Add rationale for intentional potential_query_instability allows)
 - rust-lang#153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous)
 - rust-lang#153600 (add test for proc-macros with custom panic payloads)
 - rust-lang#153643 (Avoid projection-only suggestions for inherent assoc types)
 - rust-lang#153657 (triagebot: remove myself from some mention groups)
 - rust-lang#153659 (Mark an unreachable match arm as such)
@Urhengulas
Copy link
Contributor Author

Urhengulas commented Mar 11, 2026

@Zalathar said:

This change seems extremely confusing. From reading the description and the changes, I have no idea what flag to pass when I just want to run unit tests.

I get you! I was confused a lot as well, because the behaviour of --no-doc was not what I expected.

After the PR it should be more clear, now ./x test has 4 modes which are equivalent to cargo test:

  1. ./x test: Runs unit, integration and doc tests (same as before)
  2. ./x test --doc: Runs only doc tests (same as before)
  3. ./x test --tests: Runs unit and integration tests (new! this was not possible before)
  4. ./x test --all-targets: Runs unit, integration tests, benches, examples, bins (not actually new, this was the behaviour of --no-doc before)

@RalfJung
Copy link
Member

RalfJung commented Mar 11, 2026

So IOW, --no-doc didn't actually skip doc tests before, and the new way to skip doc tests is --tests? Weird, but I guess that is inherited from cargo. The PR description could make this a lot more clear though. Currently it's a lot of text, hard to find the little bits of relevant information in there.

This may explain recent confusion I had where it built rustdoc even though I passed --no-doc. Does bootstrap now skip building rustdoc if I pass --tests?

@@ -64,7 +64,7 @@ check-aux:
library/core \
library/alloc \
$(BOOTSTRAP_ARGS) \
--no-doc
--all-targets
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was actually meant to skip the doctests. Some doctests don't work without -Zmiri-disable-isolation. I have no idea why this does not cause CI failures...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The non-bootstrap version of this uses cargo test --lib --tests which apparently is not equivalent to cargo test --tests. Is there a way to do the same from bootstrap?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RalfJung If it should only execute unit and integration tests (and no doc tests, no benchmarks, no examples and no bins) then you can use --tests.

@RalfJung
Copy link
Member

RalfJung commented Mar 11, 2026

@Urhengulas your summary is incorrect... from the cargo docs
--all-targets Test all targets (does not include doctests)

I also confirmed that --no-doc did indeed exclude doc tests before your change. So a lot of what you say just doesn't make sense to me.

EDIT: Ah, I was just thrown off by your summary at the end which claims that --all-targets also runs doc tests. The PR description explains it properly:

The problem is that while ./x test --no-doc stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well)

This PR doesn't really fix that, it just replaces one sort of confusion by another by renaming the flag from --no-doc to --all-targets. I guess it makes sense to be consistent with cargo.

@Urhengulas
Copy link
Contributor Author

@RalfJung my summary was wrong! before --no-doc did not execute doc tests and now --all-targets does not execute doc tests. Sorry for that confusion!

The problem is that while ./x test --no-doc stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well)

This PR doesn't really fix that, it just replaces one sort of confusion by another by renaming the flag from --no-doc to --all-targets. I guess it makes sense to be consistent with cargo.

In my perspective it is now possible to use --tests if you want tests but not benchmarks and not doc tests. And if you want tests and benchmarks you can use --all-targets. And this is in line with how cargo test behaves.

@Urhengulas
Copy link
Contributor Author

Anyways I realised I made a mistake in adding backwards compatibility for --no-doc. While it should keep behaving like ./x test --all-targets I mistakenly implemented it to behave like ./x test. I am fixing this in #153725.

JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 11, 2026
miri-test-libstd: use --tests and update some comments

rust-lang#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`.

Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 11, 2026
…Kobzol

Fix that `./x test --no-doc` actually keeps the same behaviour for backwards compatability

In rust-lang#153143 the `./x test --no-doc` flag was renamed to `--all-targets`.  I added a commit that keeps the `--no-doc` flag for backwards compatibility, but unfortunately I forgot to actually keep the behaviour the same, which is fixed by this PR.

r? @Kobzol
JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 11, 2026
miri-test-libstd: use --tests and update some comments

rust-lang#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`.

Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
JonathanBrouwer added a commit to JonathanBrouwer/rust that referenced this pull request Mar 11, 2026
miri-test-libstd: use --tests and update some comments

rust-lang#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`.

Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
github-actions bot pushed a commit to rust-lang/miri that referenced this pull request Mar 12, 2026
miri-test-libstd: use --tests and update some comments

rust-lang/rust#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`.

Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang/rust@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-rustc-dev-guide Area: rustc-dev-guide A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-libs Relevant to the library team, which will review and decide on the PR/issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants