Allow ./x test to run tests without doc tests and without benchmarks#153143
Allow ./x test to run tests without doc tests and without benchmarks#153143rust-bors[bot] merged 3 commits intorust-lang:mainfrom
./x test to run tests without doc tests and without benchmarks#153143Conversation
|
The rustc-dev-guide subtree was changed. If this PR only touches the dev guide consider submitting a PR directly to rust-lang/rustc-dev-guide otherwise thank you for updating the dev guide with your changes. |
|
r? @jieyouxu rustbot has assigned @jieyouxu. Use Why was this reviewer chosen?The reviewer was selected based on:
|
772d7dc to
52327a7
Compare
|
I am pretty sure that following workaround will also become unnecessary when running rust/library/alloctests/benches/vec_deque_append.rs Lines 9 to 13 in 3a70d03 |
52327a7 to
5bd3628
Compare
|
@rustbot reroll |
|
r=me with commits squashed a bit (or fully) |
|
Reminder, once the PR becomes ready for a review, use |
5f8d29e to
e6f3e44
Compare
|
Sorry for the late notice, but not that I re-read the changed if conditions, I think that |
@Kobzol Sounds good to me! Will rename it right now :) |
|
@Mark-Simulacrum I also squashed the commits into two |
e6f3e44 to
38d5071
Compare
|
@Kobzol I force pushed the rename (e6f3e447..38d5071b) |
|
Since it was only a CLI flag and not a config option, I guess I'm fine with removing the flag; not sure what's the general policy on keeping backwards compatibility with flags vs config options. I'll ask on Zulip though. |
- `./x run generate-completions` - Search and replace `--no-doc` with `--all-targets` This is to keep the behaviour the same. - Document `./x test --tests` in `rustc-dev-guide` - Add change info
…s, r=Mark-Simulacrum,kobzol Allow `./x test` to run tests without doc tests and without benchmarks # Problem For Ferrocene we would like to run only the `coretests` and `alloctests` test suites when measuring code coverage. Running `corebenches` and `allocbenches` would alter the numbers, which is not compliant with the certification. This is currently not possible in bootstrap. By default `./x test` runs unit, integration and doc tests. `./x test --doc` only runs doc tests. So far, so good. The problem is that while `./x test --no-doc` stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like `cargo test --all-targets`. # Solution This PR renames the existing `--no-doc` flag to `--all-targets` and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from `cargo --bins --examples --tests --benches` to `cargo --all-targets` because it will fail e.g. `./x test library/alloc --all-targets` with an error like "use of unstable library feature `test`". Additionally, this PR add a `./x test --tests` flag (equivalent to `cargo test --tests`) that only executes unit and integration tests. Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them. Note that this PR does _not_ change the behaviour of either `./x test` or `./x test --doc`. ## Note on `test = true` While this change enables bootstrap to run tests without doc tests and without benchmarks, executing `./x test library/core library/alloc --tests` will still build and execute `corebenches` and `allocbenches`. What?! 😱 Why all of this effort to enable it then? Good question! The reason they are still executed is, that they are marked with `test = true` in their respective `Cargo.toml` ([corebenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/coretests/Cargo.toml#L24), [allocbenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/alloctests/Cargo.toml#L32)). @bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no `./x --all-targets` is passed. This is perfectly possible with this PR. Benchmarks marked with `test = true` will be executed when calling either of `./x test`, `./x test --tests` or `./x --all-targets`. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene. ## Open questions ### Error message I added one commit that adds an error message if a user passes `--no-doc` and points them to `--all-targets` and `--tests`. I think it might be nice to have, but if you think it is not necessary, I can remove it. # How to test this change You can see the change in action by running `./x test library/alloc --tests` and `./x test library/alloc --all-targets`. The first will execute `alloctests` and `allocbenches` (which is marked with `test = true`), while the second will additionally run `benches/vec_deque_append.rs` (which is _not_ marked with `test = true`).
…uwer Rollup of 13 pull requests Successful merges: - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`) - #152457 (Pass -pg to linker when using -Zinstrument-mcount) - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks) - #153471 (Refactor `ActiveJobGuard`) - #153595 (`QueryLatch` cleanups) - #153653 (scalable vector: type renames and simple checks) - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself) - #153479 (Add rationale for intentional potential_query_instability allows) - #153600 (add test for proc-macros with custom panic payloads) - #153641 (Move `Spanned`.) - #153643 (Avoid projection-only suggestions for inherent assoc types) - #153657 (triagebot: remove myself from some mention groups) - #153659 (Mark an unreachable match arm as such)
…uwer Rollup of 13 pull requests Successful merges: - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`) - #152457 (Pass -pg to linker when using -Zinstrument-mcount) - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks) - #153471 (Refactor `ActiveJobGuard`) - #153595 (`QueryLatch` cleanups) - #153653 (scalable vector: type renames and simple checks) - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself) - #153479 (Add rationale for intentional potential_query_instability allows) - #153600 (add test for proc-macros with custom panic payloads) - #153641 (Move `Spanned`.) - #153643 (Avoid projection-only suggestions for inherent assoc types) - #153657 (triagebot: remove myself from some mention groups) - #153659 (Mark an unreachable match arm as such)
…s, r=Mark-Simulacrum,kobzol Allow `./x test` to run tests without doc tests and without benchmarks # Problem For Ferrocene we would like to run only the `coretests` and `alloctests` test suites when measuring code coverage. Running `corebenches` and `allocbenches` would alter the numbers, which is not compliant with the certification. This is currently not possible in bootstrap. By default `./x test` runs unit, integration and doc tests. `./x test --doc` only runs doc tests. So far, so good. The problem is that while `./x test --no-doc` stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like `cargo test --all-targets`. # Solution This PR renames the existing `--no-doc` flag to `--all-targets` and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from `cargo --bins --examples --tests --benches` to `cargo --all-targets` because it will fail e.g. `./x test library/alloc --all-targets` with an error like "use of unstable library feature `test`". Additionally, this PR add a `./x test --tests` flag (equivalent to `cargo test --tests`) that only executes unit and integration tests. Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them. Note that this PR does _not_ change the behaviour of either `./x test` or `./x test --doc`. ## Note on `test = true` While this change enables bootstrap to run tests without doc tests and without benchmarks, executing `./x test library/core library/alloc --tests` will still build and execute `corebenches` and `allocbenches`. What?! 😱 Why all of this effort to enable it then? Good question! The reason they are still executed is, that they are marked with `test = true` in their respective `Cargo.toml` ([corebenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/coretests/Cargo.toml#L24), [allocbenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/alloctests/Cargo.toml#L32)). @bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no `./x --all-targets` is passed. This is perfectly possible with this PR. Benchmarks marked with `test = true` will be executed when calling either of `./x test`, `./x test --tests` or `./x --all-targets`. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene. ## Open questions ### Error message I added one commit that adds an error message if a user passes `--no-doc` and points them to `--all-targets` and `--tests`. I think it might be nice to have, but if you think it is not necessary, I can remove it. # How to test this change You can see the change in action by running `./x test library/alloc --tests` and `./x test library/alloc --all-targets`. The first will execute `alloctests` and `allocbenches` (which is marked with `test = true`), while the second will additionally run `benches/vec_deque_append.rs` (which is _not_ marked with `test = true`).
…uwer Rollup of 14 pull requests Successful merges: - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`) - #152457 (Pass -pg to linker when using -Zinstrument-mcount) - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks) - #153471 (Refactor `ActiveJobGuard`) - #153595 (`QueryLatch` cleanups) - #153653 (scalable vector: type renames and simple checks) - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself) - #153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization) - #153479 (Add rationale for intentional potential_query_instability allows) - #153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous) - #153600 (add test for proc-macros with custom panic payloads) - #153643 (Avoid projection-only suggestions for inherent assoc types) - #153657 (triagebot: remove myself from some mention groups) - #153659 (Mark an unreachable match arm as such)
…uwer Rollup of 13 pull requests Successful merges: - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`) - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks) - #153471 (Refactor `ActiveJobGuard`) - #153595 (`QueryLatch` cleanups) - #153653 (scalable vector: type renames and simple checks) - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself) - #153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization) - #153479 (Add rationale for intentional potential_query_instability allows) - #153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous) - #153600 (add test for proc-macros with custom panic payloads) - #153643 (Avoid projection-only suggestions for inherent assoc types) - #153657 (triagebot: remove myself from some mention groups) - #153659 (Mark an unreachable match arm as such)
…uwer Rollup of 13 pull requests Successful merges: - #149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`) - #153143 (Allow `./x test` to run tests without doc tests and without benchmarks) - #153471 (Refactor `ActiveJobGuard`) - #153595 (`QueryLatch` cleanups) - #153653 (scalable vector: type renames and simple checks) - #152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself) - #153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization) - #153479 (Add rationale for intentional potential_query_instability allows) - #153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous) - #153600 (add test for proc-macros with custom panic payloads) - #153643 (Avoid projection-only suggestions for inherent assoc types) - #153657 (triagebot: remove myself from some mention groups) - #153659 (Mark an unreachable match arm as such)
|
This change seems extremely confusing. From reading the description and the changes, I have no idea what flag to pass when I just want to run unit tests. |
Rollup merge of #153143 - ferrocene:jh/bootstrap-test-targets, r=Mark-Simulacrum,kobzol Allow `./x test` to run tests without doc tests and without benchmarks # Problem For Ferrocene we would like to run only the `coretests` and `alloctests` test suites when measuring code coverage. Running `corebenches` and `allocbenches` would alter the numbers, which is not compliant with the certification. This is currently not possible in bootstrap. By default `./x test` runs unit, integration and doc tests. `./x test --doc` only runs doc tests. So far, so good. The problem is that while `./x test --no-doc` stops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves like `cargo test --all-targets`. # Solution This PR renames the existing `--no-doc` flag to `--all-targets` and keeps the same behaviour as before. Unfortunately it is not possible to internally switch from `cargo --bins --examples --tests --benches` to `cargo --all-targets` because it will fail e.g. `./x test library/alloc --all-targets` with an error like "use of unstable library feature `test`". Additionally, this PR add a `./x test --tests` flag (equivalent to `cargo test --tests`) that only executes unit and integration tests. Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them. Note that this PR does _not_ change the behaviour of either `./x test` or `./x test --doc`. ## Note on `test = true` While this change enables bootstrap to run tests without doc tests and without benchmarks, executing `./x test library/core library/alloc --tests` will still build and execute `corebenches` and `allocbenches`. What?! 😱 Why all of this effort to enable it then? Good question! The reason they are still executed is, that they are marked with `test = true` in their respective `Cargo.toml` ([corebenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/coretests/Cargo.toml#L24), [allocbenches](https://github.com/rust-lang/rust/blob/3f9853562c73af38a5e6af8b0da1b2734a327e19/library/alloctests/Cargo.toml#L32)). @bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no `./x --all-targets` is passed. This is perfectly possible with this PR. Benchmarks marked with `test = true` will be executed when calling either of `./x test`, `./x test --tests` or `./x --all-targets`. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene. ## Open questions ### Error message I added one commit that adds an error message if a user passes `--no-doc` and points them to `--all-targets` and `--tests`. I think it might be nice to have, but if you think it is not necessary, I can remove it. # How to test this change You can see the change in action by running `./x test library/alloc --tests` and `./x test library/alloc --all-targets`. The first will execute `alloctests` and `allocbenches` (which is marked with `test = true`), while the second will additionally run `benches/vec_deque_append.rs` (which is _not_ marked with `test = true`).
|
Part of the confusion is sadly inherited from Cargo. For unit tests only, that would be |
…nathanBrouwer Rollup of 13 pull requests Successful merges: - rust-lang#149130 (Implement coercions between `&pin (mut|const) T` and `&(mut) T` when `T: Unpin`) - rust-lang#153143 (Allow `./x test` to run tests without doc tests and without benchmarks) - rust-lang#153471 (Refactor `ActiveJobGuard`) - rust-lang#153595 (`QueryLatch` cleanups) - rust-lang#153653 (scalable vector: type renames and simple checks) - rust-lang#152302 (fix: don't suggest replacing `env!("CARGO_BIN_NAME")` with itself) - rust-lang#153283 (feat(rustdoc-json): Add optional support for rkyv (de)serialization) - rust-lang#153479 (Add rationale for intentional potential_query_instability allows) - rust-lang#153533 (Fix LegacyKeyValueFormat report from docker build: miscellaneous) - rust-lang#153600 (add test for proc-macros with custom panic payloads) - rust-lang#153643 (Avoid projection-only suggestions for inherent assoc types) - rust-lang#153657 (triagebot: remove myself from some mention groups) - rust-lang#153659 (Mark an unreachable match arm as such)
|
@Zalathar said:
I get you! I was confused a lot as well, because the behaviour of After the PR it should be more clear, now
|
|
|
| @@ -64,7 +64,7 @@ check-aux: | |||
| library/core \ | |||
| library/alloc \ | |||
| $(BOOTSTRAP_ARGS) \ | |||
| --no-doc | |||
| --all-targets | |||
There was a problem hiding this comment.
This was actually meant to skip the doctests. Some doctests don't work without -Zmiri-disable-isolation. I have no idea why this does not cause CI failures...
There was a problem hiding this comment.
The non-bootstrap version of this uses cargo test --lib --tests which apparently is not equivalent to cargo test --tests. Is there a way to do the same from bootstrap?
There was a problem hiding this comment.
@RalfJung If it should only execute unit and integration tests (and no doc tests, no benchmarks, no examples and no bins) then you can use --tests.
|
@Urhengulas your summary is incorrect... from the cargo docs I also confirmed that EDIT: Ah, I was just thrown off by your summary at the end which claims that
This PR doesn't really fix that, it just replaces one sort of confusion by another by renaming the flag from |
|
@RalfJung my summary was wrong! before
In my perspective it is now possible to use |
|
Anyways I realised I made a mistake in adding backwards compatibility for |
miri-test-libstd: use --tests and update some comments rust-lang#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`. Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
…Kobzol Fix that `./x test --no-doc` actually keeps the same behaviour for backwards compatability In rust-lang#153143 the `./x test --no-doc` flag was renamed to `--all-targets`. I added a commit that keeps the `--no-doc` flag for backwards compatibility, but unfortunately I forgot to actually keep the behaviour the same, which is fixed by this PR. r? @Kobzol
miri-test-libstd: use --tests and update some comments rust-lang#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`. Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
miri-test-libstd: use --tests and update some comments rust-lang#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`. Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
miri-test-libstd: use --tests and update some comments rust-lang/rust#153143 added `./x test --tests` matching `cargo --tests`, which is exactly what Miri wants when testing the standard library. So let's use it for that. We can then also remove a hack in `library/alloctests/benches/vec_deque_append.rs`. Also update the comment for why the other benchmarks still need to be disabled in Miri, and remove some `cfg_attr` that seem unnecessary since the entire crate that contains them is already disabled in Miri. Those were copied over in rust-lang/rust@b8fa843 -- they used to be needed since benches and tests were in the same crate, but they aren't any more.
View all comments
Problem
For Ferrocene we would like to run only the
coretestsandallocteststest suites when measuring code coverage. Runningcorebenchesandallocbencheswould alter the numbers, which is not compliant with the certification.This is currently not possible in bootstrap. By default
./x testruns unit, integration and doc tests../x test --doconly runs doc tests. So far, so good.The problem is that while
./x test --no-docstops bootstrap from executing doc tests, it surprisingly starts executing benchmarks (next to examples and bins as well). It basically behaves likecargo test --all-targets.Solution
This PR renames the existing
--no-docflag to--all-targetsand keeps the same behaviour as before. Unfortunately it is not possible to internally switch fromcargo --bins --examples --tests --benchestocargo --all-targetsbecause it will fail e.g../x test library/alloc --all-targetswith an error like "use of unstable library featuretest".Additionally, this PR add a
./x test --testsflag (equivalent tocargo test --tests) that only executes unit and integration tests.Both changes we are doing in https://github.com/ferrocene/ferrocene anyways, but believe that they may be useful for other people as well and therefore would like to contribute them.
Note that this PR does not change the behaviour of either
./x testor./x test --doc.Note on
test = trueWhile this change enables bootstrap to run tests without doc tests and without benchmarks, executing
./x test library/core library/alloc --testswill still build and executecorebenchesandallocbenches.What?! 😱 Why all of this effort to enable it then?
Good question! The reason they are still executed is, that they are marked with
test = truein their respectiveCargo.toml(corebenches, allocbenches).@bjorn3 mentioned that it is important for upstream that those benchmarks are executed by default, even if no
./x --all-targetsis passed. This is perfectly possible with this PR. Benchmarks marked withtest = truewill be executed when calling either of./x test,./x test --testsor./x --all-targets. Therefore this PR does not include a commit to change that. We will just do this change in https://github.com/ferrocene/ferrocene.Open questions
Error message
I added one commit that adds an error message if a user passes
--no-docand points them to--all-targetsand--tests. I think it might be nice to have, but if you think it is not necessary, I can remove it.How to test this change
You can see the change in action by running
./x test library/alloc --testsand./x test library/alloc --all-targets. The first will executealloctestsandallocbenches(which is marked withtest = true), while the second will additionally runbenches/vec_deque_append.rs(which is not marked withtest = true).