[1.38] `#[bench]` accepted on stable but not beta

[CAD97]

Initially reported by @e-oz here.

[playground]

fn main() {
    println!("Hello, world!");
}

#[bench]
fn example() {}

On stable, you're allowed to define #[bench] tests in your library, even if calling them requires nightly. This broke at least one stable user, so we should at least consider continuing to ignore #[bench]-annotated functions when not run via cargo bench.

Is there some way to get a crater run to determine which libraries are using #[bench] in an otherwise stable crate, so we can suggest that they move benchmarks into a dedicated folder?

[djrenren]

I think it's related to this: #62048

Unsure how it got marked stable in the first place.

[ehuss]

[bench] was unstabilized in #62507. A crater run was done during that PR, the conclusion was written here. I think it was deemed unusual enough to proceed.
cc @petrochenkov

[petrochenkov]

Reverting the unstabilization and making a beta backport is always an option, it's a one line change.

Complaints filed so far:
#62507 (comment) - route-recognizer broke, apparently it's used as a dependency somewhat commonly.
#50297 (comment) and below - unspecified dependency broke, may be the case of route-recognizer as well.
iron/iron#613 - iron/router doesn't build, again cause by route-recognizer

route-recognizer was fixed in http-rs/route-recognizer#25 and a new patch version was published.
@e-oz does this fix your use case?

[Centril]

I don't think we should re-stabilize especially given that some folks have already migrated.
(The one line change is not the issue, but forever committing to a stable #[bench].)

[CAD97]

I wish we had unsurpressable future-compat warnings so a cycle's worth of just a future-compat warning for #[bench] would actually be helpful for catching any other libraries mistakenly using (because it was stable that way) #[bench] as #[cfg(test)] #[bench].

Because we don't, I assume said grace period wouldn't help find any remaining uses, just delay the errorification.

So I guess the question is: #[bench] has been (mistakenly?) stable as #[cfg(false)] for standard builds since Rust 1.0. Is that enough to force us into supporting it for nominal stability's sake, or do we consider it a long-standing low priority bug that just got fixed?

[emilyalbini]

Accidental stabilizations are not part of our stability guarantee. Still, if possible I think we should try to avoid breaking crates.

[Centril]

I wish we had unsurpressable future-compat warnings so a cycle's worth of just a future-compat warning for #[bench] would actually be helpful for catching any other libraries mistakenly using (because it was stable that way) #[bench] as #[cfg(test)] #[bench].

Implemented in #59658; I'll be picking this up shortly to address the concerns raised there re. spam.

[petrochenkov]

@CAD97
If I'm reading your comment correctly, it has some misunderstanding.
#[cfg(false)] #[bench] still can be used on stable because bench is unconfigured before use.

What was unstabilized is the actual use of bench.

#[bench]
fn foo() {}

It previously didn't compile in --test mode due to requiring other unstable things in the fn signature, but without --test it compiled successfully and expanded into nothing (empty token stream / AST piece).

[petrochenkov]

I'm not sure the back compat warning is a good solution since #[bench] doesn't seem to be deprecated in any way, on the contrary, it's going for stabilization with custom test frameworks making progress.

[CAD97]

@petrochenkov That's just a wording issue. I meant that using #[bench] had the effect of using #[cfg(test)] #[bench].