Automatically enable cross-crate inlining for small functions

[saethlin]

This is basically reviving #70550

The #[inline] attribute can have a significant impact on code generation or runtime performance (because it enables inlining between CGUs where it would normally not happen) and also on compile-time performance (because it enables MIR inlining). But it has to be added manually, which is awkward.

This PR factors whether a DefId is cross-crate inlinable into a query, and replaces all uses of CodegenFnAttrs::requests_inline with this new query. The new query incorporates all the other logic that is used to determine whether a Def should be treated as cross-crate-inlinable, and as a last step inspects the function's optimized_mir to determine if it should be treated as cross-crate-inlinable.

The heuristic implemented here is deliberately conservative; we only infer inlinability for functions whose optimized_mir does not contain any calls or asserts. I plan to study adjusting the cost model later, but for now the compile time implications of this change are so significant that I think this very crude heuristic is well worth landing.

[rustbot]

r? @petrochenkov

(rustbot has picked a reviewer for you, use r? to override)

[saethlin]

@bors try @rust-timer queue

[bors]

⌛ Trying commit 64f45aa with merge f1cbf12...

[bors]

☀️ Try build successful - checks-actions
Build commit: f1cbf12 (f1cbf125bf7c5bdb502d2c6359397a8d7e4fe068)

[rust-timer]

Finished benchmarking commit (f1cbf12): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	28.5%	[0.2%, 610.4%]	102
Regressions ❌ (secondary)	62.8%	[0.3%, 2246.9%]	66
Improvements ✅ (primary)	-3.8%	[-56.5%, -0.2%]	136
Improvements ✅ (secondary)	-7.5%	[-85.5%, -0.3%]	151
All ❌✅ (primary)	10.1%	[-56.5%, 610.4%]	238

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	11.3%	[0.5%, 72.7%]	90
Regressions ❌ (secondary)	16.3%	[0.7%, 100.1%]	86
Improvements ✅ (primary)	-5.7%	[-15.3%, -0.9%]	9
Improvements ✅ (secondary)	-5.0%	[-18.0%, -0.7%]	36
All ❌✅ (primary)	9.8%	[-15.3%, 72.7%]	99

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	32.5%	[0.8%, 565.9%]	98
Regressions ❌ (secondary)	70.9%	[1.4%, 1900.7%]	64
Improvements ✅ (primary)	-14.9%	[-60.4%, -0.9%]	27
Improvements ✅ (secondary)	-12.5%	[-85.1%, -1.9%]	78
All ❌✅ (primary)	22.2%	[-60.4%, 565.9%]	125

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	4.5%	[0.2%, 29.4%]	107
Regressions ❌ (secondary)	6.8%	[0.1%, 70.1%]	59
Improvements ✅ (primary)	-6.9%	[-30.5%, -0.1%]	56
Improvements ✅ (secondary)	-14.8%	[-77.0%, -2.5%]	83
All ❌✅ (primary)	0.5%	[-30.5%, 29.4%]	163

Bootstrap: 625.502s -> 742.509s (18.71%)
Artifact size: 270.64 MiB -> 275.71 MiB (1.87%)

[saethlin]

This should be less silly than the last one because now we don't do the per-CGU thing for incr builds. And I also halved the default threshold.

@bors try @rust-timer queue

[bors]

⌛ Trying commit aac4020 with merge 26bbeca...

[bors]

☀️ Try build successful - checks-actions
Build commit: 26bbeca (26bbeca8f2e9d38e46c985ab930a46c608dd85f8)

[rust-timer]

Finished benchmarking commit (26bbeca): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	21.8%	[0.2%, 346.9%]	153
Regressions ❌ (secondary)	32.7%	[0.2%, 456.2%]	43
Improvements ✅ (primary)	-2.7%	[-40.3%, -0.2%]	86
Improvements ✅ (secondary)	-5.3%	[-26.4%, -0.1%]	157
All ❌✅ (primary)	13.0%	[-40.3%, 346.9%]	239

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	9.3%	[0.5%, 71.9%]	106
Regressions ❌ (secondary)	6.7%	[0.5%, 32.1%]	45
Improvements ✅ (primary)	-14.0%	[-21.1%, -3.1%]	4
Improvements ✅ (secondary)	-3.6%	[-6.5%, -0.4%]	12
All ❌✅ (primary)	8.4%	[-21.1%, 71.9%]	110

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	25.5%	[0.9%, 325.3%]	137
Regressions ❌ (secondary)	34.1%	[1.9%, 350.8%]	42
Improvements ✅ (primary)	-12.1%	[-37.9%, -0.9%]	15
Improvements ✅ (secondary)	-8.1%	[-20.6%, -1.2%]	96
All ❌✅ (primary)	21.8%	[-37.9%, 325.3%]	152

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	12.3%	[0.0%, 77.9%]	116
Regressions ❌ (secondary)	6.0%	[0.1%, 52.2%]	56
Improvements ✅ (primary)	-3.9%	[-16.6%, -0.3%]	46
Improvements ✅ (secondary)	-8.9%	[-19.0%, -0.5%]	83
All ❌✅ (primary)	7.7%	[-16.6%, 77.9%]	162

Bootstrap: 622.673s -> 704.606s (13.16%)
Artifact size: 270.68 MiB -> 273.48 MiB (1.04%)

[briansmith]

With this change, is there any convenient way for a crate to indicate that it should be compiled with full optimizations but none of its public API should be inlined unless marked #[inline] specifically? Or do we have to denote every single function as #[inline(never)]? And is #[inline(never)] on a function sufficient to ensure it won't ever be inlined into callers?

[saethlin]

And is #[inline(never)] on a function sufficient to ensure it won't ever be inlined into callers?

That attribute is documented as a hint, so no. I do not think we have a language feature that guarantees what you are asking for.

It seems like a plausible thing to add but that's a bit outside my area.