No niche optimization for enum {One(Enum), Two(Enum, Enum)}

[lyphyser]

Rust seems to fail to take advantage of niches for very simple cases where it seems like it would be trivial to do so.

#[repr(u8)]
pub enum Bar {A = 0, B = 1}

enum Foo {
    Two(Bar, Bar),
    One(Bar)
}

fn main() {
    println!("{}", std::mem::size_of::<Foo>());
}

This code gives a size of 3 for Foo while obviously it can be stored in 2 bytes, representing the Two variant as two consecutive byte-sized Bar values, and the One variant as a byte with any value other than 0 or 1 followed by a byte-size Bar value.

Using an array of two values for the Two variant, specifying non-zero discriminants for Bar and and using NonZeroU8 all seem to have no effect.

[scottmcm]

Here's a simpler case that still doesn't optimize either: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=170f393d830719b60798a5a9dd1e963c

#[repr(u8)]
pub enum Bar {A = 0, B = 1}

#[repr(u8)]
enum AlwaysTwo { Two = 2 }

enum Foo {
    Two(Bar, Bar),
    One(AlwaysTwo, Bar),
}

fn main() {
    println!("{}", std::mem::size_of::<Foo>());
}

(With just One(Bar) there's a question of whether the rest is undef, which would keep it from being usable for discriminant storage.)

[erikdesjardins]

Duplicate of #46213.

The latest entry in the saga of trying to fix this is #75866, which was closed due to perf regressions. (Niche extraction is much more complex than loading a discriminant--and to a degree, unnecessarily so.) nox was working on improvements to match codegen which may unblock it.

(With just One(Bar) there's a question of whether the rest is undef, which would keep it from being usable for discriminant storage.)

I don't think that issue applies in this case. What we can't do is use padding in another type to store the discriminant, because it's not guaranteed to be preserved when manipulating that type on its own.

It is, however, okay to use padding within enum variants themselves to store the discriminant. Which is less "using the padding" and more "replacing would-be padding in the layout with the field holding the discriminant". (edit: not true, 2nd/3rd variants below are still treated as zero size. This is actually a stronger argument for it being allowed though. :P) That is what we currently do for enums like the following; nothing significantly changes when the other variants have non-zero size:

enum NoPayload {
    Two(Bar, Bar),
    Zero1,
    Zero2,
}

[dtolnay]

Closing as a duplicate of #46213.