[Version 11.0] Feature support for UTF-8 string literals

[RexJaeschke]

This is Rex's adaptation of the corresponding MS proposal.

Back in 2025, when I was researching this PR, I created PR #1103 to tidy up a bunch of Unicode-related things. That PR is complimentary to this one and the two do not overlap.

Note 1 re overlap with raw string literal PR

Some spec changes required to support this feature touch the same section(s) as does PR #1599 for V11’s raw string literals. We need to make sure they coexist, depending on which of the two gets merged first.

What is missing from this PR and needs to be added once the raw string PR has been added is the addition of the two Utf8_Suffixes, as shown below:

fragment Single_Line_Raw_String_Literal
    : Raw_String_Literal_Delimiter  Raw_String_Literal Content
      Raw_String_Literal_Delimiter >>>Utf8_Suffix?<<<
    ;

fragment Multi_Line_Raw_String_Literal
    : Raw_String_Literal_Delimiter Whitespace* New_Line
      (Raw_String_Literal Content | New_Line)* New_Line
      Whitespace* Raw_String_Literal_Delimiter >>>Utf8_Suffix?<<<
    ;

Note 2 re Lowering

Re the “Lowering” section of the MS proposal, that appears to be about implementation details. In any event, Rex did not add anything from there to the spec.

Note 3 re “Unresolved questions”

Re the “Unresolved questions” section of the MS proposal, no text was added w.r.t implicit conversion from string. Should something be added?

[KalleOlaviNiemitalo]

allows ref struct is added in C# 13, and this PR is for C# 11, so that part is OK…

But the wording is imprecise. A string literal is syntax rather than a type so it couldn't be used as a type parameter (or rather a type argument) anyway.

[KalleOlaviNiemitalo]

From this normative text, it is not clear whether redundant parentheses are allowed.

using System;
public class C {
    public void M() {
        ReadOnlySpan<byte> a
            = (( "A"u8 + "a"u8 ))
            + (( "B"u8 + "b"u8 ));
    }
}

[KalleOlaviNiemitalo]

Should add a new subclause within 16.5.15 Safe context constraint, saying that when a UTF-8 string literal has a ReadOnlySpan<byte> value, the safe-context of that value is caller-context. That would allow it to be returned out of the function, like so:

using System;
public class C {
    public ReadOnlySpan<byte> M() {
        return "xyz"u8;
    }
}

Concatenation of UTF-8 string literals would then follow the safe-context rule in 16.5.15.5 Operators. As the safe-context of both operands would be caller-context, the safe-context of the result would be the same, without having to be specified separately for this case.

[KalleOlaviNiemitalo]

The following would then likewise be valid:

public class C {
    public ref readonly byte M() {
        return ref "xyz"u8[0];
    }
}