Create a `proc_macro::Literal` from user's string literals, pass to `format!`, and don't break rust-analyzer

[nik-rev]

I have a macro docstr! that turns documentation comments into a string that can be passed to format!, but the downside is that rust-analyzer doesn't work on it.

• Semantic highlighting doesn't work
• Renaming variables won't rename interpolated variables inside the macro
• Go to definition doesn't work
• etc.

The macro:

let hello_world: String = docstr!(format!
    /// fn say_hi() {{
    ///     println!("Hello, my name is {name}");
    /// }}
);

assert_eq!(hello_world, r#"fn say_hi() {
    println!("Hello, my name is Bob");
}"#);

// Expansion:

format!(r#"fn say_hi() {{
    println!("Hello, my name is {name}");
}}"#);

Problem: The span of the generated string literal inside of the format!() call doesn't link to the span of the documentation comment. To illustrate, I want the span of these 3 string literals (desugared the doc comments above):

#[doc = r"fn say_hi() {{"]
      a ^^^^^^^^^^^^^^^^^
#[doc = r#"]    println!("Hello, my name is {}");"#]
      b ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#[doc = r"]}}"]
      c ^^^^^^

To somehow join together when I create the string literal:

   format!("fn say_hi() {{\n    println!(\"Hello, my name is {name}\");\n}}"); 
a + b + c  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Such that:

• Renaming the name variable will rename its usage in the documentation comments and also at the definition site
• I can do go-to-definition on the name variable
• Semantic tokens works, and I get syntax highlighting of interpolated identifier name in the doc comment

In terms of proc macro code, I have the doc_comments variable that contains all the documentation comment strings (link to code):

let doc_comments: Vec<String>;

Now I combine all of those string literals into a single string literal:

let string = doc_comments
    .into_iter()
    .reduce(|mut acc, s| {
        acc.push('\n');
        acc.push_str(&s);
        acc
    })
    .unwrap_or_default();

And finally that string is injected into a macro call and essentially becomes quote!(format!(#string))

What I tried

For each doc comment, span of the string literal is also collected:

let doc_comments: Vec<(String, Span)>;

We will also Span::join all of those spans together (#![feature(proc_macro_span)]):

let (string, span) = doc_comments
    .into_iter()
    .reduce(|(mut acc_contents, acc_span), (contents, span)| {
        acc_contents.push('\n');
        acc_contents.push_str(&contents);
        (acc_contents, acc_span.join(span).unwrap())
    })
    .unwrap_or_else(|| (String::new(), Span::call_site()));

When creating the string literal, this span is used:

// format!(hello, "foo\nbar", a, b)
//                ^^^^^^^^^^
TokenTree::Literal({
    let mut string = Literal::string(&string);
    string.set_span(span);
    string
}),

This does not work

[ChayimFriedman2]

I don't think rust-analyzer will ever support that. It's fundamentally at odds with how it works.

[Veykril]

Yea this is tricky, I am not sure we can reasonably support this. Currently we downmap tokens as a whole by their span, so we might be associating the input (desugared) doc comment to the output string literal, but at that point we lost the association of interpolated positions as we are special casing those for format_args!, without those positions we cannot tell anything about the format! invocation. So unfortunately I do not think this can ever work with rust-analyzer, span joining I think is generally destructive with regards to format_args! for IDE feature support. Especially the fact that you are joining the spans but also modifying the resulting string with extra content will shift offsets and what not, all of that makes it impossible for us to reason about (the format_args! support is already quite hacky and I am surprised it even works this well)