Refactorer can eat comments in weird places

[randomPoison]

While working on #1611 I came across some pain points related to trying to handle comments correctly when comments appear in unusual places. Note that this is specifically for c2rust-refactor, see #387 for handling of comments in the transpiler.

Comment handling in the refactorer primarily works by attaching comments to AST nodes, which means that in any situation where a comment doesn't have a clear node to attach to we can end up deleting comments as part of an unrelated transform. For example, if there's a trailing comment in a block (or if the block only contains a comment), then there's not a correct AST node to attach the comment to. As a result, attempting to rewrite the block (e.g. by removing the unsafe keyword with the fix_unused_unsafe transform) will cause us to lose the comment. So for the following code:

unsafe {
    // Inner trailing comment
}

If we remove the unsafe keyword but otherwise don't touch the block's contents, we rewrite it to

{}

losing the trailing comment in the block's body. What's doubly weird is that a trailing comment in a block like this can show up as being attached to the next statement AFTER the block, which has caused weird interactions in #1611.

Comments in weird places also won't get tracked correctly, e.g. if there's a comment between the unsafe keyword and the block body (unsafe /* comment */ {}). In general, for comments that appear between tokens there isn't really a good way to associate the comment with an AST node, and rewriting the surrounding bit of the AST will lose the comment or cause it to be moved somewhere strange.

Note that this doesn't just apply to fix_unused_unsafe, this is something that will come up almost any time we rewrite parts of the AST. For example, we can hit similar cases with the convert_math_funcs and convert_exits transforms:

Examples of convert_math_funcs eating comments

let _ = sin(/* comment */ 1.0);

Becomes

let _ = 1.0.sin();

And

let _ = sin(
    // comment
    1.0
    // trailing comment
);

becomes

let _ = // comment
    1.0.sin();

[randomPoison]

We discussed this a bit and it sounds like this is a somewhat fundamental limitation of working with the AST, since the AST can't represent comments appearing between arbitrary tokens. Due to this limitation, it doesn't make sense to try to make refactorer transforms too comment-aware (though the implementation of fix_unused_unsafe in #1611 already makes a limited attempt to detect comments).

Instead, we should look at adding a postprocessing step that can identify when comments have been deleted, and integrate that with an LLM pass that can decide if the comments should be preserved and where in the rewritten code they should go. We already have support in the postprocessor for doing some of that, since we have LLM-powered comment insertion.