Let's not close issues before adding a test case or verifying the generated comments

[auduchinok]

Many of the bot comments contain hallucinations, invalid F# code, or simply do not cover the initial issues reported.

I strongly believe we should not close issues based on these responses alone.

@T-Gro Let's reopen the issues and add regression tests. We want the compiler to be robust and closing the bug reports like this only hides the issues.

[auduchinok]

Given that many of these issues can be tackled by the agents nowadays, closing them like this just doesn't help.

[T-Gro]

So the flow would be:

• AI detects something is likely fixed, but we do not trust it yet
• We have a daily/weekly PR with the tests added
  • How do we ensure test suite not rotting because of that? Especially overlap in testing the same, just using different terms.
  • One idea would be to pre-build an "index" of test folders and important test files, and let it act accordingly.
• Only once that PR adding tests passes and is merged, it closes the issue(s) referenced?

[auduchinok]

We have a daily/weekly PR with the tests added

I'd vote for creating a separate PRs, so the changes can be linked to the issues they fix. This should be done when the bot posts the appears-to-be-fixed comment.

How do we ensure test suite not rotting because of that? Especially overlap in testing the same, just using different terms.

Since the issues are not overlapping, the corresponding PRs shouldn't either.

Only once that PR adding tests passes and is merged, it closes the issue(s) referenced?

Yes, after a human verifies that the test corresponds to the issue, and the cases reported in the comments are covered too.

[T-Gro]

I meant overlap with existing tests added while features/improvements are developed, or being a subset of existing test scenario.
But it is true we can do regular test cleanups separately, via a different mechanism.

[T-Gro]

Btw. the problem with "not even valid F# code" is with security-induced angle brackets escaping in the overall Github agentic workflow, I will file an issue at githubnext/agentics#294 .

The issues have been verified before closing, but you are right it is not immediately clear from the commenting (with invalid code) posted, and I did not check the final comments rendered here on GH - my apologies.

[auduchinok]

Thanks @T-Gro!

The issues have been verified before closing, but you are right it is not immediately clear from the commenting (with invalid code) posted, and I did not check the final comments rendered here on GH - my apologies.

Yes, from the outsider perspective it looked like the responses were trusted without actual verification, since the comments contained invalid code.

[T-Gro]

I will also retroactively check the test coverage for each issue and add new tests if needed.
Not yet decided between /tests/regressions/1245 scheme, or best-fit according to compiler area and existing test suite(s)?

[auduchinok]

Not yet decided between /tests/regressions/1245 scheme, or best-fit according to compiler area and existing test suite(s)?

I'd vote for the latter and add a mention of the issue in the commit itself: it'd both allow proper navigation on GitHub and would fit existing tests.