[WIP] Define how to run tests

[furtib]

Why:
The requirements for running tests are not well defined.
This patch aims to define which targets should be run for testing.
This patch does not define how the user should set up their environment, only how the tests should be run. That is a todo for the next patch.

What:

• Added a script that runs all checks.
• Modified the CONTRIBUTING.md to specify which tests must be run

Addresses:
none?

[nettle]

Good start! Thanks @furtib!

[nettle]

I'm sure, before running anything we must crealy describe how to initialize the environment.
See Development Environment above

[nettle]

Hm... discussable...
Maybe we should setup correct environment in this script?

[Szelethus]

Set up through program arguments/environmental variables? I understand that our clientbase have their own bazel and CodeChecker distributions, so we can't generalize too much here.

This sounds like a reasonable compromise:
USE_BAZEL_VERSION=8.5.0 CODECHECKER_BIN_PATH=/path/to/codechecker/25 run_tests.sh

[nettle]

This is about Development Environment not Execution Environment since we run tests, right?
Here we have to find common ground, common environment that works for all possible contributors, starting from you and me, but extend to anyone. Which already means at least different versions of Ubuntu and RedHat, different version sets of Bazel and Clang etc.
That's exactly why I suggested Mise and/or Micromamba to solve this problem.
Both are still not ideal, but at least something. Maybe some alternatives are possible too.
(BTW, native bazel dependency manager may work, but we did not try)

[furtib]

We could very well set up the python virtual environment for testing in this script, but I would be hesitant to also do that for the llvm toolchain (maybe thats where im wrong?).

My first idea is whether a bazel provided llvm toolchain exist and can we use that?

My experience is that most of these crashes should be due to diagtool not being found by clang. (You can check the codechecker log for the failing analyzed file to confirm this, even though the plist files arent getting created the log still does)

[nettle]

We could very well set up the python virtual environment for testing in this script, but I would be hesitant to also do that for the llvm toolchain (maybe thats where im wrong?).

Right, unfortunately, venv (virtual environment) is not sufficient, that's exactly why I suggested mise and/or micromamba.

My first idea is whether a bazel provided llvm toolchain exist and can we use that?

Yes, it is theoretically possible!
Practically... I dont know, it might be difficult to find llvm package that we could use.

My experience is that most of these crashes should be due to diagtool not being found by clang. (You can check the codechecker log for the failing analyzed file to confirm this, even though the plist files arent getting created the log still does)

There are so many different failures depending on versions and environments, so did not check them.
We need to fix the environment first, then set of tests and the way we run them.
After that I can report and investigate fails.

[furtib]

I have investigated the possibility of using the bazel module: https://github.com/bazel-contrib/toolchains_llvm
And I found it to be surprisingly doable. I don't even think the big download size is a huge blocker, since it won't redownload it even after a bazel clean.
With that said, it introduces the complexity of whether we require all users to use it or not. (How do we test both methods if we do not require it?)

So instead, I tried this different approach, setting up the environment "manually".
I try not to set any dependency that we need to vary, like Bazel; I'm still figuring out how to handle that.
I chose this because it produces an environment that I think is more easily reproducible to end users compared to what Micromamba provides.