Support FOSS tests running locally

[nettle]

Describe the task

We need to support multiple bazel versions (6, 7, 8)
when running FOSS tests locally, not only from CI 🤦
As of now we copy just .bazelversion with fixed old version.

Scope

1. Remove .bazelversion from templates
   See test/foss/templates/.bazelversion
2. Use .bazelversion only from root directory in CI
3. Change CI to create .bazelversion in root directory (the same scenario for ALL tests!)
4. Add more logging
5. Fix problem that foss/*/test-proj directories cannot be removed 🐞
6. Add possibility (script?) to run FOSS tests locally
7. Make sure FOSS tests work both in CI and locally the same way!

Definition of Done

Can successfully run and debug FOSS tests locally

[nettle]

OMG! Just found out that FOSS tests depend on CI where it copies .bazelversion file.
So, no chance to test locally?

[furtib]

FOSS tests are running for me locally. Maybe we should add a script for that too...
I placed the .bazelversion file into the template directory, since not everyone used Bazelisk, and I didn't want to clutter up the main directory. Now, with supporting more bazel versions, we should not have a .bazelversion file at all, and have them created dynamically.

[furtib]

This might be the right place for this:

In general, I see many problems, unfortunately. But I'm trying to look beyond works-for-me environment. You should do that too!

Here is the short list of what's on the surface:

1. Tests cannot be run locally (partially addressed by Rewrite testfiles to have multiversion testing locally #165 and Support FOSS tests running locally #149) - by that I mean the results are not predictable

I tried to address this in #169.

2. Tests do not exercise different versions and environments (trying to address by Introduce Mise #132, see also Consider tools for multi-configuration testing #163 and Execution requirements #164)

Addressed by #132.

3. Tests do have hidden dependencies on environment and "special" undocumented conditions (see Execution requirements #164)

TODO

4. Tests do not provide enough information, logging (partially addressed in Fix fails on older Bazel versions in unit tests #143, see also Print out stdout and stderr when test executed with -vvv #147)

Fixed by #147.

5. Tests create artifacts (can be addressed by using tempfile, see my comment)

Main paintpoints (FOSS) addressed by #169, TODO: for caching and external tests.

6. Tests create hidden! artifacts (test-proj) which may lead to unpredictable results

Addressed by #169.

7. Tests depends on arifacts created externally (.bazelversion), and it is still a problem

Addressed in #169. (I believe only the FOSS tests were dependent on .bazelversion; in the external tests were handled as optional)

8. Tests requires external cleaning (temporarily we can workaround with rm -rf test-proj, but this not a solution!)

(To be) Fixed by #169.

9. Tests create cached artifacts (.pytest_cache and pycache), well, this is a minor problem :)

These caches are designed specifically for testing, no? For now, we can live with them.
Pytest uses its cache to store failed tests, so they can be re-run with the --lf flag.
I'm aware that it is possible to cache other values with fixtures, but this is something that must be coded into the test to be used.

10. Tests do not use bazel test ... for testing (actually, we can use that, and probably should)

I'm not sure how the unittests would be run with bazel test ..., specifically, the cache tests are seemingly impossible to do through bazel.

Basically, what is essential:

• All tests must be possible to run locally

• All tests must produce the same results when running locally and in CI

• All tests must produce the same results for different versions of all tools

[nettle]

This might be the right place for this:

In general, I see many problems, unfortunately. But I'm trying to look beyond works-for-me environment. You should do that too!
Here is the short list of what's on the surface:

1. Tests cannot be run locally (partially addressed by Rewrite testfiles to have multiversion testing locally #165 and Support FOSS tests running locally #149) - by that I mean the results are not predictable

I tried to address this in #169.

2. Tests do not exercise different versions and environments (trying to address by Introduce Mise #132, see also Consider tools for multi-configuration testing #163 and Execution requirements #164)

Addressed by #132.

3. Tests do have hidden dependencies on environment and "special" undocumented conditions (see Execution requirements #164)

TODO

4. Tests do not provide enough information, logging (partially addressed in Fix fails on older Bazel versions in unit tests #143, see also Print out stdout and stderr when test executed with -vvv #147)

Fixed by #147.

5. Tests create artifacts (can be addressed by using tempfile, see my comment)

Main paintpoints (FOSS) addressed by #169, TODO: for caching and external tests.

6. Tests create hidden! artifacts (test-proj) which may lead to unpredictable results

Addressed by #169.

7. Tests depends on arifacts created externally (.bazelversion), and it is still a problem

Addressed in #169. (I believe only the FOSS tests were dependent on .bazelversion; in the external tests were handled as optional)

8. Tests requires external cleaning (temporarily we can workaround with rm -rf test-proj, but this not a solution!)

(To be) Fixed by #169.

9. Tests create cached artifacts (.pytest_cache and pycache), well, this is a minor problem :)

These caches are designed specifically for testing, no? For now, we can live with them. Pytest uses its cache to store failed tests, so they can be re-run with the --lf flag. I'm aware that it is possible to cache other values with fixtures, but this is something that must be coded into the test to be used.

10. Tests do not use bazel test ... for testing (actually, we can use that, and probably should)

I'm not sure how the unittests would be run with bazel test ..., specifically, the cache tests are seemingly impossible to do through bazel.

Basically, what is essential:

• All tests must be possible to run locally
• All tests must produce the same results when running locally and in CI
• All tests must produce the same results for different versions of all tools

Thanks a lot @furtib!
Really good work and visible progress! :)