Add parallel tuning on multiple remote GPUs using Ray

[isazi]

Working on a simple parallel runner that uses Ray to distribute the benchmarking of different configurations to remote Ray workers.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[stijnh]

The current parallel runner works. I've been able to run on multiple GPUs on DAS6-VU and DAS6-Leiden.

There are several remaining problems:

• The timings are incorrect as the host assumes that the total time is just the sum over individual configurations
• Use of tuning_options need to be refactored, as now it the entire object is sent to every node for each benchmark job
• Logging information can be improved
• The strategies are not parallel-aware yet (except brute-force)
• A guide needs to be added to the docs explaining how to launch a Ray cluster on DAS6

[benvanwerkhoven]

Most tests pass now. On my system it is just the test/test_time_budgets.py::test_some_time_budget test that keeps failing. Likely still something with how time is accounted for with the new costfunc implementation.

[benvanwerkhoven]

I have resolved a couple of issues with sequential tuning that had arisen from the changes to the 'eval_all' costfunc. I haven't actually tested the parallel runner yet. But there are couple of other things I need to attend to now.

[sonarqubecloud]

Quality Gate passed

Issues
67 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud