Parallelize solve_problems

[MaxenceGollier]

@dpo, @tmigot

#120

(hopefully) supersedes #127, #167, #176.
To parallelize over solvers instead of problems, CUTEst problems make it very very difficult, i have tried multiple times but failed. I think it is fine just to parallelize over problems.

To make things clear,

• I added a private function _run_problem which just performs one iteration of the loop.
• I first make a serial run where first_problem is set to false to avoid race conditions on this variable and other variables accessed when first_problem is true.
• I then make the parallel run.
• I added a lock when we modify and read from stats to prevent race conditions.

[MaxenceGollier]

The race condition causes issue, for example, see MaxenceGollier/ExactPenalty.jl#57. I will fix the BSD tests as well.

[tmigot]

@MaxenceGollier If I understand correctly you propose to do multi-threading over problems. This assumes that problems are thread-safe, which is the case for CUTEst and (in theory) OptimizationProblems.jl, however it also needs the solvers to be thread safe, which I don't think we really test.

If we are going for a multi-threaded version, I think it should be a different function than solve_problems also, because it will never be safe in general (for any set of problems, and any set of solvers).

[tmigot]

Would it be better to have one stats per thread and merge all of them in the end ?

[tmigot]

The idea would be that your version is faster when it is possible, and #194 would be safer. So, both approaches might be complementary.