Fomo generalized

[d-monnet]

Add Nesterov's Heavy-Ball and non-linear conjugate gradient (Polak-Ribière and Fletcher-Reeves) momentum strategies.

[tmigot]

Suggested change

[tmigot]

does this allocate ? Should it be d .= .-g ?

[tmigot]

Suggested change

	"""
	# function update_kappa(αk::T, αk₋::T, ::ia_momentum) where {T}
	# 1
	# end
	"""

[d-monnet]

Looks like the multiprec test fails because kkt_checker function times out.
Here is the test output :

fomo_tr_ia_momentum : nonmonotone configuration |   46     2     48    30.6s
      Problem (x₁ - 1)² + 4(x₂ - 1)²                |    4            4     0.0s
      Problem Diagonal quadratic                    |    4            4     0.0s
      Problem Tridiagonal quadratic                 |    4            4     0.0s
      Problem Rosenbrock                            |    4            4     0.0s
      Problem Extended Rosenbrock                   |    4            4     0.0s
      Precision Float16 for ptype unc               |    7            7     0.0s
      Precision Float32 for ptype unc               |    7            7     0.0s
      Precision Float64 for ptype unc               |    7            7     0.0s
      Precision BigFloat for ptype unc              |    5     2      7    30.0s

Test can be reproduced as

T = BFloat16
f(x) = (x[1] - 1)^2 + 4 * (x[2] - x[1]^2)^2
nlp = ADNLPModel(f, x0)
ϵ = eps(T)^T(1 / 4)
ng0 = norm(grad(nlp, nlp.meta.x0))
stats = fomo(nlp,momentum_backend = ia_momentum(), step_backend = r2_step()) # this doesn't time out

@test eltype(stats.solution) == T
@test stats.objective isa T
@test stats.dual_feas isa T
@test stats.primal_feas isa T
primal, dual = kkt_checker(nlp, stats.solution) # this seems to timeout, causing the 2 next tests to fail
@test all(abs.(dual) .< ϵ * ng0 + ϵ) # this fails
@test all(abs.(primal) .< ϵ * ng0 + ϵ) # this fails
@test stats.dual_feas < ϵ * ng0 + ϵ

Problem is solved with first order status on my machine, but kkt_checker times out, hence primal and dual variables are not defined causing two tests to fail.