Job progress logging

[chrisiacovella]

Currently, optimization workflows do not provide much in terms of incremental information in terms of progress. A report is provide after each iteration is completed, but the status of an individual iteration (e.g., number of runs completed, number total to run, etc.) are not really provided. It would be beneficial as a user to understand to have some finer grained logging of the status (as well as some reports of total elapsed time per iteration).

[chrisiacovella]

Also logging time of individual simulations, in particular those that are the e.g. 10 slowest. We may want to exclude those that are massively slower.

[lilyminium]

Pain points with current Evaluator logging

• some warnings are repeated, e.g. every time you apply NAGL charges it logs which method was used to apply it. Perhaps just log once and then stop
• it would be good to document what the output is telling you
• but generally knowing which simulations converged, how many didn't, mean time ± std, and maybe ten slowest can be dynamically removed from the next iteration.
• Perhaps walltime caps on simulatoins if that's reasonable.

[chrisiacovella]

A few other thoughts:

• It could be easy to include a mode that would eliminate any state points from the optimization that exceed a wall time cap (or was a significant outlier in time).
• Similar logic: mode that would remove a state point from the optimization if it fails to compute N times in a row (i.e., don't waste time on state points that don't simulate well).
• weights and biases (wandb.ai) has a nice API that makes it easy to push datapoints and custom plots to their dashboard. This could give (1) a nice dashboard to monitor progress, (2) a way to save important artifacts, (3) a way to easily compare different variants of force fields. It would be easy to add in API hooks (that can be toggled on and off).