Agent service (linux, systemd) might get stuck if upgrade attempts are backed up and are processed too quickly

[antoinemzs]

Description

Lots of backed up upgrade jobs that fail to complete, service ends up stuck and doesn't restart because of too many successive restarts.

Jun 27 08:40:34 vm1.obas.lan systemd[1]: openbas-agent.service: Start request repeat
ed too quickly.
Jun 27 08:40:34 vm1.obas.lan systemd[1]: openbas-agent.service: Failed with result '
start-limit-hit'.

Environment

1. OS (where OpenBAS server runs): { e.g. Mac OS 10, Windows 10, Ubuntu 16.4, etc. }
2. OpenBAS version: { e.g. OpenBAS 1.0.2 }
3. OpenBAS client: { e.g. frontend or python }
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. { e.g. Run ... }
2. { e.g. Click ... }
3. { e.g. Error ... }

Expected Output

Actual Output

Additional information

Screenshots (optional)

[DeniseMonreale]

Hello, may you tell us more about this bug and how is important and block user please, do you have an idea to how fixed? @antoinemzs

[antoinemzs]

Sure, systemd on linux systems gives up if a service restarts too many times in a short period, meaning we risk the case that the openbas agent might be left not running and not processing injects.
This might happen if an agent upgrade fails to bring the agent to the desired version and has to repeat and fail indefinitly because of a permanent issue.

Currently the OpenAEV server controls putting upgrade jobs in the queue but these aren't normal injects so there isn't any feedback if the upgrade failed. When the agent that has failed upgrading restarts (via systemd), it reports its older version prompting OpenAEV to line up yet another upgrade job, and so on in a tight loop, which eventually makes systemd suspect something is wrong with the service and halts the restarts.

For fixing, my thoughts:

1. we could reverse the auto-upgrade relationship: currently it's openaev server blindly instructing agents to auto upgrade, in the reverse relationship agents might control their auto-upgrade behaviour in a more graceful manner, avoiding the tight restart loop
2. add in the agent registration some indication of the latest upgrade attempt logs, result... alongside with agent version, so OpenAEV can report to the user
3. some form of dashboard in the openaev UI allowing for spotting agents failing to upgrade