Hotfix: Metric size limits

[wk9874]

Hotfix: Metric size limits

Python Version(s) Tested: 3.11

Operating System(s): Ubuntu

📝 Summary

The total grid metric size limit was set to the same as the individual grid metric size limit (50k points, ~50KB). The total limit should be 10MB

This also exposed that the runs would go into lost state even though metrics were still being uploaded. Suspect this is due to the heartbeat thread being joined before the dispatcher thread was joined in _tidy_run or _error, meaning if there are lots of grids left to dispatch, the heartbeat would stop prematurely.

🔍 Diagnosis

Veeeery slow performance when loading large FDS simulations with lots of grid data. Run would become lost even while still uploading grid metrics.

🔄 Changes

Change the total metrics limit to be 1e6, and moved the joining of the heartbeat thread to below the joining of the dispatcher thread in _error and _tidy_run

✔️ Checklist

• Unit and integration tests passing.
• Pre-commit hooks passing.
• Quality checks passing.