-
Notifications
You must be signed in to change notification settings - Fork 101
Source flux statistics reporting errors in MPI #4010
Description
Describe the bug
There are two issues identified with the source flux reporting in GEOS:
-
CSV Output Issue: The source flux CSV file contains only a single time step (appearing to be the final one), even though the time column is labeled as 0.
-
MPI Scaling/Calculation Issue: When running the case with multiple MPI ranks (e.g., 4), the reported production mass and rates are physically strange (orders of magnitude such as
$10^{56}$ and$10^{47}$ ) and appear to increase over time. The same case produces reasonable results when run on a single MPI rank.
To Reproduce
Steps to reproduce the behaviour:
-
Attached case with source fluxes (e.g.,
INJECTOR.LEFTandINJECTOR.RIGHT). -
A
source flux report(fluxStatistics) to the configuration. -
Run the case with
mpirun -n 1. Observe reasonable rates ($\approx 11.57$ kg/s). -
Run the same case with
mpirun -n 4. -
Compare the flux statistics table in the logs and the generated CSV file.
Expected behaviour
-
The CSV file should contain data for all logged time steps.
-
The production rates and mass values should remain consistent and physically accurate regardless of the number of MPI ranks used.
Screenshots
1 MPI Rank (Correct):
-------------------------------------------------------------------------------------------------
| fluxStatistics, flux statistics for: INJECTOR.LEFT, INJECTOR.RIGHT |
|-----------------------------------------------------------------------------------------------|
| Flux(es) | Region | Element Count | Prod. mass [kg] | Prod. rate [kg/s] |
|------------------|---------------|-----------------|--------------------|---------------------|
| INJECTOR.LEFT | all_regions | 1 | [0, 9999999936] | [0, 11.574074] |
| INJECTOR.RIGHT | all_regions | 1 | [0, -9999999936] | [0, -11.574074] |
| flux_set | all_regions | 2 | [0, 0] | [0, 0] |
-------------------------------------------------------------------------------------------------
4 MPI Ranks (Incorrect):
---------------------------------------------------------------------------------------------------------------------
| fluxStatistics, flux statistics for: INJECTOR.LEFT, INJECTOR.RIGHT |
|-------------------------------------------------------------------------------------------------------------------|
| Flux(es) | Region | Element Count | Prod. mass [kg] | Prod. rate [kg/s] |
|------------------|---------------|-----------------|------------------------------|-------------------------------|
| INJECTOR.LEFT | all_regions | 1 | [0, 8.589625290559929e+56] | [0, 9.941695938148062e+47] |
| INJECTOR.RIGHT | all_regions | 1 | [0, -8.58962529055992e+56] | [0, -9.941695938148053e+47] |
| flux_set | all_regions | 2 | [0, 8.711228593176025e+41] | [0, 9.735556609752802e+32] |
---------------------------------------------------------------------------------------------------------------------
Platform (please complete the following information):
- Machine: Pine
- Compiler: gcc 12.2.0
- GEOS Version: 1.1.0 (sha1: 8c90dae)
- MPI version: Open MPI v5.0.5
Additional context
- Files and logs attached in zip.