Skip to content

Source flux statistics reporting errors in MPI #4010

@dkachuma

Description

@dkachuma

Describe the bug
There are two issues identified with the source flux reporting in GEOS:

  1. CSV Output Issue: The source flux CSV file contains only a single time step (appearing to be the final one), even though the time column is labeled as 0.

  2. MPI Scaling/Calculation Issue: When running the case with multiple MPI ranks (e.g., 4), the reported production mass and rates are physically strange (orders of magnitude such as $10^{56}$ and $10^{47}$) and appear to increase over time. The same case produces reasonable results when run on a single MPI rank.

To Reproduce
Steps to reproduce the behaviour:

  1. Attached case with source fluxes (e.g., INJECTOR.LEFT and INJECTOR.RIGHT).

  2. A source flux report (fluxStatistics) to the configuration.

  3. Run the case with mpirun -n 1. Observe reasonable rates ($\approx 11.57$ kg/s).

  4. Run the same case with mpirun -n 4.

  5. Compare the flux statistics table in the logs and the generated CSV file.

Expected behaviour

  1. The CSV file should contain data for all logged time steps.

  2. The production rates and mass values should remain consistent and physically accurate regardless of the number of MPI ranks used.

Screenshots
1 MPI Rank (Correct):

-------------------------------------------------------------------------------------------------
|              fluxStatistics, flux statistics for: INJECTOR.LEFT, INJECTOR.RIGHT               |
|-----------------------------------------------------------------------------------------------|
|     Flux(es)     |    Region     |  Element Count  |  Prod. mass [kg]   |  Prod. rate [kg/s]  |
|------------------|---------------|-----------------|--------------------|---------------------|
|   INJECTOR.LEFT  |  all_regions  |              1  |   [0, 9999999936]  |     [0, 11.574074]  |
|  INJECTOR.RIGHT  |  all_regions  |              1  |  [0, -9999999936]  |    [0, -11.574074]  |
|        flux_set  |  all_regions  |              2  |            [0, 0]  |             [0, 0]  |
-------------------------------------------------------------------------------------------------

4 MPI Ranks (Incorrect):

---------------------------------------------------------------------------------------------------------------------
|                        fluxStatistics, flux statistics for: INJECTOR.LEFT, INJECTOR.RIGHT                         |
|-------------------------------------------------------------------------------------------------------------------|
|     Flux(es)     |    Region     |  Element Count  |       Prod. mass [kg]        |       Prod. rate [kg/s]       |
|------------------|---------------|-----------------|------------------------------|-------------------------------|
|   INJECTOR.LEFT  |  all_regions  |              1  |  [0, 8.589625290559929e+56]  |   [0, 9.941695938148062e+47]  |
|  INJECTOR.RIGHT  |  all_regions  |              1  |  [0, -8.58962529055992e+56]  |  [0, -9.941695938148053e+47]  |
|        flux_set  |  all_regions  |              2  |  [0, 8.711228593176025e+41]  |   [0, 9.735556609752802e+32]  |
---------------------------------------------------------------------------------------------------------------------

Platform (please complete the following information):

  • Machine: Pine
  • Compiler: gcc 12.2.0
  • GEOS Version: 1.1.0 (sha1: 8c90dae)
  • MPI version: Open MPI v5.0.5

Additional context

  • Files and logs attached in zip.

fluxStats.zip

Metadata

Metadata

Assignees

Labels

type: bugSomething isn't workingtype: newA new issue has been created and requires attention

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions