5 create minimal set of variable functions for final energy

.github/copilot-instructions.md

@@ -4,61 +4,89 @@
 This repository implements a reusable Python module (`pypsa_validation_processing`) that:
 - Takes a definitions folder holding IAMC-formatted variable definitions
 - Executes the corresponding function (if available) to extract the value of the respective variable from a given PyPSA NetworkCollection
-- Returns the results as a `pyam.IamDataFrame`
+- Saves the result as IAMC-formatted xlsx-file.
 
 ## Code Structure
 
 ```mermaid
 classDiagram
-    class workflow.py{
-        config_path: pathlib.Path
-        build_parser()
+    class Network_Processor {
+        +path_config: pathlib_Path
+        +config: dict
+        +country: str
+        +definition_path: pathlib_Path
+        +mapping_path: pathlib_Path
+        +output_path: pathlib_Path
+        +network_results_path: pathlib_Path
+        +model_name: str
+        +scenario_name: str
+        +network_collection: pypsa_NetworkCollection
+        +dsd: nomenclature_DataStructureDefinition
+        +functions_dict: dict
+        +dsd_with_values: pyam_IamDataFrame
+        +path_dsd_with_values: pathlib_Path
+        +__init__(path_config: pathlib_Path)
+        +_read_config() dict
+        +_read_mappings() dict
+        +_read_pypsa_network_collection() pypsa_NetworkCollection
+        +read_definitions() nomenclature_DataStructureDefinition
+        +_execute_function_for_variable(variable: str, n: pypsa_Network) pd_Series
+        +_postprocess_statistics_result(variable: str, result: pd_Series) pd_DataFrame
+        +structure_pyam_from_pandas(df: pd_DataFrame) pyam_IamDataFrame
+        +calculate_variables_values() None
+        +write_output_to_xlsx() None
     }
 
-    class statistics_functions.py{
-        one function per variable
+    class statistics_functions_py {
+        +Final_Energy_by_Carrier__Electricity(n: pypsa_Network) pd_DataFrame
+        +Final_Energy_by_Sector__Transportation(n: pypsa_Network) pd_DataFrame
     }
 
-    class mapping.yaml{
-        mapping variables to function names
+    class utils_py {
+        +EU27_COUNTRY_CODES: dict~str,str~
+        +UNITS_MAPPING: dict~str,str~
     }
 
-    class config.yaml{
-        user configuration
+    class config_default_yaml {
+        +country: str
+        +definitions_path: str
+        +output_path: str
+        +network_results_path: str
+        +model_name: str
+        +scenario_name: str
     }
 
-    Network_Processor <-- workflow.py : executes
-    statistics_functions.py <-- Network_Processor : executes
-    mapping.yaml <|-- Network_Processor : includes
-    config.yaml <|-- Network_Processor : includes
-
-    class Network_Processor{
-        config_path: pathlib.Path
-        config: dict
-        network_results_path: pathlib.Path
-        definitions_path: pathlib.Path
-        mappings_path: pathlib.Path
-        country: str
-        model_name: str
-        scenario_name: str
-        network_collection: pypsa.NetworkCollection
-        dsd: nomenclature.DataStructureDefinition
-        functions_dict: dict
-        dsd_with_values: dict
-        path_dsd_with_values: pathlib.Path
-
-        __init__()
-        __repr__()
-        _read_config()
-        _read_mappings()
-        _read_pypsa_network_collection()
-        read_definitions()
-        _execute_function_for_variable()
-        structure_pyan_from_pandas()
-        calculate_variable_values()
-        write_output_to_xlsx()
+    class mapping_default_yaml {
+        +Final_Energy_by_Carrier__Electricity: str
+        +Final_Energy_by_Sector__Transportation: str
     }
-    note for Network_Processor "in class_definitions.py"
+
+    class pypsa_NetworkCollection
+    class pypsa_Network {
+        +name: str
+        +statistics: pypsa_StatisticsAccessor
+    }
+    class pypsa_StatisticsAccessor {
+        +energy_balance(components: list, carrier: str, groupby: list) pd_Series
+    }
+
+    class nomenclature_DataStructureDefinition {
+        +variable: pd_Series
+    }
+
+    class pyam_IamDataFrame
+    class pd_DataFrame
+    class pd_Series
+
+    Network_Processor --> pypsa_NetworkCollection : owns
+    Network_Processor --> nomenclature_DataStructureDefinition : uses
+    Network_Processor --> pyam_IamDataFrame : creates
+    Network_Processor --> statistics_functions_py : calls
+    Network_Processor --> utils_py : imports
+    Network_Processor --> config_default_yaml : reads
+    Network_Processor --> mapping_default_yaml : reads
+    pypsa_NetworkCollection --> pypsa_Network : contains
+    pypsa_Network --> pypsa_StatisticsAccessor : has
 ```
 
 ## Folder Structure
@@ -75,6 +103,7 @@ classDiagram
 |  |  `- mapping.default.yaml
 |  |- class_definitions.py
 |  |- statistics_functions.py
+|  `- utils.py
 |  `- workflow.py
 |- workflow.py
 |- resources
@@ -90,6 +119,7 @@ classDiagram
 - `mapping.default.yaml` (or another mapping-file provided by the config-file) holds the mapping of IAMC variable to the respective function in `pypsa_validation_processing/statistics_functions.py`
 - The package workflow entrypoint is `pypsa_validation_processing/workflow.py`; the root `workflow.py` is a thin compatibility wrapper
 - Default configs are packaged inside `pypsa_validation_processing/configs/`
+- Pixi is used as environment package manager. Use `pixi run` before your statement in cli to use the intended pixi-environment.
 - The `resources/` directory holds non-versioned resources
 - The `sister_packages/` directory holds related packages for background information
 - The `tests/` directory holds unit and integration tests
@@ -102,7 +132,7 @@ A task is complete when:
 - Changes are integrated into existing folder structure.
 - A short summary of changes is provided.
 - In chat mode: the user has reviewed the changes and given approval.
-- For a pull-request: the user has to be reviewer of the pull request to give approval.
+- For a pull-request: the user is reviewer of the pull request to give approval.
 
 ## Forbidden Actions
 - Do NOT invent datasets, files, or APIs.

.github/workflows/tests.yml

@@ -0,0 +1,30 @@
+name: Tests
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Pixi
+        uses: prefix-dev/setup-pixi@v0.9.4
+        with:
+          pixi-version: v0.66.0
+
+      - name: Run Tests
+        run: pixi run pytest tests/ -v --cov=pypsa_validation_processing --cov-report=xml
+
+      - name: Upload Coverage Reports
+        uses: codecov/codecov-action@v4
+        with:
+          files: ./coverage.xml
+          flags: unittests
+          name: codecov-umbrella
+          fail_ci_if_error: false

.gitignore

@@ -46,8 +46,8 @@ coverage.xml
 cover/
 
 # Jupyter Notebook
-.ipynb_checkpoints
-.ipynb
+*.ipynb_checkpoints
+*.ipynb
 
 # IPython
 profile_default/

README.md

@@ -23,6 +23,18 @@ This package processes a PyPSA network for a given set of defined IAMC-Variables
 ```bash
 pip install .
 ```
+### Run the workflow
+Run the workflow with 
+```bash
+pixi run workflow
+```
+This statement runs `"python workflow.py"` 
+### Run tests
+Run tests with
+```bash
+pixi run test
+```
+This statement runs `"pytest tests/ -v"` 
 
 ## Project structure
 
@@ -33,9 +45,87 @@ pypsa_validation_processing/
 |   |-- workflow.py                     # package-level workflow orchestration
 |   |-- class_definitions.py            # core processing classes
 |   |-- statistics_functions.py         # pypsa statistics functions
+|   |-- utils.py                        # static information and general utility functions
 |   `-- configs/                        # package configuration files
         `-- config.default.yaml         # default configuration file
         `-- mapping.default.yaml        # mapping IAMC-variable - statistics-function 
 |-- resources/                          # non-versioned resources
 `-- tests/                              # test suite
-```
+```
+
+## Variable's Statistics - Functions
+
+This section describes the conventions for adding new variable statistics functions to `pypsa_validation_processing/statistics_functions.py`.
+
+### Naming Convention
+
+Function names follow the IAMC variable name with these substitutions:
+
+- Each `|` (pipe / hierarchy separator) is replaced by `__` (double underscore).
+- Spaces are replaced by `_` (single underscore)
+- Other special characters are fully removed.
+
+Examples:
+
+| IAMC Variable | Function Name |
+|---|---|
+| `Final Energy [by Carrier]\|Electricity` | `Final_Energy_by_Carrier__Electricity` |
+| `Final Energy [by Sector]\|Transportation` | `Final_Energy_by_Sector__Transportation` |
+
+### Function Signature (fixed)
+
+Every function receives exactly one argument – a single `pypsa.Network` object representing one investment year – and returns a `pandas.Series`:
+
+```python
+def <function_name>(n: pypsa.Network) -> pd.Series:
+    ...
+```
+
+**The returned `Series` is of the structure of the direct outcome of a `pypsa.statistics` - Function.** It therefore must have a multi-level index that includes a level named `"unit"` and `"variable"`, so that the post-processing step can extract the unit information. It is possible to return multiple values with different units. Units are then converted to IAMC-valid units and summed over. Do not mix energy- and emissions- units in one statement!
+
+#### Example output
+
+- statistics-statement, grouped by country and unit:
+```python
+n.statistics.energy_balance(
+    carrier = ["land transport EV", "land transport fuel cell", "kerosene for aviation", "shipping methanol"],
+    components = "Load",
+    groupby = ["carrier", "unit", "country"],
+    direction = "withdrawal"
+).groupby(["country", "unit"]).sum()
+```
+
+- returns a processable `pd.Series`:
+```
+country  unit
+AL       MWh_LHV    1.073021e+06
+         MWh_el     1.996662e+06
+AT       MWh_LHV    1.319779e+07
+         MWh_el     2.105799e+07
+BA       MWh_LHV    3.214431e+05
+                        ...
+SI       MWh_el     5.576678e+06
+SK       MWh_LHV    1.185324e+06
+         MWh_el     8.633450e+06
+XK       MWh_LHV    8.771836e+04
+         MWh_el     1.081549e+06
+Length: 68, dtype: float64
+```
+
+### Mapping File
+
+`configs/mapping.default.yaml` maps each IAMC variable name to the corresponding function name in `statistics_functions.py`:
+
+```yaml
+Final Energy [by Carrier]|Electricity: Final_Energy_by_Carrier__Electricity
+Final Energy [by Sector]|Transportation: Final_Energy_by_Sector__Transportation
+```
+
+At runtime, `Network_Processor` reads this mapping, looks up the function for each defined variable, and calls it for every network in the collection.  Variables without a mapping entry are silently skipped. 
+
+### Register statistics for a new variable
+To register a new variable
+- add an entry to the mapping file
+- implement the corresponding function
+- add a corresponding test-function
+- make shure, that the introduces variable is also part of your variable set to be executed.