[feat] Support custom indexed and external references

docs/config_reference.rst

@@ -2064,6 +2064,21 @@ General Configuration
    .. versionadded:: 3.7.0
 
 
+.. py:attribute:: general.reference_prefix
+
+   :required: No
+   :default: :obj:`None`
+
+   Directory prefix for resolving paths of external reference files.
+
+   When a test's :attr:`~reframe.core.pipeline.RegressionTest.reference`
+   attribute uses the ``$ref`` key to load references from an :ref:`external
+   reference file <external-references>`, that file is looked up
+   under this prefix. When not set, the test's prefix directory is used.
+
+   .. versionadded:: 4.10
+
+
 .. py:attribute:: general.ignore_check_conflicts
 
    :required: No
@@ -2534,4 +2549,4 @@ This is the builtin configuration that ReFrame always loads.
 
 .. seealso::
 
-   See also how configuration files are :ref:`loaded <manpage-configuration>` and how you can specify them with the :option:`--config-file` option.
+   See also how configuration files are :ref:`loaded <manpage-configuration>` and how you can specify them with the :option:`--config-file` option.

docs/howto.rst

@@ -314,6 +314,93 @@ In this case, you could set the :attr:`build_system` to ``'CustomBuild'`` and su
     You should use this build system with caution, because environment management, reproducibility and any potential side effects are all controlled by the custom build system.
 
 
+.. _howto-reference-index:
+
+Custom performance reference indexing
+=====================================
+
+.. versionadded:: 4.10
+
+By default the :attr:`~reframe.core.pipeline.RegressionTest.reference` attribute is indexed by the system and/or system/partition combination.
+However, it is often the case that the reference values depend on test variables and/or parameters.
+ReFrame allows you to define a custom index for the reference dictionary by using the special ``$index`` key.
+Here is an example reference definition for the stream benchmark tutorial example where references are defined per number of threads and thread placement:
+
+.. literalinclude:: ../examples/howto/reference_index.py
+   :lines: 5-
+
+Special keys are also supported to allow users to index their references by the system, environment, processor and device details.
+For example, we could define different references for different environments by using the ``$environ`` special key as follows:
+
+.. code-block:: python
+
+    reference = {
+        '$index': ('$environ', 'num_threads'),
+        'gnu': {
+            '1': {
+                'copy_bw': (10000, -0.2, 0.2, 'MB/s'),
+            },
+            '2': {
+                'copy_bw': (20000, -0.2, 0.2, 'MB/s'),
+            },
+            '4': {
+                'copy_bw': (40000, -0.2, 0.2, 'MB/s'),
+            },
+        },
+        'clang': {
+            '1': {
+                'copy_bw': (10000, -0.2, 0.2, 'MB/s'),
+            },
+            '2': {
+                'copy_bw': (20000, -0.2, 0.2, 'MB/s'),
+            },
+            '4': {
+                'copy_bw': (40000, -0.2, 0.2, 'MB/s'),
+            },
+        }
+    }
+
+External references
+-------------------
+
+.. versionadded:: 4.10
+
+Users can also keep test references in a separate YAML file instead of in the test class.
+To achieve this, the special ``$ref`` key must be used in the :attr:`reference` dictionary.
+
+.. code-block:: python
+
+    reference = {'$ref': 'references/stream.yaml'}
+
+By default, reference files are resolved relative to the test's :attr:`prefix` directory, but this can be controlled by the :attr:`~config.general.reference_prefix` configuration option or the :envvar:`RFM_REFERENCE_PREFIX`.
+
+The reference file can contain references for multiple tests and the general structure is an 1-1 match to the inline reference dictionary.
+Here is how the reference file for the previous example would look like:
+
+.. code-block:: yaml
+
+    stream_test:
+      $index: ['$environ', 'num_threads']
+      gnu:
+         1:
+            'copy_bw': [10000, -0.2, 0.2, 'MB/s']
+         2:
+            'copy_bw': [20000, -0.2, 0.2, 'MB/s']
+         4:
+            'copy_bw': [40000, -0.2, 0.2, 'MB/s']
+      clang:
+         1:
+            'copy_bw': [10000, -0.2, 0.2, 'MB/s']
+         2:
+            'copy_bw': [20000, -0.2, 0.2, 'MB/s']
+         4:
+            'copy_bw': [40000, -0.2, 0.2, 'MB/s']
+
+
+.. seealso::
+   Check the API docs for the :attr:`~reframe.core.pipeline.RegressionTest.reference` test's attribute for all the details on how to define references.
+
+
 .. _working-with-environment-modules:
 
 Working with environment modules

docs/manpage.rst

@@ -2310,6 +2310,25 @@ Whenever an environment variable is associated with a configuration option, its
    .. versionadded:: 3.7.0
 
 
+.. envvar:: RFM_REFERENCE_PREFIX
+
+   Directory prefix for resolving paths of external reference files.
+
+   When a test's reference attribute uses the ``$ref`` key to load references
+   from a file, that file is looked up under this prefix. When not set, the
+   test's prefix directory is used.
+
+   .. table::
+      :align: left
+
+      ================================== ==================
+      Associated command line option     N/A
+      Associated configuration parameter :attr:`~config.general.reference_prefix`
+      ================================== ==================
+
+   .. versionadded:: 4.10
+
+
 .. envvar:: RFM_REPORT_FILE
 
    The file where ReFrame will store its report.

docs/tutorial.rst

@@ -291,6 +291,9 @@ The lower and upper thresholds are deviations from the target reference expresse
 In our example, we allow the ``copy_bw`` to be 10% lower than the target reference and no more than 30% higher.
 Sometimes, especially in microbenchmarks, it is a good practice to set an upper threshold to denote the absolute maximum that cannot be exceeded.
 
+It is also possible to define a custom index for the reference dictionary, that is not based on the system and/or partition combination.
+Check the :ref:`howto-reference-index` in the :doc:`howto` for more details.
+
 
 Dry-run mode
 ------------

examples/howto/reference_index.py

@@ -0,0 +1,94 @@
+# Copyright 2016-2026 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
+# ReFrame Project Developers. See the top-level LICENSE file for details.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+import os
+import reframe as rfm
+import reframe.utility.sanity as sn
+
+
+class build_stream(rfm.CompileOnlyRegressionTest):
+    build_system = 'SingleSource'
+    sourcepath = 'stream.c'
+    executable = './stream.x'
+    array_size = variable(int, value=0)
+
+    @run_before('compile')
+    def prepare_build(self):
+        omp_flag = self.current_environ.extras.get('omp_flag')
+        self.build_system.cflags = ['-O3', omp_flag]
+        if self.array_size:
+            self.build_system.cppflags = [f'-DARRAY_SIZE={self.array_size}']
+
+
+@rfm.simple_test
+class stream_test(rfm.RunOnlyRegressionTest):
+    valid_systems = ['*']
+    valid_prog_environs = ['+openmp']
+    stream_binary = fixture(build_stream, scope='environment')
+    num_threads = parameter([1, 2, 4, 8])
+    thread_placement = parameter(['close', 'spread'])
+    reference = {
+        '$index': ('num_threads', 'thread_placement'),
+        1: {
+            'close': {
+                'copy_bw': (10000, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (8000, -0.2, 0.2, 'MB/s'),
+            },
+            'spread': {
+                'copy_bw': (10500, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (8500, -0.2, 0.2, 'MB/s'),
+            },
+        },
+        2: {
+            'close': {
+                'copy_bw': (18000, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (16000, -0.2, 0.2, 'MB/s'),
+            },
+            'spread': {
+                'copy_bw': (18500, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (16500, -0.2, 0.2, 'MB/s'),
+            },
+        },
+        4: {
+            'close': {
+                'copy_bw': (32000, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (29500, -0.2, 0.2, 'MB/s'),
+            },
+            'spread': {
+                'copy_bw': (33000, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (30500, -0.2, 0.2, 'MB/s'),
+            },
+        },
+        8: {
+            'close': {
+                'copy_bw': (60000, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (55000, -0.2, 0.2, 'MB/s'),
+            },
+            'spread': {
+                'copy_bw': (62000, -0.2, 0.2, 'MB/s'),
+                'triad_bw': (57000, -0.2, 0.2, 'MB/s'),
+            },
+        },
+    }
+
+    @run_after('setup')
+    def set_executable(self):
+        self.executable = os.path.join(self.stream_binary.stagedir, 'stream.x')
+
+    @run_before('run')
+    def setup_threading(self):
+        self.env_vars['OMP_NUM_THREADS'] = self.num_threads
+        self.env_vars['OMP_PROC_BIND'] = self.thread_placement
+
+    @sanity_function
+    def validate(self):
+        return sn.assert_found(r'Solution Validates', self.stdout)
+
+    @performance_function('MB/s')
+    def copy_bw(self):
+        return sn.extractsingle(r'Copy:\s+(\S+)', self.stdout, 1, float)
+
+    @performance_function('MB/s')
+    def triad_bw(self):
+        return sn.extractsingle(r'Triad:\s+(\S+)', self.stdout, 1, float)

examples/tutorial/stream/stream_parameters.py

@@ -26,8 +26,8 @@ class stream_test(rfm.RunOnlyRegressionTest):
     valid_systems = ['*']
     valid_prog_environs = ['+openmp']
     stream_binary = fixture(build_stream, scope='environment')
-    num_threads = parameter([1, 2, 4, 8])
-    thread_placement = parameter(['true', 'close', 'spread'])
+    num_threads = parameter([1, 2, 4, 8], type=int)
+    thread_placement = parameter(['true', 'close', 'spread'], type=str)
 
     @run_after('setup')
     def set_executable(self):

reframe/core/exceptions.py

@@ -349,6 +349,16 @@ class UnexpectedSuccessError(ReframeError):
     '''Raised when a test unexpectedly passes'''
 
 
+class ReferenceParseError(ReframeError):
+    '''Exception raised when a reference file cannot be parsed.
+
+    .. seealso::
+
+       :attr:`~reframe.core.pipeline.RegressionTest.reference` for details on
+       how to set test references.
+    '''
+
+
 def user_frame():
     '''Return the first user frame as a :py:class:`FrameInfo` object.
 

reframe/core/fields.py

@@ -94,7 +94,8 @@ def _check_type(self, value):
     def __set__(self, obj, value):
         try:
             self._check_type(value)
-        except TypeError:
+        except TypeError as err:
+            last_error = err
             raw_value = remove_convertible(value)
             if raw_value is value and not self._allow_implicit:
                 # value was not convertible and the field does not allow
@@ -106,7 +107,8 @@ def __set__(self, obj, value):
             for t in self._types:
                 try:
                     value = t(value)
-                except TypeError:
+                except TypeError as err:
+                    last_error = err
                     continue
                 else:
                     return super().__set__(obj, value)
@@ -116,7 +118,7 @@ def __set__(self, obj, value):
             raise TypeError(
                 f'failed to set variable {self._name!r}: '
                 f'could not convert to any of the supported types: '
-                f'{typenames}'
+                f'{typenames}: {last_error}'
             )
         else:
             return super().__set__(obj, value)

reframe/core/meta.py

@@ -9,6 +9,7 @@
 
 import functools
 import inspect
+import os
 import types
 import collections
 
@@ -19,6 +20,7 @@
 import reframe.core.fixtures as fixtures
 import reframe.core.hooks as hooks
 import reframe.utility as utils
+import reframe.utility.osext as osext
 
 from reframe.core.exceptions import ReframeSyntaxError
 
@@ -427,6 +429,29 @@ def __init__(cls, name, bases, namespace, **kwargs):
                         with_code_context=True
                     )
 
+        # Set the test prefix
+        #
+        # First check if the current test pins the prefix and store this, so
+        # as to reuse in derived tests
+        curr_prefix = os.path.abspath(
+            os.path.dirname(inspect.getfile(cls))
+        )
+        if kwargs.pop('pin_prefix', False):
+            cls._rfm_pinned_prefix = curr_prefix
+
+        try:
+            prefix = kwargs['custom_prefix']
+        except KeyError:
+            if osext.is_interactive():
+                prefix = os.getcwd()
+            else:
+                try:
+                    prefix = cls._rfm_pinned_prefix
+                except AttributeError:
+                    prefix = curr_prefix
+
+        cls._rfm_prefix = prefix
+
     def __call__(cls, *args, **kwargs):
         '''Inject test builtins during object construction.