I think it would be useful if benchmark users had a single notebook that includes both (1) the benchmark documentation and (2) the evaluation workflow.
Current situation:
Documentation is stored as markdown in git and built with ReadTheDocs which was developed by @srosenbu .
Evaluation is done in a separate notebook that fetches results from ROHub and visualizes example outputs from all relevant ROCrates done by @M-Jafarkhani .
Goal: Ideally, we would combine these into a single notebook.
However, IMO for a new benchmark, this will later move into an ROCrate that contains only the benchmark definition (MD documentation file or jupyter notebook for postprocessing, Snakefile templates, KG with parameters, etc.). In principle, the MD could simply be embedded into the notebook.
If we want a documentation site listing all benchmarks, I see two options:
- Automatically query ROHub for all benchmark ROCrates, download them, run the notebooks (including result generation), and build a static documentation site (similar to ReadTheDocs)
- Let users browse ROHub, select the benchmark they need, and launch the executable notebook directly on the NFDI(4ING) JupyterHub via a link included in a README.
Challenge:
The evaluation notebook is currently part of our test pipeline (@jpthiele), and duplicating the code might be undesirable.
Options for handling this:
- For tests, download the benchmark ROCrate from ROHub, convert the notebook to a Python script, and run it. This requires internet access and introduces a dependency on ROHub availability—undesirable for local CI (e.g. pytest).
- Alternatively, store the ROCrate for the plate with a hole (only for this benchmark, since we use that for testing purposes) in git and upload it to ROHub periodically (e.g. per release). However, this raises questions about changes over time and whether we need explicit versioning of the ROCrate to refer back to older benchmark definitions @doigl?
What do you think @berndflemisch, @div-tyg
I think it would be useful if benchmark users had a single notebook that includes both (1) the benchmark documentation and (2) the evaluation workflow.
Current situation:
Documentation is stored as markdown in git and built with ReadTheDocs which was developed by @srosenbu .
Evaluation is done in a separate notebook that fetches results from ROHub and visualizes example outputs from all relevant ROCrates done by @M-Jafarkhani .
Goal: Ideally, we would combine these into a single notebook.
However, IMO for a new benchmark, this will later move into an ROCrate that contains only the benchmark definition (MD documentation file or jupyter notebook for postprocessing, Snakefile templates, KG with parameters, etc.). In principle, the MD could simply be embedded into the notebook.
If we want a documentation site listing all benchmarks, I see two options:
Challenge:
The evaluation notebook is currently part of our test pipeline (@jpthiele), and duplicating the code might be undesirable.
Options for handling this:
What do you think @berndflemisch, @div-tyg