[Testing] Integrate Automated Test Coverage Reporting into CI Pipeline #30
Description
Hi @1brahimmohamed and @jan-petr,
Following the successful implementation of the foundational CI pipeline (Phase 1) #21 and the end-to-end deterministic regression fixtures (Phase 2) #22 , I would like to propose the next logical step to ensure the long-term reliability of the repository.
Currently, our CI runs the tests successfully, but we lack visibility into our testing blind spots. As we prepare to scale the architecture to support complex new modalities (like DCE/DSC), it is critical that we know exactly which lines of code are protected by our fixtures and which are not.
The Proposal:
Integrate automated test coverage reporting directly into our GitHub Actions workflow using pytest-cov.
Implementation Details:
- Add pytest-cov to the CI dependency matrix.
- Update the pytest execution command in our workflow to output a terminal coverage report (e.g., pytest tests/ -v --cov=app --cov-report=term-missing).
- Generate terminal coverage report (
--cov=app --cov-report=term-missing) - Optionally upload XML report for future Codecov integration
- Introduce a soft baseline coverage threshold (e.g., 70%) without immediately enforcing failure to allow incremental improvement
The Value:
- Instant Reviewer Feedback: Whenever a contributor opens a new PR, mentors can instantly check the CI logs to see the exact percentage of the codebase that is covered, and specifically which lines of code remain untested.
- Guided Test Expansion: It gives us a concrete metric to systematically improve the reliability of the pyaslreport package before adding new features.
- Establishes a measurable reliability metric as the project scales toward multi-modality support.
If you agree this is a valuable addition to our testing infrastructure, I can quickly put together a PR to inject this into our existing CI workflow!
Best,
Aviral