feat: Add openqa-llm-investigate and integrate into auto-review hook#536
Open
okurz wants to merge 5 commits intoos-autoinst:masterfrom
Open
feat: Add openqa-llm-investigate and integrate into auto-review hook#536okurz wants to merge 5 commits intoos-autoinst:masterfrom
okurz wants to merge 5 commits intoos-autoinst:masterfrom
Conversation
667281a to
95d9ae0
Compare
Motivation: The CI was failing because it relied on a hardcoded list of dependencies in the workflow file, which didn't include the newly added 'httpx' and 'typer'. Design Choices: - Added a '[build-system]' block to 'pyproject.toml' to make the project installable via pip. - Moved 'dev' dependency group to '[project.optional-dependencies]' for better compatibility with standard pip. - Updated '.github/workflows/ci.yml' to use 'pip install .[dev]' instead of a hardcoded list. Benefits: - Future-proofs the CI by automatically including any new dependencies added to 'pyproject.toml'. - Simplifies the workflow configuration.
Motivation: The CI was failing during 'pip install .[dev]' because setuptools was unable to find 'Readme.md' (the file is 'README.md'), was failing to discover packages in a flat layout with multiple top-level directories, and was missing configuration for dynamic versioning. Design Choices: - Corrected readme filename to 'README.md'. - Explicitly set 'packages = []' in '[tool.setuptools]' as this project is a collection of standalone scripts, not a traditional Python package.
95d9ae0 to
1edd8e1
Compare
d3flex
reviewed
Apr 17, 2026
d3flex
reviewed
Apr 17, 2026
d3flex
reviewed
Apr 17, 2026
Martchus
approved these changes
Apr 17, 2026
Motivation: We want to integrate LLM analysis into the openQA investigation workflow to provide concise summaries of test failures. This helps reviewers quickly understand if an issue is a new product regression, a test regression, or an infrastructure problem, without scheduling potentially costly openqa-investigate jobs prematurely. Design Choices: - Created a standalone Python script `openqa-llm-investigate` using `typer` and `httpx`. - Fetches job details, test results, and test history from the openQA API to build a comprehensive prompt. - Used `pytest` and `unittest.mock` for unit testing the Python script. - Updated the existing Bash test suite to mock and assert the execution of `openqa-llm-investigate`. Benefits: - Reduces the number of unnecessary and costly `openqa-investigate` jobs by filtering them through an LLM first. - Provides immediate, actionable summaries of failures directly as openQA comments. - Improves efficiency of test reviewers by providing context directly in the job page. Related issue: os-autoinst/os-autoinst#2857
Motivation: Ensure that costly openqa-investigate and bisection jobs are only triggered after an LLM has confirmed the necessity. Design Choices: - Modified 'investigate-and-bisect' in 'openqa-label-known-issues-and-investigate-hook' to call 'openqa-llm-investigate'. - The bash function now captures the output of the LLM script. If no URL is returned (meaning the LLM decided against investigation), the process terminates early. - Updated the test suite to include mocks for the LLM script and verified both the 'YES' and 'NO' investigation paths. Benefits: - Prevents redundant resource consumption by filtering investigation candidates through an intelligent gatekeeper. - Provides consistent behavior with the existing 'label' mechanism.
Motivation: Ensure the investigation workflow continues even if the LLM server is unavailable or the openqa-llm-investigate script fails for other reasons. Design Choices: - Modified the investigate-and-bisect function to catch failures from openqa-llm-investigate. - Added a warning message when a failure occurs. - Implemented a fallback that uses the original test URL for standard investigation when the LLM script fails. - Expanded the Bash test suite to verify the fallback mechanism and ensure correct behavior when the LLM script suggests skipping. Benefits: - Increases robustness of the automated investigation pipeline. - Prevents blocking the entire investigation process due to transient LLM service issues.
1edd8e1 to
1456376
Compare
Member
Author
|
included all suggestions from @d3flex |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
We want to integrate LLM analysis into the openQA investigation workflow to
provide concise summaries of test failures. This helps reviewers quickly
understand if an issue is a new product regression, a test regression, or an
infrastructure problem, without scheduling potentially costly openqa-investigate
jobs prematurely.
Design Choices:
openqa-llm-investigateusingtyperand
httpx.openqa-label-known-issues-and-investigate-hook: it parses the LLM's responseand only outputs the job URL to trigger further bisections if the LLM
determines it is necessary.
build a comprehensive prompt.
pytestandunittest.mockfor unit testing the Python script.openqa-llm-investigate.Example run
openqa-llm-investigate -v https://openqa.opensuse.org/tests/5841514 --dry:Generated and posted example comment:
https://openqa.opensuse.org/tests/5841514#comment-887840
Benefits:
openqa-investigatejobs byfiltering them through an LLM first.
comments.
page.
Related issue: https://progress.opensuse.org/issues/198056