Skip to content

Commit 3081cfb

Browse files
jemeza-codegenjemeza-codegen
authored
feat: added codecov agent trigger example (#52)
* added codecov agent example * added missing files. updated path names * Automated pre-commit update
1 parent 984c5b7 commit 3081cfb

6 files changed

Lines changed: 343 additions & 27 deletions

File tree

‎README.md‎

Lines changed: 12 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -2,50 +2,35 @@
22

33
[![Documentation](https://img.shields.io/badge/docs-docs.codegen.com-blue)](https://docs.codegen.com)
44

5-
This is a collection of examples using [Codegen](https://codegen.com). You can use these examples to learn how to use Codegen and build custom code transformations.
5+
This is a collection of examples using [Codegen](https://codegen.com). You can use these examples to learn how to use Codegen and integrate the codegen agent into your workflows.
66

77
## Setup
88

99
We recommend using [`uv`](https://github.com/astral-sh/uv) with Python 3.13 for the best experience.
1010

11-
To install Codegen, please follow the [official installation guide](https://docs.codegen.com/introduction/installation). Once Codegen is installed, use these steps to run the examples in this repository:
12-
13-
Install the Codegen CLI globally
14-
```bash
15-
uv tool install codegen
16-
```
17-
18-
Initialize Codegen in your project
19-
```bash
20-
codegen init
21-
```
22-
23-
Activate the virtual environment
11+
Install Codegen
2412
```bash
25-
source .codegen/.venv/bin/activate
13+
uv pip install codegen
2614
```
2715

28-
Your environment is now ready to run example codemods.
29-
30-
### IDE Configuration (Optional)
16+
In order to use the codegen API you will need to generate a token [here](https://www.codegen.sh/token)
3117

32-
To configure your IDE for optimal use with Codegen, follow our [IDE setup guide](https://docs.codegen.com/introduction/ide-usage#configuring-your-ide-interpreter).
18+
Your environment is now ready to use the codegen API!
3319

3420
## Examples
3521

3622
Within the examples folder, each subdirectory contains a self-contained example with:
37-
- An explanation of the transformation (`README.md`)
38-
- A Codegen script that performs the transformation (`run.py`)
39-
- Sample code to transform, if not using a repository (`input_repo/`)
40-
41-
To see a transformation, simply run the `run.py` script within the desired directory.
23+
- An explanation of the use case (`README.md`)
24+
- Files that leverage codegen for powerful applications
4225

4326
## Learn More
4427

4528
- [Documentation](https://docs.codegen.com)
46-
- [Getting Started Guide](https://docs.codegen.com/introduction/getting-started)
47-
- [Tutorials](https://docs.codegen.com/tutorials/at-a-glance)
48-
- [API Reference](https://docs.codegen.com/api-reference)
29+
- [API](https://docs.codegen.com/introduction/api)
30+
- [Agent Capabilities](https://docs.codegen.com/introduction/capabilities)
31+
- [Prompting](https://docs.codegen.com/introduction/prompting)
32+
- [Community](https://docs.codegen.com/introduction/community)
33+
4934

5035
## Contributing
5136

‎examples/codecov_agent_trigger/README.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Codecov Agent Trigger
2+
3+
[![Documentation](https://img.shields.io/badge/docs-docs.codegen.com-blue)](https://docs.codegen.com)
4+
5+
This example demonstrates how to automatically trigger a Codegen agent to fix code coverage issues when a pull request's coverage falls below a specified threshold. The script integrates with GitHub Actions and Codecov to maintain high code quality standards.
6+
7+
## Overview
8+
9+
The codecov agent trigger consists of two main components:
10+
11+
1. **Coverage Report Processor** (`process_coverage_report.py`)
12+
- Parses Codecov XML reports
13+
- Evaluates coverage against thresholds
14+
- Triggers Codegen agent when needed
15+
16+
2. **Agent Prompt Generator** (`generate_codecov_agent_prompt.py`)
17+
- Creates contextual prompts for the Codegen agent
18+
- Provides necessary information for test generation
19+
- Includes setup and test execution instructions
20+
21+
## How It Works
22+
23+
The workflow operates in several steps:
24+
25+
1. **Coverage Analysis**
26+
```python
27+
coverage_data = parse_coverage_xml(xml_file)
28+
if coverage_data["coverage_percentage"] < threshold:
29+
# Trigger agent to fix coverage
30+
```
31+
- Parses XML coverage reports
32+
- Extracts key metrics (line coverage, branch coverage)
33+
- Compares against defined threshold (77%)
34+
35+
2. **Agent Triggering**
36+
```python
37+
from codegen import Agent
38+
39+
new_agent = Agent(token=token, org_id=ORG_ID)
40+
second_task = new_agent.run(generate_codecov_agent_prompt(pr_number, repo))
41+
```
42+
- Creates new Codegen agent instance
43+
- Generates contextual prompt
44+
- Initiates automated fix process
45+
46+
3. **GitHub Integration**
47+
```yaml
48+
- name: Codecov Agent Trigger
49+
env:
50+
PR_NUMBER: ${{ github.event.number }}
51+
REPO: ${{ github.repository }}
52+
TOKEN: ${{ secrets.CODEGEN_AGENT_TOKEN }}
53+
run: |
54+
python process_coverage_report.py coverage.xml $PR_NUMBER $REPO $TOKEN
55+
```
56+
- Runs as part of CI/CD pipeline
57+
- Passes PR context to processor
58+
- Manages authentication securely
59+
60+
## Setup
61+
62+
1. Install dependencies:
63+
```bash
64+
pip install codegen
65+
```
66+
67+
2. Configure GitHub secrets:
68+
- `CODECOV_TOKEN`: Your Codecov API token
69+
- `CODEGEN_AGENT_TOKEN`: Your Codegen agent token
70+
- Get your codegen token [here](https://www.codegen.sh/token)
71+
72+
3. Add to your GitHub Actions workflow:
73+
```yaml
74+
- name: Upload coverage reports to Codecov
75+
uses: codecov/codecov-action@v4.5.0
76+
with:
77+
token: ${{ secrets.CODECOV_TOKEN }}
78+
file: coverage.xml
79+
80+
- name: Codecov Agent Trigger
81+
run: python process_coverage_report.py coverage.xml $PR_NUMBER $REPO $TOKEN
82+
```
83+
84+
## Usage
85+
86+
The script is typically run automatically as part of your CI/CD pipeline, but you can also run it manually:
87+
88+
```bash
89+
python process_coverage_report.py <coverage_xml_file> <pr_number> <repo> <token>
90+
```
91+
92+
Arguments:
93+
- `coverage_xml_file`: Path to the coverage XML report
94+
- `pr_number`: GitHub PR number
95+
- `repo`: GitHub repository name
96+
- `token`: Codegen agent token
97+
98+
## Output
99+
100+
When coverage is below threshold:
101+
```
102+
WARNING: Coverage 75.50% is below threshold of 77%
103+
Agent will be notified.
104+
Agent has been notified. URL: https://codegen.com/tasks/123
105+
```
106+
107+
When coverage meets threshold:
108+
```
109+
Coverage is above threshold.
110+
```
111+
112+
## Coverage Metrics
113+
114+
The script tracks several key metrics:
115+
- Line coverage
116+
- Branch coverage
117+
- Overall coverage percentage
118+
- Lines covered vs. total lines
119+
- Branches covered vs. total branches
120+
121+
## Configuration
122+
123+
The default coverage threshold is set to 77%, but you can modify this in `process_coverage_report.py`:
124+
125+
```python
126+
threshold = 77 # Modify this value to change the coverage threshold
127+
```
128+
129+
## Learn More
130+
131+
- [Codecov Documentation](https://docs.codecov.com)
132+
- [Codegen Documentation](https://docs.codegen.com)
133+
- [GitHub Actions Documentation](https://docs.github.com/en/actions)
134+
135+
## Contributing
136+
137+
Feel free to submit issues and enhancement requests! We welcome contributions to improve the codecov agent trigger functionality.

‎examples/codecov_agent_trigger/action.yml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: Backend Unit Tests
2+
3+
on:
4+
push:
5+
branches:
6+
- "main"
7+
pull_request:
8+
branches:
9+
- "main"
10+
workflow_dispatch:
11+
12+
jobs:
13+
unit-tests:
14+
# changing the following value will significantly affect github's billing. Be careful and consult with the team before changing it.
15+
runs-on: ubuntu-latest-8
16+
concurrency:
17+
group: ${{ github.workflow }}-${{ github.ref }}
18+
cancel-in-progress: ${{ github.event_name != 'push'}}
19+
20+
steps:
21+
- uses: actions/checkout@v4
22+
- name: Setup backend
23+
id: setup
24+
uses: ./.github/actions/setup-backend
25+
26+
- name: Test with pytest
27+
if: ${{ steps.setup.outputs.skip == 'false' }}
28+
timeout-minutes: 5
29+
run: |
30+
cd path/to/your/package
31+
ENV=staging \
32+
PYTHONPATH=. \
33+
uv run pytest \
34+
-n auto \
35+
-vv \
36+
--cov \
37+
--cov-report=xml \
38+
path/to/your/tests
39+
40+
- name: Publish Test Report (Unit tests)
41+
uses: mikepenz/action-junit-report@v4
42+
if: ${{ steps.setup.outputs.skip == 'false' && (success() || failure()) }} # always publish report even if the tests fail
43+
continue-on-error: true
44+
with:
45+
report_paths: "**/build/test-results/test/TEST.xml"
46+
detailed_summary: true
47+
check_name: "JUnit Test Report - ${{ github.workflow }}"
48+
annotate_only: true
49+
50+
- name: Upload coverage reports to Codecov
51+
if: ${{ steps.setup.outputs.skip == 'false' && (success() || failure()) }} # always upload coverage reports even if the tests fail
52+
continue-on-error: true
53+
uses: codecov/codecov-action@v4.5.0
54+
with:
55+
token: ${{ secrets.CODECOV_TOKEN }}
56+
file: path/to/your/coverage.xml
57+
disable_search: true
58+
name: ${{ github.workflow }}
59+
60+
- name: Codecov Agent Trigger
61+
if: ${{ steps.setup.outputs.skip == 'false' && (success() || failure()) }}
62+
continue-on-error: true
63+
env:
64+
PR_NUMBER: ${{ github.event.number }}
65+
REPO: ${{ github.repository }}
66+
TOKEN: ${{ secrets.CODEGEN_AGENT_TOKEN }}
67+
run: |
68+
cd path/to/your/package
69+
uv run python process_coverage_report.py coverage.xml $PR_NUMBER $REPO $TOKEN

‎examples/codecov_agent_trigger/generate_codecov_agent_prompt.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
def generate_codecov_agent_prompt(pr_number, repo_name):
2+
return f"""
3+
# Overview
4+
Hello Codegen, I just generated a new PR {pr_number} for repo {repo_name} however, it failed the codecove (code coverage) check.
5+
Please view the PR checks, see the codecov report, and **update the PR** to fix the codecov check by adding any missing tests.
6+
7+
Please make sure to run the tests you add locally to ensure they are working.
8+
9+
DO NOT COMMIT TESTS IF THEY ARE FAILING.
10+
11+
# Use these commands to run the tests:
12+
13+
# Install Python Dependencies
14+
```
15+
cd path/to/your/package
16+
uv venv
17+
source .venv/bin/activate
18+
uv pip install .
19+
```
20+
21+
22+
# Run Tests
23+
```
24+
chmod +x ./run_test.sh
25+
run_test.sh
26+
```
27+
28+
Good luck!
29+
"""

‎examples/codecov_agent_trigger/proces_coverage_report.py‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env python
2+
3+
import sys
4+
import xml.etree.ElementTree as ET
5+
from typing import Any
6+
7+
from codegen import Agent
8+
9+
from generate_codecov_agent_prompt import generate_codecov_agent_prompt
10+
11+
ORG_ID = 1
12+
13+
14+
def parse_coverage_xml(xml_file: str) -> dict[str, Any]:
15+
"""Parse the coverage.xml file and extract relevant information.
16+
17+
Args:
18+
xml_file: Path to the coverage XML file.
19+
20+
Returns:
21+
Dictionary containing parsed coverage data.
22+
"""
23+
try:
24+
tree = ET.parse(xml_file)
25+
root = tree.getroot()
26+
27+
# Extract overall coverage statistics
28+
coverage_data = {
29+
"version": root.get("version"),
30+
"timestamp": root.get("timestamp"),
31+
"lines_covered": int(root.get("lines-covered", 0)),
32+
"lines_valid": int(root.get("lines-valid", 0)),
33+
"line_rate": float(root.get("line-rate", 0)),
34+
"branches_covered": int(root.get("branches-covered", 0)),
35+
"branches_valid": int(root.get("branches-valid", 0)),
36+
"branch_rate": float(root.get("branch-rate", 0)),
37+
"complexity": float(root.get("complexity", 0)),
38+
}
39+
40+
# Calculate overall coverage percentage
41+
if coverage_data["lines_valid"] > 0:
42+
coverage_data["coverage_percentage"] = coverage_data["lines_covered"] / coverage_data["lines_valid"] * 100
43+
else:
44+
coverage_data["coverage_percentage"] = 0
45+
46+
return coverage_data
47+
48+
except Exception as e:
49+
print(f"Error parsing coverage XML: {e}")
50+
return {}
51+
52+
53+
def main():
54+
"""Main function to process the coverage report."""
55+
if len(sys.argv) < 5:
56+
print("Usage: python process_coverage_report.py <coverage_xml_file> <pr_number> <repo> <token>")
57+
sys.exit(1)
58+
59+
xml_file = sys.argv[1]
60+
pr_number = sys.argv[2]
61+
repo = sys.argv[3]
62+
token = sys.argv[4]
63+
64+
coverage_data = parse_coverage_xml(xml_file)
65+
coverage_data["pr_number"] = pr_number
66+
coverage_data["repo"] = repo
67+
coverage_data["token"] = token
68+
69+
if not coverage_data:
70+
print("Failed to parse coverage data")
71+
sys.exit(1)
72+
73+
# Example: Check if coverage meets a threshold
74+
threshold = 77 # 77% coverage threshold
75+
if coverage_data["coverage_percentage"] < threshold:
76+
print(f"\nWARNING: Coverage {coverage_data['coverage_percentage']:.2f}% is below threshold of {threshold}%")
77+
print("Agent will be notified.")
78+
new_agent = Agent(token=token, org_id=ORG_ID)
79+
second_task = new_agent.run(generate_codecov_agent_prompt(pr_number, repo))
80+
print(f"Agent has been notified. URL: {second_task.web_url}")
81+
else:
82+
print("Coverage is above threshold.")
83+
84+
85+
if __name__ == "__main__":
86+
main()

‎examples/codecov_agent_trigger/run_test.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#! /bin/bash
2+
3+
# Set environment variables
4+
export ENV_VAR_1=1
5+
export ENV_VAR_2=2
6+
export ENV_VAR_3=3
7+
8+
# Run the tests
9+
cd path/to/your/package
10+
pytest -v --cov=./path/to/your/tests --cov-report=xml:coverage.xml "$@"

0 commit comments

Comments
 (0)