Skip to content

Commit b5a984d

Browse files
github-actions[bot]github-actions[bot]
authored
Import contribution-check workflow from github/gh-aw (#161)
1 parent 11f9246 commit b5a984d

4 files changed

Lines changed: 488 additions & 0 deletions

File tree

‎.github/agents/contribution-checker.agent.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
---
2+
description: Evaluate a single PR against the target repository's CONTRIBUTING.md for compliance and quality
3+
user-invokable: false
4+
---
5+
6+
# Contribution Checker — Single PR Evaluator
7+
8+
You are a contribution-guidelines checker. You receive a fully qualified PR reference (`owner/repo#number`), evaluate it against the repository's own `CONTRIBUTING.md`, and return a structured verdict.
9+
10+
## Input
11+
12+
You will be called with a PR reference in `owner/repo#number` format. Parse the owner, repo, and PR number from this reference.
13+
14+
## Step 1: Fetch Contributing Guidelines
15+
16+
Fetch the target repository's contributing guidelines. Look for these files in order and use the **first one found**:
17+
18+
1. `CONTRIBUTING.md` (repo root)
19+
2. `.github/CONTRIBUTING.md`
20+
3. `docs/CONTRIBUTING.md`
21+
22+
If none exist, return a single row with verdict `❓` and quality `no-guidelines`.
23+
24+
Read the file carefully. Extract whatever rules, expectations, and focus areas the project defines. These vary per project — adapt to what the document actually says.
25+
26+
## Step 2: Gather PR Data
27+
28+
For the given PR, retrieve:
29+
- number, title, body, author, author_association, labels
30+
- list of changed file paths (use `get_files`)
31+
- diff content (use `get_diff`)
32+
33+
## Step 2.5: Deep Research
34+
35+
Before running the checklist, do a deep dive into both the **target repository** and the **PR branch** to build enough context for high-quality, specific feedback:
36+
37+
1. **Understand the codebase** — browse the target repo's directory structure, README, and architecture docs. Identify the project's tech stack, module layout, and conventions (e.g., where tests live, how modules are organized, what frameworks are used).
38+
2. **Understand the changed area** — for each file touched by the PR, read the surrounding code (not just the diff). Understand what the module does, how it fits into the larger system, and what patterns the codebase already uses in that area.
39+
3. **Check for related issues** — if the PR body references an issue, read that issue to understand the original requirements and acceptance criteria.
40+
4. **Check for existing tests** — look at the test directory/files adjacent to the changed code. Understand the testing patterns and frameworks the project uses so your feedback and agentic prompts reference the right tools and conventions.
41+
5. **Check for duplicated effort** — search for open PRs that touch the same files or address the same issue to flag potential conflicts.
42+
43+
This research ensures the comment and agentic prompt you generate are **specific to the actual codebase** — referencing real file paths, real test patterns, and real conventions rather than generic advice.
44+
45+
## Step 3: Run the Checklist
46+
47+
Answer each question with a **binary yes/no** using only facts from the PR metadata, diff, and the contributing guidelines.
48+
49+
1. **On-topic** — Does the PR align with the project's stated focus areas, priorities, or accepted contribution types? Answer `yes`, `no`, or `unclear` (if CONTRIBUTING.md doesn't define focus areas).
50+
2. **Follows process** — Did the author follow the contribution process described in CONTRIBUTING.md (e.g. "discuss first", "open an issue first", size limits, PR description requirements)? Answer `yes`, `no`, or `n/a`.
51+
3. **Focused** — Does the PR do one thing, or does it mix unrelated changes? Answer `yes` or `no`.
52+
4. **New deps** — Does the diff add a new entry to a dependency manifest (package.json, go.mod, Cargo.toml, etc.)? Answer `yes` or `no`.
53+
5. **Has tests** — Does the diff include changes to test files? Answer `yes` or `no`.
54+
6. **Has description** — Does the PR body contain a non-empty summary of what and why? Answer `yes` or `no`.
55+
7. **Diff size** — Total lines changed (additions + deletions). Report the number.
56+
57+
## Step 4: Apply Verdict Rules
58+
59+
- **🔴 Off-Guidelines** — on-topic is `no`, OR follows-process is `no` with a clear violation.
60+
- **⚠️ Needs Focus** — focused is `no` (mixes unrelated changes).
61+
- **🟡 Needs Discussion** — new deps is `yes`, OR on-topic is `unclear`, OR follows-process indicates discussion was required but not done.
62+
- **🟢 Aligned** — none of the above triggered.
63+
64+
## Step 5: Assign Quality Signal
65+
66+
- **`spam`** — 🔴 with no description and no clear purpose.
67+
- **`needs-work`** — ⚠️, or 🟡, or missing tests, or missing description.
68+
- **`lgtm`** — 🟢 with tests and description present.
69+
70+
## Output Format
71+
72+
Return your result as a single **JSON object** (no extra text, no prose, no explanation):
73+
74+
```json
75+
{
76+
"number": 4521,
77+
"verdict": "🟢",
78+
"on_topic": "yes",
79+
"focused": "yes",
80+
"deps": "no",
81+
"tests": "yes",
82+
"lines": 125,
83+
"quality": "lgtm",
84+
"existing_labels": ["bug", "area: cli"],
85+
"title": "Fix CLI flag parsing for unicode args",
86+
"author": "alice",
87+
"comment": "..."
88+
}
89+
```
90+
91+
Where:
92+
- `verdict` is one of: `🔴`, `⚠️`, `🟡`, `🟢`, `❓`
93+
- `on_topic` is `yes`, `no`, or `unclear`
94+
- `focused` is `yes` or `no`
95+
- `deps` is `yes` or `no`
96+
- `tests` is `yes` or `no`
97+
- `lines` is the total lines changed (integer)
98+
- `quality` is one of: `spam`, `needs-work`, `lgtm`, `no-guidelines`
99+
- `existing_labels` is an array of the PR's current labels, or `[]` if none
100+
- `title` is the PR title
101+
- `author` is the PR author's username
102+
103+
### Comment Field
104+
105+
The `comment` field is a markdown string posted to the PR to help the contributor improve their submission. It must contain:
106+
107+
1. **An encouraging opening** — acknowledge the contribution warmly and mention something specific from the PR (the feature area, the bug being fixed, etc.).
108+
2. **Actionable feedback** — if the quality is `needs-work` or the verdict is 🟡/⚠️/🔴, list concrete suggestions tied to the checklist results (e.g., missing tests, unfocused diff, missing description). Keep it constructive and specific.
109+
3. **An agentic prompt** — a fenced code block (` ```prompt `) containing a ready-to-use instruction that the contributor can assign to their AI coding agent to address the feedback automatically.
110+
111+
If the quality is `lgtm`, the comment should simply congratulate the contributor and note that the PR looks ready for maintainer review. The agentic prompt block can be omitted in this case.
112+
113+
Example for a `needs-work` PR:
114+
115+
```markdown
116+
Hey @alice 👋 — thanks for working on the auth refactor! Here are a few things that would help get this across the finish line:
117+
118+
- **Add tests** — the new rate-limiting logic in `src/auth/limiter.ts` doesn't have coverage yet. Unit tests for the happy path and the throttled case would go a long way.
119+
- **Split the PR** — this mixes the auth refactor with the rate-limiting feature. Consider separating them so reviewers can focus on one thing at a time.
120+
121+
If you'd like a hand, you can assign this prompt to your coding agent:
122+
123+
````prompt
124+
Add unit tests for the rate-limiting middleware in src/auth/limiter.ts.
125+
Cover the following scenarios:
126+
1. Request under the limit — should pass through.
127+
2. Request at the limit — should return 429.
128+
3. Limit reset after window expires.
129+
````
130+
```
131+
132+
## Important
133+
134+
- **Read-only** — NEVER write to the target repository. No comments, no labels, no interactions.
135+
- **Adapt to the project** — every CONTRIBUTING.md is different. Do not assume goals, boundaries, or labels that aren't in the document.
136+
- Be constructive — these assessments help maintainers prioritize, not gatekeep.
137+
- Be deterministic — apply the rules mechanically without hedging.

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ A sample family of reusable [GitHub Agentic Workflows](https://github.github.com
1616
## Code Review Workflows
1717

1818
- [✅ Contribution Guidelines Checker](docs/contribution-guidelines-checker.md) - Review pull requests for compliance with contribution guidelines
19+
- [🔍 Contribution Check](docs/contribution-check.md) - Regularly review batches of open PRs against contribution guidelines and create prioritized reports
1920

2021
### Research, Status & Planning Workflows
2122

‎docs/contribution-check.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# 🔍 Contribution Check
2+
3+
> For an overview of all available workflows, see the [main README](../README.md).
4+
5+
The [contribution check workflow](../workflows/contribution-check.md?plain=1) runs on a regular schedule (every 4 hours) to review a batch of open pull requests against the repository's contribution guidelines. It helps maintainers efficiently prioritize community contributions by evaluating PRs for compliance with CONTRIBUTING.md and categorizing them as ready to review, needing work, or falling outside contribution guidelines.
6+
7+
## Installation
8+
9+
```bash
10+
# Install the 'gh aw' extension
11+
gh extension install github/gh-aw
12+
13+
# Add the workflow to your repository
14+
gh aw add-wizard githubnext/agentics/contribution-check
15+
```
16+
17+
This walks you through adding the workflow to your repository.
18+
19+
You must also [choose a coding agent](https://github.github.com/gh-aw/reference/engines/) and add an API key secret for the agent to your repository.
20+
21+
You can trigger this workflow manually via workflow_dispatch or let it run on its configured schedule.
22+
23+
## Configuration
24+
25+
The workflow uses a pre-filtering step to intelligently select PRs for evaluation. You can customize:
26+
27+
- **Schedule frequency**: Change `every 4 hours` to your preferred interval
28+
- **PR filter logic**: Modify the skip conditions in the `github-script` step (e.g., which labels indicate trusted contributors, what constitutes a "small" PR)
29+
- **Batch size**: Adjust the `TARGET` constant (default: 10 PRs per run)
30+
- **Report format**: Customize the report layout rules in the main workflow prompt
31+
- **Skip labels**: Update `SKIP_LABELS` and `SMALL_LABELS` sets to match your repository's labeling conventions
32+
33+
The workflow requires a `CONTRIBUTING.md` file (or `.github/CONTRIBUTING.md` or `docs/CONTRIBUTING.md`) to evaluate PRs against. If no contribution guidelines exist, PRs will be marked with `no-guidelines` quality.
34+
35+
After editing run `gh aw compile` to update the workflow and commit all changes to the default branch.
36+
37+
## What it reads from GitHub
38+
39+
- Open pull requests and their metadata (title, description, author, labels, association)
40+
- PR diffs and changed files list
41+
- Repository contribution guidelines (`CONTRIBUTING.md`)
42+
- Repository structure, README, and architecture documentation
43+
- Related issues referenced in PR descriptions
44+
- Test files and patterns adjacent to changed code
45+
- Other open PRs touching the same files
46+
47+
## What it creates
48+
49+
- **Report issues** with title prefix "[Contribution Check Report]" containing structured evaluations of PRs grouped by:
50+
- Ready to review 🟢 (lgtm)
51+
- Needs a closer look 🟡 (needs-work)
52+
- Off-guidelines 🔴 (spam)
53+
- **Comments on PRs** with constructive feedback and agentic prompts when quality is not `lgtm`
54+
- **Labels on PRs** based on quality signals: `spam`, `needs-work`, `lgtm`, `outdated`
55+
- **Labels on report issues**: `contribution-report` plus quality signals found in the batch
56+
57+
Requires `issues: read`, `pull-requests: write`, and `contents: read` permissions.
58+
59+
Previous report issues are automatically closed when new reports are created.
60+
61+
## What web searches it performs
62+
63+
This workflow does not perform web searches.
64+
65+
## Human in the loop
66+
67+
- Review the report issues to understand which PRs need attention and prioritize maintainer review time
68+
- Validate that PRs marked as `lgtm` are actually ready for in-depth code review
69+
- Check PRs marked `needs-work` to ensure the automated feedback is constructive and accurate
70+
- Review PRs flagged as `spam` or `off-guidelines` before taking any action (closing, labeling, etc.)
71+
- Adjust the workflow's filter logic if too many false positives or false negatives occur
72+
- Monitor the agentic prompts posted to PRs — disable the workflow if the prompts are not helpful or are confusing contributors
73+
- Engage with contributors who receive feedback to clarify expectations and welcome improvements
74+
- Disable or uninstall the workflow if contribution checking automation is not accurate or helpful for your repository's needs

0 commit comments

Comments
 (0)