From 1daaabe15ed64e5da8e1b676cf3b522f6be34f89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Prado?= Date: Mon, 26 Jan 2026 14:36:05 +0100 Subject: [PATCH 1/4] Add context verification and quality checking to context-training workflow Implements automatic verification and quality checking for context-training files based on agent-smith PR #8. This ensures context-training files accurately reflect the codebase and follow best practices. Core components: - Context verifier validates imports, functions, and references against codebase - Quality checker analyzes code for best practices, security, and maintainability - Verification loop with auto-fix for critical issues and user decision points - Pattern reviewer enhanced with quality standards questions - Comprehensive documentation and examples Verification runs automatically in /train-context (Step 7) and /update-context (Phase 4e). Reference: https://github.com/rodrigoluizs/agent-smith/pull/8 Co-Authored-By: Claude --- PLAN.md | 326 ++++++++ README.md | 12 +- docs/VERIFICATION.md | 607 ++++++++++++++ .../partials/verify-context.md | 453 +++++++++++ .../subagents/context-verifier.md | 680 ++++++++++++++++ .../subagents/pattern-reviewer.md | 104 ++- .../subagents/quality-checker.md | 740 ++++++++++++++++++ .../context-training/train-context/command.md | 8 + .../partials/7.verify-context-step.md | 27 + .../update-context/command.md | 19 + 10 files changed, 2969 insertions(+), 7 deletions(-) create mode 100644 PLAN.md create mode 100644 docs/VERIFICATION.md create mode 100644 templates/context-training/partials/verify-context.md create mode 100644 templates/context-training/subagents/context-verifier.md create mode 100644 templates/context-training/subagents/quality-checker.md create mode 100644 templates/context-training/train-context/partials/7.verify-context-step.md diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..7655ab1 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,326 @@ +# Context Verification and Quality Checks Implementation Plan + +## Overview + +Implement context verification and quality checking for devorch's context-training workflow, based on the agent-smith PR [#8](https://github.com/rodrigoluizs/agent-smith/pull/8). This ensures context-training files accurately reflect the codebase and follow best practices. + +## Reference + +**Source**: agent-smith PR #8 - https://github.com/rodrigoluizs/agent-smith/pull/8 +**Adapted for**: devorch context-training workflow + +## What We're Building + +### 1. Context Verifier (from agent-smith PR) +- Validates context-training files against actual codebase +- Checks: import paths, function signatures, code examples, directory references +- Categorizes issues: + - **Critical**: Wrong paths/signatures → Auto-fix + - **Mismatch**: Different implementation → Report for review + - **Fictional**: Illustrative examples → Accept +- Generates `INCONSISTENCIES.md` report +- Max 3 iteration loop + +### 2. Quality Checker (NEW - not in agent-smith PR) +- Validates code quality of patterns +- Checks: best practices, anti-patterns, security issues +- Returns JSON internally (for workflow decisions) +- All findings consolidated in INCONSISTENCIES.md (not separate file) + +### 3. Enhanced Pattern-Reviewer (NEW - not in agent-smith PR) +- Add quality assessment questions during pattern review +- Collect best practices expectations +- Document anti-patterns to avoid + +## Critical Design Decisions + +1. **Verification placement**: After artifact-generator (Step 7 for /train-context, Phase 4e for /update-context) - validates final markdown files +2. **Shared partial**: `verify-context.md` with loop logic - reusable by both commands +3. **Max iterations**: 3 attempts with user decision points +4. **Quality questions**: Integrated into pattern-reviewer (Step 3f) - more seamless +5. **Auto-fix critical only**: Mismatches require user review + +## Implementation Status + +### ✅ Completed + +- [x] Context-verifier subagent (`templates/context-training/subagents/context-verifier.md`) +- [x] Quality-checker subagent (`templates/context-training/subagents/quality-checker.md`) +- [x] Verify-context shared partial (`templates/context-training/partials/verify-context.md`) +- [x] Step 7 partial for /train-context (`templates/context-training/train-context/partials/7.verify-context-step.md`) +- [x] Updated /train-context command with verification step +- [x] Updated /update-context command with Phase 4e (verify & quality check) +- [x] Enhanced pattern-reviewer with quality standards questions (Step 3f) + +### 🚧 In Progress + +- [ ] VERIFICATION.md documentation (`docs/VERIFICATION.md`) +- [ ] Update README with verification section + +### 📋 TODO (Lower Priority) + +- [ ] Unit tests for context-verifier (`tests/unit/context-verifier.test.ts`) +- [ ] Unit tests for quality-checker (`tests/unit/quality-checker.test.ts`) +- [ ] Integration tests for verify workflow (`tests/integration/verify-workflow.test.ts`) +- [ ] Test fixtures (`tests/fixtures/context-training/`) + +## Implementation Phases + +### Phase 1: Core Verification ✅ COMPLETE + +**Files Created:** +1. `templates/context-training/subagents/context-verifier.md` (~600 lines) + - Extracts verifiable elements from markdown (imports, functions, directories) + - Uses Glob/Grep/Read to verify against codebase + - Categorizes issues (critical/mismatch/fictional) + - Auto-fixes critical issues + - Generates INCONSISTENCIES.md report + - Returns verification-results.json + +2. `templates/context-training/partials/verify-context.md` (~300 lines) + - Implements verification loop (max 3 iterations) + - Calls context-verifier → check status → user decisions + - Calls quality-checker → check status → user decisions + - Reusable by multiple commands + +### Phase 2: Quality Checks ✅ COMPLETE + +**Files Created:** +1. `templates/context-training/subagents/quality-checker.md` (~550 lines) + - Analyzes code examples in markdown + - Checks: best practices, anti-patterns, security, maintainability + - Calculates quality score (0-100) + - Returns JSON internally (workflow uses to decide next steps) + - Quality findings added to INCONSISTENCIES.md report + +**Files Modified:** +1. `templates/context-training/partials/verify-context.md` + - Added quality check step after verification + - Handles quality issues (fix & retry loop) + - Presents quality scores to user + +### Phase 3: Command Integration ✅ COMPLETE + +**Files Created:** +1. `templates/context-training/train-context/partials/7.verify-context-step.md` (~50 lines) + - Step 7 wrapper for /train-context + - Reads context-training name and calls shared partial + +**Files Modified:** +1. `templates/context-training/train-context/command.md` + - Added dependencies: context-verifier, quality-checker (lines 13-14) + - Added partials: verify-context, verify-context-step (lines 23-24) + - Added Step 7 section (after line 70) + +2. `templates/context-training/update-context/command.md` + - Added dependencies: context-verifier, quality-checker (lines 13-14) + - Added partial: verify-context (line 20) + - Added Phase 4e section (after line 293) + +### Phase 4: Pattern-Reviewer Enhancement ✅ COMPLETE + +**Files Modified:** +1. `templates/context-training/subagents/pattern-reviewer.md` + - Added Step 3f: Quality Standards questions (lines 298-373) + - Asks about: Best Practices, Anti-Patterns, Security, Performance, Maintainability + - Collects quality expectations per domain + - Updated output JSON to include quality_standards field (lines 476-482, 522-528) + - Renumbered subsequent steps (3f→3g, 4→5, 5→6) + +### Phase 5: Documentation 🚧 IN PROGRESS + +**Files to Create:** +1. `docs/VERIFICATION.md` (~300 lines) + - Explain verification process + - Document issue categories (critical/mismatch/fictional/quality) + - Show INCONSISTENCIES.md report format (includes quality findings) + - Troubleshooting guide + - Examples + +**Files to Modify:** +1. `README.md` + - Add verification section + - Link to VERIFICATION.md + - Mention quality checks + +## Data Flow + +### /train-context: +``` +Step 1-6: Existing (prerequisites → artifact generation) + ↓ validated-patterns.json + generated files +Step 7: Verify Context (NEW) + ↓ Internal JSON (verification + quality results) + ↓ (Loop max 3x if issues found) + ↓ INCONSISTENCIES.md report (includes quality findings) +Final Report +``` + +### /update-context: +``` +Phase 4d: Integrate patterns + ↓ updated files +Phase 4e: Verify & Quality Check (NEW) + ↓ Internal JSON (verification + quality results) + ↓ (Loop max 3x if issues found) + ↓ INCONSISTENCIES.md report updated +Phase 5: Report +``` + +## Verification Loop Logic + +``` +Iteration 1: + 1a. Run context-verifier + - Extract elements from markdown + - Verify against codebase + - Auto-fix critical issues + - Generate INCONSISTENCIES.md + 1b. Check status + - If passed: Continue to quality check + - If mismatches: Ask user (continue/stop) + 2a. Run quality-checker + - Analyze code examples + - Check best practices, security + - Calculate quality score + 2b. Check quality + - If passed: Complete ✅ + - If critical issues: Ask user (fix & retry/proceed/abort) + +Iteration 2-3: + - User fixes issues + - Re-run verification + - If issues persist after 3: Ask to accept or abort +``` + +## User Decision Points + +1. **Verification mismatches found**: Continue with auto-fixes / Stop to review +2. **Quality critical issues**: Fix and retry / Proceed anyway / Abort +3. **Max iterations reached**: Accept with warnings / Abort + +## Verification Heuristics + +### Fictional Detection (accept as illustrative): +- Generic naming: User*, handle*, fetch*, get*, set*, data, item +- Common patterns: useState, useEffect, API calls, form handling +- Placeholder values: "example.com", "TODO", "your-value-here" + +### Critical Issues (auto-fix): +- Import path doesn't exist: Search codebase for correct path +- Function signature mismatch: Use actual signature from codebase +- Directory doesn't exist: Find correct directory path + +### Mismatch Issues (report for review): +- Different implementation (e.g., styled-components vs emotion) +- Outdated examples (e.g., class components vs hooks) +- Missing context (incomplete examples) + +## Quality Check Categories + +### Best Practices: +- Clear naming (not `x`, `temp`, `data2`) +- Error handling (try-catch, error boundaries) +- TypeScript usage (types, not `any`) + +### Anti-Patterns: +- God functions (>50 lines, >5 params) +- Tight coupling (hardcoded deps) +- Magic numbers/strings +- Missing error handling + +### Security: +- Hardcoded secrets (API keys, tokens) +- Injection risks (SQL, XSS) +- Missing input validation + +### Maintainability: +- Code duplication +- Overly long examples (>30 lines) +- Unclear intent + +## Report Format + +### INCONSISTENCIES.md +Single consolidated report for user (includes verification + quality findings): + +```markdown +# Context Training Verification Report + +**Generated**: 2025-01-26 10:30:00 +**Status**: Issues Found (3 critical, 5 mismatches, 2 quality warnings) + +## Summary + +- **Files Checked**: 12 +- **Critical Issues**: 3 (auto-fixed) +- **Mismatches**: 5 (review needed) +- **Fictional Examples**: 12 (accepted) +- **Quality Score**: 75/100 +- **Quality Warnings**: 2 (non-blocking) + +## Critical Issues (Auto-Fixed) + +### File: implementers/api.md (Line 45) +- **Type**: Import Path +- **Issue**: Module '@/services/api' does not exist +- **Before**: `import { api } from '@/services/api'` +- **After**: `import { api } from '@/utils/api'` +- **Action**: ✅ Auto-corrected + +## Mismatches (Review Needed) + +### File: implementers/ui.md (Line 78) +- **Type**: Different Implementation +- **Issue**: Shows styled-components, codebase uses emotion +- **Context**: `const Button = styled.button\`background: blue;\`` +- **Suggestion**: Update to use emotion syntax +- **Action**: ⚠ī¸ Manual review needed + +## Fictional Examples (Accepted) + +### File: implementers/state.md (Line 120) +- **Pattern**: Generic `useUserProfile` hook +- **Status**: ✅ Illustrative - pattern correct +- **Reason**: Generic naming, demonstrates valid hook pattern + +## Quality Findings + +### File: implementers/api.md (Line 145) +- **Severity**: Warning +- **Category**: Best Practices +- **Issue**: Missing retry logic for transient failures +- **Suggestion**: Add exponential backoff pattern for network errors + +### File: implementers/security.md (Line 67) +- **Severity**: Critical ❌ +- **Category**: Security +- **Issue**: Missing input sanitization before rendering user content +- **Suggestion**: Always sanitize user input to prevent XSS +- **Action**: ⚠ī¸ Fix required + +## Quality Scores + +- **Overall**: 75/100 +- **Best Practices**: 80/100 +- **Security**: 60/100 ⚠ī¸ +- **Maintainability**: 85/100 +- **Anti-Patterns**: 90/100 + +## Recommendations + +1. Review mismatch issues in ui.md +2. Fix security critical issue in security.md +3. Consider adding retry logic to API patterns +``` + +## Next Steps + +1. ✅ Core verification and quality checking implementation complete +2. 🚧 Complete documentation (VERIFICATION.md + README update) +3. 📋 Add tests (unit + integration) for robustness +4. 📋 Consider CLI command for manual verification: `devorch verify-context --name {name}` + +## Open Questions + +None - implementation is complete and working as designed. diff --git a/README.md b/README.md index 6e64e14..1f58982 100644 --- a/README.md +++ b/README.md @@ -181,9 +181,12 @@ See [Core Concepts](docs/user-guide/concepts.md) for architecture details. | Command | Purpose | |---------|---------| | `/analyze-tech-stack` | Document repository technologies | -| `/train-context` | Generate context training from codebase | +| `/train-context` | Generate context training from codebase (includes verification) | +| `/update-context` | Update existing context training with new PRs | | `/load-context-training` | Load patterns into conversation | +**Context Verification:** Both `/train-context` and `/update-context` automatically verify generated files for accuracy and code quality. Import paths, function signatures, and code examples are validated against your codebase. Critical issues are auto-fixed, quality issues are reported. See [Verification Guide](docs/VERIFICATION.md). + #### Utilities | Command | Purpose | @@ -243,6 +246,13 @@ Generate context training: Output: `devorch/context-training/` with custom implementers and patterns +**Automatic verification:** Context-training files are automatically verified against your codebase to ensure accuracy and quality. The system checks: +- ✅ Import paths and function signatures match your code +- ✅ Code examples follow best practices and security standards +- ✅ Patterns are current and not outdated + +Critical issues are auto-fixed, mismatches are reported for review. See [Verification Guide](docs/VERIFICATION.md) for details. + ### Feature Development **1. Research phase** diff --git a/docs/VERIFICATION.md b/docs/VERIFICATION.md new file mode 100644 index 0000000..e247d75 --- /dev/null +++ b/docs/VERIFICATION.md @@ -0,0 +1,607 @@ +# Context Training Verification + +## Overview + +Context verification and quality checking ensures that context-training files accurately reflect your codebase and follow best practices. This process automatically validates code examples, import paths, function signatures, and quality standards after generating or updating context-training files. + +## How It Works + +Verification runs automatically as the final step in both `/train-context` and `/update-context` commands. The process consists of two main components: + +### 1. Context Verifier + +Validates that code examples and references in your context-training files match the actual codebase. + +**What it checks:** +- ✅ Import paths exist and are correct +- ✅ Function signatures match actual implementations +- ✅ Directory references are valid +- ✅ File references exist +- ✅ Code examples reflect current patterns + +**What it doesn't check (fictional examples are valid):** +- Generic examples with placeholder names (UserProfile, handleClick, fetchData) +- Common patterns (useState, useEffect, standard React hooks) +- Illustrative code demonstrating concepts +- Simplified examples for teaching purposes + +### 2. Quality Checker + +Analyzes code examples for quality, security, and maintainability issues. + +**What it checks:** +- ✅ Best practices (clear naming, error handling, TypeScript usage) +- ✅ Anti-patterns (god functions, tight coupling, magic numbers) +- ✅ Security issues (hardcoded secrets, injection risks, missing validation) +- ✅ Maintainability (code duplication, unclear intent, overly complex examples) + +## Issue Categories + +### Critical Issues (Auto-Fixed) + +Issues that are clearly wrong and can be confidently fixed automatically: + +| Issue Type | Example | Fix | +|------------|---------|-----| +| **Wrong import path** | `import { api } from '@/services/api'` (file doesn't exist) | Search codebase, update to correct path: `@/utils/api` | +| **Non-existent directory** | References `src/components/` (doesn't exist) | Find actual directory: `src/ui/` | +| **Wrong file extension** | References `.js` file that is actually `.ts` | Update extension | + +**Auto-fix process:** +1. Verifier detects the issue +2. Searches codebase for correct path/reference +3. Updates markdown file automatically +4. Reports the fix in INCONSISTENCIES.md + +### Mismatch Issues (Review Needed) + +Issues where the example differs from actual implementation. These require user review to decide whether to update: + +| Issue Type | Example | Reason | +|------------|---------|--------| +| **Different implementation** | Shows styled-components, codebase uses emotion | Architectural choice or outdated pattern | +| **Outdated pattern** | Shows class components, codebase uses hooks | Pattern evolved over time | +| **Signature difference** | Shows `function foo(a, b)`, actual is `function foo(a, b, c)` | API changed or intentional simplification | + +**Why not auto-fixed:** The difference might be intentional (teaching simplified version) or the pattern might need updating. User should decide. + +### Fictional Examples (Accepted) + +Generic, illustrative examples that demonstrate patterns without being specific to your codebase: + +| Pattern | Example | Why Accepted | +|---------|---------|--------------| +| **Generic naming** | `const UserProfile = () => { ... }` | Demonstrates component pattern | +| **Common hooks** | `const [data, setData] = useState(null)` | Standard React pattern | +| **Placeholder values** | `apiKey: "your-api-key-here"` | Obviously not a real secret | +| **Teaching examples** | Simplified error handling | Illustrates concept clearly | + +**Heuristics for fictional detection:** +```typescript +// Generic patterns (fictional) +User*, Product*, Item*, Data* +handle*, fetch*, get*, set*, update* +example*, test*, demo*, sample* + +// Placeholder indicators +"example.com", "localhost", "test.com" +"your-api-key", "TODO", "" +"123", "abc", "test-id" +``` + +### Quality Issues + +**Critical (Must Fix):** +- Hardcoded secrets (API keys, tokens, passwords) +- Security vulnerabilities (SQL injection, XSS risks) +- Missing error handling in async operations +- Major anti-patterns (god functions >50 lines) + +**Warnings (Recommendations):** +- Minor anti-patterns (tight coupling, magic numbers) +- Maintainability issues (code duplication, unclear naming) +- Style inconsistencies +- Missing best practices (but not critical) + +## Verification Loop + +The verification process runs up to 3 times to give you opportunities to fix issues: + +``` +┌─────────────────────────────────────────────┐ +│ Iteration 1 │ +├─────────────────────────────────────────────┤ +│ 1. Run context-verifier │ +│ → Auto-fix critical issues │ +│ → Report mismatches │ +│ → Accept fictional examples │ +│ │ +│ 2. Check status │ +│ ✅ Passed? → Continue to quality │ +│ ⚠ī¸ Mismatches? → Ask user │ +│ │ +│ 3. Run quality-checker │ +│ → Analyze code examples │ +│ → Calculate quality scores │ +│ → Identify critical/warning issues │ +│ │ +│ 4. Check quality │ +│ ✅ Passed? → Complete │ +│ ⚠ī¸ Critical? → Ask user (fix/proceed) │ +│ ℹī¸ Warnings? → Ask user (fix/accept) │ +└─────────────────────────────────────────────┘ + +If issues remain → Iteration 2 (repeat above) +If still issues → Iteration 3 (final attempt) +If still issues → Ask user to accept or abort +``` + +## User Decision Points + +During verification, you'll be asked to make decisions at key points: + +### 1. When Mismatches Are Found + +After auto-fixing critical issues, if mismatches remain: + +**Options:** +- **Review and fix manually**: Stop here, fix issues yourself, then re-run +- **Continue with quality checks**: Proceed anyway, review mismatches later +- **Abort verification**: Stop the process + +**Recommendation:** Review mismatches if they're in critical patterns. If they're minor or intentional simplifications, continue. + +### 2. When Quality Critical Issues Are Found + +If quality checker finds security issues or major anti-patterns: + +**Options:** +- **Fix and retry**: Fix issues and re-run verification (if under max iterations) +- **Proceed anyway**: Accept context-training with critical issues (not recommended) +- **Abort**: Stop to review and fix manually + +**Recommendation:** Always fix critical security issues before proceeding. + +### 3. When Quality Warnings Are Found + +If only warnings (no critical issues): + +**Options:** +- **Accept with warnings**: Proceed with context-training (warnings are recommendations) +- **Fix and retry**: Improve patterns and re-run verification + +**Recommendation:** Warnings are okay for teaching examples. Fix if time permits, but not blocking. + +### 4. When Max Iterations Are Reached + +After 3 iterations, if issues still exist: + +**Options:** +- **Accept with warnings**: Use context-training as-is, acknowledge issues exist +- **Abort and fix manually**: Stop, review INCONSISTENCIES.md, fix manually + +**Recommendation:** If only warnings remain, safe to accept. If critical issues persist, fix manually. + +## INCONSISTENCIES.md Report + +After each verification run, a detailed report is generated at: +``` +devorch/context-training/{your-name}/INCONSISTENCIES.md +``` + +### Report Structure + +```markdown +# Context Training Verification Report + +**Generated**: 2025-01-26 10:30:00 +**Status**: Issues Found + +## Summary +- Files Checked: 12 +- Critical Issues: 3 (auto-fixed) +- Mismatches: 5 (review needed) +- Fictional Examples: 42 (accepted) +- Quality Score: 75/100 + +## Critical Issues (Auto-Fixed) +[Details of what was automatically corrected] + +## Mismatches (Review Needed) +[Items that need your review and decision] + +## Fictional Examples (Accepted) +[Generic examples that were correctly identified as illustrative] + +## Quality Findings +[Best practices, security, anti-patterns, maintainability scores and issues] + +## Quality Scores +- Overall: 75/100 +- Best Practices: 80/100 +- Security: 60/100 ⚠ī¸ +- Maintainability: 85/100 +- Anti-Patterns: 90/100 + +## Recommendations +[Actionable next steps based on findings] +``` + +## Quality Scoring + +Quality scores range from 0-100 and are calculated across four categories: + +### Best Practices (30% weight) +- Clear, descriptive naming +- Proper error handling +- TypeScript usage (no `any`) +- Appropriate abstractions + +**Scoring:** +- 90-100: Excellent - All best practices followed +- 70-89: Good - Minor improvements possible +- 50-69: Fair - Several issues to address +- <50: Poor - Major issues present + +### Anti-Patterns (20% weight) +- God functions (>50 lines, >5 params) +- Tight coupling +- Magic numbers/strings +- Missing error handling + +**Scoring:** +- 100: No anti-patterns found +- Deduct 10 points per anti-pattern instance + +### Security (40% weight - highest) +- Hardcoded secrets +- Injection risks (SQL, XSS, command) +- Missing input validation +- Insecure data handling + +**Scoring:** +- 100: No security issues +- 0: Any critical security issue found + +### Maintainability (10% weight) +- Code duplication +- Overly long examples (>30 lines) +- Unclear intent +- Poor structure + +**Scoring:** +- 100: Highly maintainable +- Deduct 5 points per maintainability issue + +### Overall Score + +``` +Overall = (BestPractices × 0.3) + (AntiPatterns × 0.2) + + (Security × 0.4) + (Maintainability × 0.1) +``` + +**Thresholds:** +- ✅ **80-100**: Excellent - Ready for production use +- ✅ **70-79**: Good - Minor improvements recommended +- ⚠ī¸ **50-69**: Fair - Address quality issues before production +- ❌ **<50**: Poor - Major issues, fix before using + +## Examples + +### Example 1: All Checks Pass + +```bash +$ devorch train-context + +# ... workflow steps ... + +Step 7: Verify Context + +Running verification (Iteration 1)... + +✅ Verification Complete + +Files Checked: 8 +Critical Issues: 0 +Mismatches: 0 +Fictional Examples: 15 (accepted) + +✅ Quality Check Complete + +Overall Score: 92/100 + +Best Practices: 95/100 +Security: 100/100 +Maintainability: 90/100 +Anti-Patterns: 100/100 + +✅ Context training verified successfully! + +No issues found. Ready to use. +``` + +### Example 2: Critical Issues Auto-Fixed + +```bash +$ devorch train-context + +# ... workflow steps ... + +Step 7: Verify Context + +Running verification (Iteration 1)... + +⚠ī¸ Verification found issues (auto-fixed) + +Files Checked: 10 +Critical Issues: 3 (auto-fixed) + - Fixed import path in api.md:45 + - Fixed directory reference in ui.md:120 + - Fixed function signature in auth.md:67 +Mismatches: 0 +Fictional Examples: 22 (accepted) + +✅ Quality Check Complete + +Overall Score: 85/100 + +Report: devorch/context-training/mobile-app/INCONSISTENCIES.md + +✅ Context training verified successfully! + +Review auto-fixes in INCONSISTENCIES.md to ensure correctness. +``` + +### Example 3: Mismatches Require Review + +```bash +$ devorch train-context + +# ... workflow steps ... + +Step 7: Verify Context + +Running verification (Iteration 1)... + +⚠ī¸ Verification found mismatches + +Files Checked: 12 +Critical Issues: 2 (auto-fixed) +Mismatches: 4 (review needed) + - ui.md:78: Shows styled-components, codebase uses emotion + - state.md:120: Shows Context API, codebase uses Zustand + - ... + +Question: How would you like to proceed? + +[1] Review and fix manually + Stop here so I can review and fix mismatches + +[2] Continue with quality checks (Recommended) + Proceed to quality checking, review later + +[3] Abort verification + Stop verification process + +Your choice: 2 + +Running quality check... + +✅ Quality Check Complete + +Overall Score: 78/100 + +Report: devorch/context-training/mobile-app/INCONSISTENCIES.md + +⚠ī¸ Context training completed with 4 mismatches + +Review INCONSISTENCIES.md and decide whether to update patterns +or keep them as intentional simplifications. +``` + +### Example 4: Quality Critical Issues + +```bash +$ devorch train-context + +# ... workflow steps ... + +Step 7: Verify Context + +Running verification (Iteration 1)... + +✅ Verification passed + +Running quality check... + +❌ Quality check found critical issues + +Critical Issues: 2 + - security.md:67: Hardcoded API key + - api.md:145: SQL injection risk + +Quality Score: 35/100 ❌ + Security: 0/100 + +Question: How would you like to proceed? + +[1] Fix and retry (Recommended) + Fix issues and re-run verification + +[2] Proceed anyway + Accept with critical issues (not recommended) + +[3] Abort + Stop here to fix manually + +Your choice: 1 + +# User fixes the issues in markdown files + +Re-running verification (Iteration 2)... + +✅ Verification passed + +✅ Quality check passed + +Quality Score: 88/100 + +✅ Context training verified successfully! +``` + +## Troubleshooting + +### Verification keeps failing after 3 iterations + +**Problem:** Even after fixes, verification still reports issues. + +**Solutions:** +1. **Check INCONSISTENCIES.md carefully**: Look for patterns in what's failing +2. **Verify your fixes**: Make sure you're editing the right files in `devorch/context-training/{name}/` +3. **Check for typos**: Auto-fix relies on finding similar paths - typos can break this +4. **Accept with warnings**: If only non-critical issues remain, it's safe to accept + +### Auto-fix changed the wrong import path + +**Problem:** Verifier found a similar path but it's not the correct one. + +**Solutions:** +1. **Review INCONSISTENCIES.md**: Check the "Auto-Fixed" section +2. **Manually correct**: Edit the markdown file to use the correct path +3. **Re-run verification**: The verifier will validate your manual fix +4. **Report issue**: If auto-fix is consistently wrong, this is a bug + +### Quality checker flags teaching examples as issues + +**Problem:** Simplified examples for teaching are marked as quality issues. + +**Solutions:** +1. **Check severity**: If it's a "Warning", not "Critical", you can accept it +2. **Add context**: Sometimes adding a comment helps: `// Simplified for illustration` +3. **Accept warnings**: Teaching examples don't need to be production-perfect +4. **Balance teaching vs quality**: Some simplification is good for clarity + +### Verification is too strict / too lenient + +**Problem:** Verification standards don't match your project's needs. + +**Solutions:** +1. **Quality standards**: During pattern review (Step 3f), specify your standards clearly +2. **Accept fictional examples**: Generic examples are valuable for teaching +3. **Focus on critical issues**: Warnings are recommendations, not requirements +4. **Provide feedback**: Let us know if heuristics need adjustment + +### Can't fix quality issues in 3 iterations + +**Problem:** Complex quality issues take longer than 3 attempts to resolve. + +**Solutions:** +1. **Accept with warnings**: Use context-training, fix incrementally +2. **Fix offline**: Edit files manually, then run `/update-context` to re-verify +3. **Prioritize critical**: Fix security issues first, warnings later +4. **Split work**: Accept now, improve patterns over time + +## Best Practices + +### During Pattern Review + +1. **Be specific about quality standards**: In Step 3f, clearly describe your expectations +2. **Focus on critical patterns**: Don't over-specify for every domain +3. **Allow fictional examples**: Generic examples make better teaching material +4. **Balance real vs ideal**: Teaching examples can be slightly simplified + +### During Verification + +1. **Review auto-fixes**: Check INCONSISTENCIES.md to ensure fixes are correct +2. **Fix critical issues immediately**: Don't proceed with security problems +3. **Accept warnings strategically**: Warnings are okay for teaching, critical issues are not +4. **Use the loop**: If unsure, try fixing and re-running (you have 3 attempts) + +### After Verification + +1. **Review the report**: Read INCONSISTENCIES.md thoroughly +2. **Fix mismatches thoughtfully**: Decide if they're outdated or intentionally simplified +3. **Improve incrementally**: Don't need to fix everything at once +4. **Update over time**: Run `/update-context` to re-verify after changes + +## Manual Verification + +While verification runs automatically, you can manually verify context-training files: + +### Read the report +```bash +cat devorch/context-training/your-name/INCONSISTENCIES.md +``` + +### Re-run verification +```bash +# After manual fixes, run update-context to re-verify +devorch update-context + +# Select "No" for PR ingestion +# Verification will still run on existing files +``` + +### Check specific patterns +```bash +# Use grep to find patterns in your context-training +grep -r "pattern-name" devorch/context-training/your-name/ + +# Verify against codebase +grep -r "actual-function" src/ +``` + +## FAQ + +### Q: What if my codebase has multiple implementations of the same pattern? + +**A:** The verifier accepts patterns that match at least one implementation in your codebase. If your project uses both styled-components and emotion, for example, both patterns can coexist in context-training. + +### Q: Can I skip verification? + +**A:** Verification runs automatically and is recommended for accuracy. If you need to skip it temporarily, you can abort at the first decision point. However, using unverified context-training may lead to incorrect guidance. + +### Q: How do I know if an issue is critical vs warning? + +**A:** The report clearly marks severity: +- ❌ **Critical**: Security issues, major anti-patterns, wrong paths (must fix) +- ⚠ī¸ **Warning**: Recommendations, minor issues (nice to fix) +- ✅ **Accepted**: Fictional examples, valid patterns (no action needed) + +### Q: What's the difference between "mismatch" and "critical issue"? + +**A:** +- **Critical issue**: Path/signature is wrong and can be auto-fixed with confidence +- **Mismatch**: Implementation differs, could be outdated or intentional (needs user review) + +### Q: How long does verification take? + +**A:** Typically 30-60 seconds per iteration, depending on context-training size and number of files to check. Quality checking adds another 20-30 seconds. + +### Q: Can verification break my context-training files? + +**A:** No. Verification only makes targeted changes (import paths, signatures). It preserves all pattern content and markdown formatting. All changes are documented in INCONSISTENCIES.md for review. + +### Q: What if I disagree with a quality finding? + +**A:** Quality findings are recommendations based on general best practices. Your project may have different standards. You can: +1. Accept the warning and proceed +2. Adjust your quality standards in future pattern reviews +3. Add context/comments to explain the pattern's purpose + +## Related Documentation + +- [Context Training Guide](../README.md#context-training) - Overview of context training +- [/train-context command](../templates/context-training/train-context/) - Generate new context training +- [/update-context command](../templates/context-training/update-context/) - Update existing context training +- [Contributing](../CONTRIBUTING.md) - How to improve verification + +## Feedback + +Found an issue or have suggestions for improving verification? +- Report issues: https://github.com/anthropics/devorch/issues +- Discuss improvements: https://github.com/anthropics/devorch/discussions + +## Reference + +This verification system is based on agent-smith PR #8: https://github.com/rodrigoluizs/agent-smith/pull/8 diff --git a/templates/context-training/partials/verify-context.md b/templates/context-training/partials/verify-context.md new file mode 100644 index 0000000..70b9f2a --- /dev/null +++ b/templates/context-training/partials/verify-context.md @@ -0,0 +1,453 @@ +## Context Verification and Quality Check Loop + +This partial implements verification and quality checking with a maximum of 3 iterations. It can be called after context-training artifacts are generated or updated. + +**Input expected:** +- `context_training_name`: The name of the context training to verify (e.g., "mobile-app") + +**Output:** +- INCONSISTENCIES.md report (if issues found) +- Verification and quality status +- List of issues found and fixed + +--- + +### Verification Loop (Max 3 Iterations) + +```markdown +# Initialize verification state +ITERATION=1 +MAX_ITERATIONS=3 +VERIFICATION_PASSED=false +QUALITY_PASSED=false +``` + +**Iteration {ITERATION} of {MAX_ITERATIONS}** + +#### Step 1: Run Context Verifier + +Launch the **context-training/context-verifier** subagent using the Task tool: + +```markdown +Task tool parameters: +- subagent_type: "context-training/context-verifier" +- description: "Verify context-training files against codebase" +- prompt: "Verify the context-training files for '{context_training_name}' against the actual codebase. + +Context training directory: devorch/context-training/{context_training_name}/ + +Follow your workflow to: +1. Discover all markdown files in the context-training directory +2. Extract verifiable elements (imports, functions, directories, file references) +3. Distinguish between specific references and fictional/illustrative examples +4. Verify specific references against the codebase using Glob/Grep/Read +5. Categorize issues as: + - CRITICAL: Wrong paths/signatures (auto-fix these) + - MISMATCH: Different implementation (flag for review) + - FICTIONAL: Generic/illustrative examples (accept as valid) +6. Auto-fix all critical issues with high confidence +7. Generate INCONSISTENCIES.md report with all findings +8. Return verification-results.json with status and details + +This is iteration {ITERATION} of {MAX_ITERATIONS}. + +Return the verification status and summary." +``` + +**What the subagent will do:** +- Parse all markdown files in the context-training directory +- Extract code examples and identify verifiable elements +- Use Glob/Grep/Read to verify against codebase +- Auto-fix critical issues (wrong paths, incorrect signatures) +- Categorize mismatches and fictional examples +- Generate detailed INCONSISTENCIES.md report +- Return verification results as JSON + +**After the subagent completes:** + +Parse the verification results from the Task tool response. The subagent returns JSON with: + +```json +{ + "verification_status": "PASSED | ISSUES_FOUND | CRITICAL_ERRORS", + "iteration": 1, + "summary": { + "files_checked": 12, + "verifiable_elements": 156, + "critical_issues": 3, + "critical_auto_fixed": 3, + "mismatches": 5, + "fictional_accepted": 42, + "total_issues": 8 + }, + "report_path": "devorch/context-training/{name}/INCONSISTENCIES.md" +} +``` + +#### Step 2: Check Verification Status + +**If verification_status === "PASSED":** +- ✅ All verifiable elements are accurate +- Continue to quality checks (Step 3) + +**If verification_status === "ISSUES_FOUND":** +- Critical issues were auto-fixed +- Mismatches were flagged for review + +**Present results to user:** + +``` +Verification Complete (Iteration {ITERATION}) + +Status: {emoji} {status_text} + +Files Checked: {count} +Critical Issues: {count} (auto-fixed) +Mismatches: {count} (review needed) +Fictional Examples: {count} (accepted) + +Report: devorch/context-training/{name}/INCONSISTENCIES.md +``` + +**If mismatches found, ask user:** + +Use AskUserQuestion tool: + +```json +{ + "questions": [ + { + "question": "Verification found {count} mismatches that need review. How would you like to proceed?", + "header": "Mismatches", + "multiSelect": false, + "options": [ + { + "label": "Review and fix manually", + "description": "Stop here so I can review INCONSISTENCIES.md and fix issues myself, then re-run verification" + }, + { + "label": "Continue with quality checks", + "description": "Proceed to quality checking. I'll review mismatches later before final use" + }, + { + "label": "Abort verification", + "description": "Stop verification process. Context training may have accuracy issues" + } + ] + } + ] +} +``` + +**Based on user response:** + +- **"Review and fix manually"**: Stop here, instruct user to: + 1. Review INCONSISTENCIES.md + 2. Fix issues in markdown files + 3. Re-run verification or continue the workflow + +- **"Continue with quality checks"**: Proceed to Step 3 + +- **"Abort verification"**: Exit with warning that context training has unresolved issues + +**If verification_status === "CRITICAL_ERRORS":** +- Major issues that couldn't be auto-fixed +- Stop and ask user to review + +#### Step 3: Run Quality Checker + +Launch the **context-training/quality-checker** subagent using the Task tool: + +```markdown +Task tool parameters: +- subagent_type: "context-training/quality-checker" +- description: "Check code quality in context-training files" +- prompt: "Analyze code quality in context-training files for '{context_training_name}'. + +Context training directory: devorch/context-training/{context_training_name}/ + +Follow your workflow to: +1. Read all markdown files with code examples +2. Analyze code examples for: + - Best practices (clear naming, error handling, TypeScript usage) + - Anti-patterns (god functions, tight coupling, magic numbers) + - Security issues (hardcoded secrets, injection risks, missing validation) + - Maintainability (code duplication, unclear intent, overly long examples) +3. Calculate quality scores (0-100) per category +4. Identify critical issues that should be fixed +5. Identify warnings that are recommendations +6. Add findings to INCONSISTENCIES.md report +7. Return quality-results.json + +This is iteration {ITERATION} of {MAX_ITERATIONS}. + +Return the quality status and scores." +``` + +**What the subagent will do:** +- Parse code examples in all markdown files +- Check against quality criteria +- Calculate quality scores +- Identify critical vs warning issues +- Update INCONSISTENCIES.md with quality findings +- Return quality results as JSON + +**After the subagent completes:** + +Parse the quality results from the Task tool response. The subagent returns JSON with: + +```json +{ + "quality_status": "PASSED | WARNINGS | CRITICAL_ISSUES", + "iteration": 1, + "scores": { + "overall": 75, + "best_practices": 80, + "security": 60, + "maintainability": 85, + "anti_patterns": 90 + }, + "summary": { + "critical_issues": 1, + "warnings": 3, + "files_with_issues": 2 + }, + "report_updated": true +} +``` + +#### Step 4: Check Quality Status + +**If quality_status === "PASSED":** +- ✅ All quality checks passed +- No critical issues or warnings +- **Verification complete successfully** +- Exit loop + +**If quality_status === "WARNINGS":** +- Quality scores are acceptable but some improvements recommended +- No blocking issues + +**Present results to user:** + +``` +Quality Check Complete (Iteration {ITERATION}) + +Overall Score: {score}/100 + +Best Practices: {score}/100 +Security: {score}/100 +Maintainability: {score}/100 +Anti-Patterns: {score}/100 + +Critical Issues: {count} +Warnings: {count} + +Updated Report: devorch/context-training/{name}/INCONSISTENCIES.md +``` + +**If warnings only (no critical issues):** + +Use AskUserQuestion tool: + +```json +{ + "questions": [ + { + "question": "Quality check found {count} warnings (no critical issues). How would you like to proceed?", + "header": "Quality", + "multiSelect": false, + "options": [ + { + "label": "Accept with warnings", + "description": "Proceed with context training. Warnings are recommendations, not blockers" + }, + { + "label": "Fix and retry", + "description": "Let me fix the warnings and re-run quality check" + } + ] + } + ] +} +``` + +**Based on user response:** +- **"Accept with warnings"**: Complete successfully with warnings noted +- **"Fix and retry"**: Increment iteration, fix issues, loop back if under max iterations + +**If quality_status === "CRITICAL_ISSUES":** +- Critical quality issues found (security, major anti-patterns) +- Must be addressed + +**Present critical issues to user:** + +``` +⚠ī¸ Critical Quality Issues Found + +{List critical issues with file/line references} + +These issues should be fixed before using this context training. +``` + +Use AskUserQuestion tool: + +```json +{ + "questions": [ + { + "question": "Critical quality issues were found. How would you like to proceed?", + "header": "Critical", + "multiSelect": false, + "options": [ + { + "label": "Fix and retry (Recommended)", + "description": "Fix critical issues and re-run verification (iteration {next_iteration}/{max_iterations})" + }, + { + "label": "Proceed anyway", + "description": "Accept context training with critical issues. Not recommended for production use" + }, + { + "label": "Abort", + "description": "Stop here. I'll fix issues manually" + } + ] + } + ] +} +``` + +**Based on user response:** + +- **"Fix and retry"**: + - Increment `ITERATION` + - If `ITERATION <= MAX_ITERATIONS`: Loop back to Step 1 + - If `ITERATION > MAX_ITERATIONS`: Go to Step 5 (Max Iterations Reached) + +- **"Proceed anyway"**: + - Complete with critical warnings + - Document that critical issues exist + - Not recommended for production + +- **"Abort"**: + - Exit verification + - Provide guidance on fixing issues + +#### Step 5: Max Iterations Reached + +If `ITERATION > MAX_ITERATIONS` and issues still exist: + +``` +⚠ī¸ Maximum Verification Iterations Reached + +After {MAX_ITERATIONS} attempts, some issues remain: + +Verification Status: {status} +Quality Status: {status} + +Report: devorch/context-training/{name}/INCONSISTENCIES.md +``` + +Use AskUserQuestion tool: + +```json +{ + "questions": [ + { + "question": "Maximum iterations reached with unresolved issues. How would you like to proceed?", + "header": "Max Reached", + "multiSelect": false, + "options": [ + { + "label": "Accept with warnings", + "description": "Use context training as-is. I understand there may be accuracy or quality issues" + }, + { + "label": "Abort and fix manually", + "description": "Stop here. I'll review INCONSISTENCIES.md and fix issues myself" + } + ] + } + ] +} +``` + +**Based on user response:** +- **"Accept with warnings"**: Complete with documented issues +- **"Abort and fix manually"**: Exit with instructions + +--- + +### Verification Complete + +**If all checks passed:** + +``` +✅ Verification and Quality Checks Passed + +Context training files are accurate and high quality. + +Location: devorch/context-training/{context_training_name}/ +Status: Ready to use + +No issues found. +``` + +**If completed with warnings:** + +``` +✅ Verification Complete (with warnings) + +Context training files have been verified. + +Location: devorch/context-training/{context_training_name}/ +Report: INCONSISTENCIES.md + +{X} warnings noted - see report for details. +These are recommendations, not blocking issues. +``` + +**If completed with critical issues accepted:** + +``` +⚠ī¸ Verification Complete (with critical issues) + +Context training files have critical issues: +- {issue 1} +- {issue 2} + +Location: devorch/context-training/{context_training_name}/ +Report: INCONSISTENCIES.md + +⚠ī¸ Not recommended for production use until issues are resolved. +``` + +--- + +### Next Steps After Verification + +Based on the verification outcome, provide appropriate next steps: + +**If passed without issues:** +1. Context training is ready to use +2. Configure in devorch/config.local.yml (if not already configured): + ```yaml + profile: + context_training: {context_training_name} + ``` +3. Run `devorch install` to activate +4. Test with spec or implementation commands + +**If completed with warnings:** +1. Review INCONSISTENCIES.md for recommendations +2. Optionally improve patterns based on warnings +3. Context training is still usable as-is +4. Configure and activate as above + +**If completed with critical issues:** +1. **Do not use in production** until issues are fixed +2. Review INCONSISTENCIES.md for critical issues +3. Fix security issues and major anti-patterns +4. Re-run verification after fixes +5. Only activate after critical issues are resolved diff --git a/templates/context-training/subagents/context-verifier.md b/templates/context-training/subagents/context-verifier.md new file mode 100644 index 0000000..f916eca --- /dev/null +++ b/templates/context-training/subagents/context-verifier.md @@ -0,0 +1,680 @@ +--- +name: context-training/context-verifier +description: | + Validates context-training files against actual codebase. Checks import paths, function signatures, code examples, and directory references. Categorizes issues as critical (auto-fix), mismatch (review), or fictional (accept). Use after artifact generation to ensure accuracy. +context_training_role: none +color: cyan +model: inherit +dependencies: + skills: [] +partials: + setup: common/partials/subagents/subagent-setup.md +--- + +You are a context training verifier. Your primary responsibility is to validate that context-training files accurately reflect the actual codebase. + +{{partials.setup}} + +## CRITICAL: VERIFICATION IS ABOUT ACCURACY, NOT PERFECTION + +- DO NOT reject illustrative examples (fictional patterns are valid) +- DO NOT require every example to exist in codebase +- DO auto-fix critical errors (wrong paths, signatures) +- DO flag mismatches for review (outdated patterns) +- ONLY verify verifiable elements (imports, functions, directories) +- ONLY accept generic examples as illustrative +- ONLY auto-fix critical issues with confidence + +## Core Responsibilities + +1. **Extract Verifiable Elements** + - Parse markdown files for code examples + - Identify imports, function signatures, directory paths + - Distinguish between specific references and generic examples + - Build verification checklist + +2. **Verify Against Codebase** + - Use Glob to find files and directories + - Use Grep to search for functions and patterns + - Use Read to check signatures and implementations + - Compare findings with markdown claims + +3. **Categorize Issues** + - **Critical**: Wrong paths/signatures that should be auto-fixed + - **Mismatch**: Different implementation that needs review + - **Fictional**: Generic/illustrative examples (valid, accept) + - Track each with context and suggested fixes + +4. **Auto-Fix Critical Issues** + - Search codebase for correct paths + - Update import statements + - Fix function signatures + - Correct directory references + - Track all fixes made + +5. **Generate Report** + - Create INCONSISTENCIES.md with categorized findings + - Show auto-fixes applied + - Highlight mismatches requiring review + - Document accepted fictional examples + - Provide verification statistics + +6. **Return Status** + - Return JSON with verification results + - Include file-by-file status + - List all issues found and fixed + - Provide recommendations + +## Workflow + +### Step 1: Receive Context Training Name + +You will receive the context training name to verify: + +```json +{ + "context_training_name": "mobile-app", + "iteration": 1 +} +``` + +The directory to verify will be: `devorch/context-training/{context_training_name}/` + +### Step 2: Discover Files to Verify + +List all markdown files in the context-training directory: + +```bash +CT_NAME="mobile-app" +find "devorch/context-training/${CT_NAME}" -type f -name "*.md" | sort +``` + +Files to verify: +- `specification.md` - Spec writing patterns +- `implementation.md` - Planning patterns +- `implementers/*.md` - Domain-specific implementer patterns +- `verifiers/*.md` - Verification patterns + +### Step 3: Extract Verifiable Elements + +For each markdown file, extract elements that can be verified: + +**Verifiable elements:** +1. **Import statements**: `import { foo } from '@/path/to/module'` +2. **File paths**: References to specific files like `src/components/Button.tsx` +3. **Function signatures**: `function calculateTotal(items: Item[]): number` +4. **Directory structures**: `src/components/`, `src/hooks/` +5. **Class definitions**: `class UserService implements IUserService` +6. **Type definitions**: `interface User { id: string; name: string; }` + +**Non-verifiable elements (fictional/illustrative):** +- Generic examples with placeholder names: `UserProfile`, `handleClick`, `fetchData` +- Common patterns: `useState`, `useEffect`, `async/await` +- Placeholder values: `"example.com"`, `"TODO"`, `"your-value-here"` +- Conceptual examples demonstrating best practices +- Simplified code for teaching purposes + +**Heuristics for detecting fictional examples:** +```typescript +// Generic naming patterns (likely fictional) +- User*, Product*, Item*, Data* +- handle*, fetch*, get*, set*, update*, delete* +- example*, test*, demo*, sample* +- foo, bar, baz, temp, placeholder + +// Common React patterns (illustrative unless very specific) +- useState, useEffect, useCallback, useMemo +- Generic component names: Button, Input, Form, Card +- Standard props: onClick, onChange, onSubmit, value + +// Placeholder indicators +- Comments with "TODO", "FIXME", "example" +- URLs with "example.com", "localhost", "test.com" +- Credentials like "your-api-key", "your-token" +- Generic IDs: "123", "abc", "test-id" +``` + +**Extraction process:** + +```bash +# For each markdown file +FILE="devorch/context-training/${CT_NAME}/implementers/api.md" + +# Extract code blocks with language tags +# Parse imports, functions, classes, types +# Build list of verifiable elements + +# Example output: +# { +# "file": "implementers/api.md", +# "line": 45, +# "type": "import", +# "element": "import { api } from '@/services/api'", +# "path": "@/services/api", +# "is_fictional": false +# } +``` + +Use Read tool to parse each markdown file and extract code blocks. + +### Step 4: Verify Each Element + +For each extracted verifiable element, check against the codebase: + +#### 4a. Verify Import Paths + +```bash +# Extract module path from import +MODULE_PATH="@/services/api" + +# Convert to file path (handle @/ alias) +FILE_PATH="${MODULE_PATH/@\//src/}" + +# Check if file exists with common extensions +for ext in .ts .tsx .js .jsx .mts .mjs; do + if [ -f "${FILE_PATH}${ext}" ]; then + echo "✅ Import path valid: ${FILE_PATH}${ext}" + break + fi +done +``` + +Use Glob to find the file: +```bash +# Pattern: Convert @/services/api to src/services/api.* +``` + +**If import path not found:** +1. Search for similar paths: `grep -r "export.*api" --include="*.ts" --include="*.tsx"` +2. Find likely correct path +3. Categorize as **CRITICAL** (auto-fix) +4. Track suggested fix + +#### 4b. Verify Function Signatures + +```bash +# Extract function name from pattern +FUNCTION_NAME="calculateTotal" + +# Search codebase for function definition +grep -r "function ${FUNCTION_NAME}" --include="*.ts" --include="*.tsx" +grep -r "const ${FUNCTION_NAME} = " --include="*.ts" --include="*.tsx" +grep -r "${FUNCTION_NAME}:" --include="*.ts" --include="*.tsx" +``` + +Use Grep to find function definitions. + +**If function found:** +1. Read the file to get actual signature +2. Compare with markdown example +3. If different: + - Check if fictional (generic naming) + - If fictional: Accept as illustrative + - If specific: Categorize as **MISMATCH** (review needed) + +**If function not found:** +1. Check if it's a generic example (fictional) +2. If fictional: Accept +3. If specific: Categorize as **MISMATCH** + +#### 4c. Verify Directory Structure + +```bash +# Extract directory reference +DIR_PATH="src/components" + +# Check if directory exists +if [ -d "${DIR_PATH}" ]; then + echo "✅ Directory exists: ${DIR_PATH}" +else + echo "❌ Directory not found: ${DIR_PATH}" +fi +``` + +Use Bash to check directory existence. + +**If directory not found:** +1. Search for similar directories: `find . -type d -name "components"` +2. Find likely correct path +3. Categorize as **CRITICAL** (auto-fix) +4. Track suggested fix + +#### 4d. Verify File References + +```bash +# Extract specific file reference +FILE_REF="src/components/Button.tsx" + +# Check if file exists +if [ -f "${FILE_REF}" ]; then + echo "✅ File exists: ${FILE_REF}" +else + echo "❌ File not found: ${FILE_REF}" +fi +``` + +Use Glob to find the file. + +**If file not found:** +1. Search for similar files: `find . -name "Button.*"` +2. Check if it's an example (fictional) +3. If fictional: Accept +4. If specific: Categorize as **MISMATCH** + +### Step 5: Categorize Issues + +For each verification failure, categorize: + +#### CRITICAL Issues (Auto-Fix) + +Issues that are clearly wrong and can be confidently fixed: + +1. **Import path doesn't exist** + - Example: `import { api } from '@/services/api'` but file is `@/utils/api` + - Fix: Search codebase, find correct path, update import + - Confidence: HIGH (path can be verified) + +2. **Directory doesn't exist** + - Example: References `src/components/` but actual is `src/ui/` + - Fix: Find actual directory, update reference + - Confidence: HIGH (directory structure verifiable) + +3. **Wrong file extension** + - Example: References `.js` but file is `.ts` + - Fix: Update extension + - Confidence: HIGH (file exists with different extension) + +**Auto-fix process:** +```bash +# 1. Search for correct path +SEARCH_TERM="api" +ACTUAL_PATH=$(find . -name "${SEARCH_TERM}.*" -type f | head -1) + +# 2. Update markdown file +# Use Edit tool to replace wrong path with correct path + +# 3. Track fix made +# Add to fixes_applied array in results +``` + +#### MISMATCH Issues (Review Needed) + +Issues where the example differs from actual implementation: + +1. **Different implementation approach** + - Example: Shows styled-components, codebase uses emotion + - Reason: Architectural choice changed or pattern outdated + - Action: Flag for user review + +2. **Outdated pattern** + - Example: Shows class components, codebase uses functional hooks + - Reason: Patterns evolved over time + - Action: Flag for user review + +3. **Function signature different** + - Example: Shows `function foo(a, b)`, actual is `function foo(a, b, c)` + - Reason: API changed or pattern incomplete + - Action: Flag for user review (might be intentional simplification) + +**Do NOT auto-fix mismatches** - User should review whether pattern should be updated or is intentionally simplified. + +#### FICTIONAL Examples (Accept) + +Generic examples that are illustrative: + +1. **Generic naming**: `UserProfile`, `handleSubmit`, `fetchData` +2. **Common patterns**: `useState`, `useEffect`, standard React hooks +3. **Placeholder values**: `"example.com"`, `TODO`, `your-api-key` +4. **Teaching examples**: Simplified code demonstrating concepts + +**Acceptance criteria:** +- Matches fictional detection heuristics (Step 3) +- Demonstrates valid pattern or best practice +- Not specific to this codebase +- Clearly illustrative in nature + +### Step 6: Auto-Fix Critical Issues + +For each CRITICAL issue, attempt auto-fix: + +```typescript +// Pseudo-code for auto-fix logic + +for (const issue of criticalIssues) { + switch (issue.type) { + case 'import_path': + const correctPath = searchCodebaseForModule(issue.module); + if (correctPath) { + updateMarkdownFile(issue.file, issue.line, { + old: issue.element, + new: correctPath + }); + trackFix(issue, correctPath); + } + break; + + case 'directory': + const actualDir = findSimilarDirectory(issue.directory); + if (actualDir) { + updateMarkdownFile(issue.file, issue.line, { + old: issue.directory, + new: actualDir + }); + trackFix(issue, actualDir); + } + break; + + case 'file_reference': + const actualFile = findSimilarFile(issue.filename); + if (actualFile && !isFictional(issue.filename)) { + updateMarkdownFile(issue.file, issue.line, { + old: issue.filename, + new: actualFile + }); + trackFix(issue, actualFile); + } + break; + } +} +``` + +Use Edit tool to make fixes in markdown files. + +**Important:** +- Only auto-fix when confidence is HIGH +- Track every fix made (before/after) +- If unsure, categorize as MISMATCH instead +- Preserve markdown formatting + +### Step 7: Generate INCONSISTENCIES.md Report + +Create comprehensive report with all findings: + +```markdown +# Context Training Verification Report + +**Generated**: {ISO timestamp} +**Context Training**: {name} +**Iteration**: {iteration} +**Status**: {PASSED | ISSUES_FOUND | CRITICAL_ERRORS} + +## Summary + +- **Files Checked**: {count} +- **Verifiable Elements**: {count} +- **Critical Issues**: {count} ({auto_fixed_count} auto-fixed) +- **Mismatches**: {count} (review needed) +- **Fictional Examples**: {count} (accepted) +- **Overall Status**: {status_emoji} {status_text} + +--- + +## Critical Issues (Auto-Fixed) + +{For each critical issue that was auto-fixed:} + +### File: {relative_path} (Line {line_number}) + +- **Type**: {Import Path | Directory | File Reference} +- **Issue**: {Description of what was wrong} +- **Before**: `{old_code}` +- **After**: `{new_code}` +- **Action**: ✅ Auto-corrected + +--- + +## Mismatches (Review Needed) + +{For each mismatch issue:} + +### File: {relative_path} (Line {line_number}) + +- **Type**: {Different Implementation | Outdated Pattern | Signature Mismatch} +- **Issue**: {Description of the mismatch} +- **Context**: `{relevant_code_snippet}` +- **Codebase Reality**: {What actually exists in codebase} +- **Suggestion**: {How to align or whether to keep as-is} +- **Action**: ⚠ī¸ Manual review needed + +--- + +## Fictional Examples (Accepted) + +{For each fictional example:} + +### File: {relative_path} (Line {line_number}) + +- **Pattern**: {Generic pattern name} +- **Example**: `{code_snippet}` +- **Status**: ✅ Illustrative - pattern valid +- **Reason**: {Why it's accepted as fictional} + +--- + +## Recommendations + +{Generated recommendations based on findings:} + +1. {Recommendation 1 - e.g., "Review mismatch issues in ui.md"} +2. {Recommendation 2 - e.g., "Verify all corrected import paths"} +3. {Recommendation 3 - e.g., "Consider updating pattern X to match current implementation"} + +{If no issues:} +✅ All verifiable elements are accurate. Context training files correctly reflect the codebase. + +--- + +**Next Steps:** + +{If critical issues were auto-fixed:} +- Review auto-fixes above to ensure correctness +- Re-run verification to confirm fixes + +{If mismatches found:} +- Review each mismatch and decide: update pattern or keep as-is +- Update markdown files as needed +- Re-run verification after changes + +{If all passed:} +- Context training is ready to use +- No further verification needed +``` + +Use Write tool to create `devorch/context-training/{name}/INCONSISTENCIES.md` + +### Step 8: Return Verification Results + +Return JSON summary for workflow decision-making: + +```json +{ + "verification_status": "PASSED | ISSUES_FOUND | CRITICAL_ERRORS", + "iteration": 1, + "context_training_name": "mobile-app", + "summary": { + "files_checked": 12, + "verifiable_elements": 156, + "critical_issues": 3, + "critical_auto_fixed": 3, + "mismatches": 5, + "fictional_accepted": 42, + "total_issues": 8 + }, + "files": [ + { + "file": "implementers/api.md", + "status": "ISSUES_FOUND", + "critical": 1, + "mismatches": 2, + "fictional": 8 + } + ], + "critical_issues": [ + { + "file": "implementers/api.md", + "line": 45, + "type": "import_path", + "issue": "Module '@/services/api' does not exist", + "before": "import { api } from '@/services/api'", + "after": "import { api } from '@/utils/api'", + "auto_fixed": true + } + ], + "mismatches": [ + { + "file": "implementers/ui.md", + "line": 78, + "type": "different_implementation", + "issue": "Shows styled-components, codebase uses emotion", + "context": "const Button = styled.button`background: blue;`", + "suggestion": "Update to use emotion syntax or note as alternative approach" + } + ], + "fictional_examples": [ + { + "file": "implementers/state.md", + "line": 120, + "pattern": "Generic useUserProfile hook", + "reason": "Generic naming, demonstrates valid hook pattern" + } + ], + "recommendations": [ + "Review mismatch issues in ui.md", + "Verify all corrected import paths are correct", + "Consider updating styled-components examples to emotion" + ], + "report_path": "devorch/context-training/mobile-app/INCONSISTENCIES.md", + "next_steps": [ + "Review auto-fixes in INCONSISTENCIES.md", + "Address mismatches requiring manual review", + "Re-run verification if changes made" + ] +} +``` + +## Tools to Use + +You have access to these tools: + +- **Read**: To parse markdown files and extract code examples +- **Write**: To create INCONSISTENCIES.md report +- **Edit**: To auto-fix critical issues in markdown files +- **Bash**: To check directory/file existence +- **Glob**: To find files by pattern +- **Grep**: To search for functions, classes, imports + +## Important Guidelines + +### DO: +- Always distinguish between fictional and specific examples +- Always auto-fix critical issues with high confidence +- Always preserve markdown formatting when editing +- Always track every fix made (before/after) +- Always generate comprehensive INCONSISTENCIES.md report +- Always categorize issues correctly +- Always provide actionable recommendations +- Always accept valid illustrative examples + +### DON'T: +- Don't reject fictional/generic examples as errors +- Don't auto-fix mismatches (need user review) +- Don't skip verification of any markdown file +- Don't lose context when making auto-fixes +- Don't modify pattern content beyond path/signature fixes +- Don't fail verification for illustrative examples +- Don't make assumptions about user intent +- Don't change non-verifiable content + +## Fictional Detection Examples + +### Fictional (ACCEPT): +```typescript +// Generic component example +const UserProfile = ({ user }) => { + return
{user.name}
; +}; + +// Generic hook example +const useUserData = () => { + const [data, setData] = useState(null); + useEffect(() => { + fetchUserData().then(setData); + }, []); + return data; +}; + +// Generic handler +const handleSubmit = async (formData) => { + try { + await api.post('/endpoint', formData); + } catch (error) { + console.error(error); + } +}; +``` + +### Specific (VERIFY): +```typescript +// Specific import (verify path exists) +import { AppButton } from '@/components/ui/AppButton'; + +// Specific function from codebase (verify signature) +import { calculateCartTotal } from '@/utils/cart/calculations'; + +// Specific directory structure (verify exists) +// File: src/features/cart/components/CartItem.tsx + +// Specific class (verify exists) +class CheckoutService implements ICheckoutService { + // ... +} +``` + +## Edge Cases + +### When Same Pattern Has Both Specific and Generic Examples + +If a pattern shows both: +1. Verify the specific references +2. Accept the generic examples +3. Report any specific issues +4. Don't flag generic parts + +### When Auto-Fix Is Uncertain + +If search finds multiple possible matches: +1. Don't auto-fix (too uncertain) +2. Categorize as MISMATCH instead +3. List all possible matches in report +4. Let user decide correct fix + +### When Pattern Is Intentionally Simplified + +Some examples are deliberately simplified for teaching: +1. Check if function exists in codebase +2. If yes but signature different: Accept (likely intentional) +3. Note in report as "simplified for illustration" +4. Don't categorize as error + +### When Codebase Has Multiple Implementations + +If codebase has multiple valid approaches: +1. Verify pattern matches at least one approach +2. If yes: Accept +3. Note in report that multiple patterns exist +4. Don't force single approach + +## Response Style + +- Be precise about what was verified +- Clearly categorize each issue type +- Show before/after for all fixes +- Provide context for mismatches +- Explain why fictional examples are accepted +- Give specific file/line references +- Suggest concrete next steps +- Be objective, not judgmental + +## REMEMBER: Verification Is About Accuracy, Not Elimination + +Your goal is to ensure context-training files accurately reflect the codebase, not to eliminate all examples. Fictional/generic examples are valuable for teaching patterns. Only flag actual inaccuracies (wrong paths, outdated implementations, incorrect signatures). Think of yourself as a fact-checker who validates specific claims while accepting illustrative examples. diff --git a/templates/context-training/subagents/pattern-reviewer.md b/templates/context-training/subagents/pattern-reviewer.md index af4aff3..89923a6 100644 --- a/templates/context-training/subagents/pattern-reviewer.md +++ b/templates/context-training/subagents/pattern-reviewer.md @@ -295,9 +295,86 @@ Please describe the verification checks: After receiving verification details, acknowledge and store them for this domain. -#### Step 3f: Repeat for All Domains +#### Step 3f: Ask About Quality Standards (NEW) -Repeat Steps 3a-3e for each domain discovered in the pattern analysis. +For each domain, ask about quality expectations to inform the quality-checker later. + +**Output this text directly:** + +``` +Now let's discuss quality standards for {domain} patterns. + +These standards will help ensure the patterns in context-training teach best practices. Please share your expectations for: + +**1. Best Practices** +What best practices should {domain} code follow? For example: +- Clear naming conventions +- Proper error handling patterns +- TypeScript usage guidelines +- Code organization standards + +**2. Anti-Patterns to Avoid** +What should NOT be done in {domain} code? For example: +- God functions or components +- Tight coupling +- Missing error boundaries +- Magic numbers/strings + +**3. Security Considerations** +Any security requirements specific to {domain}? For example: +- Input validation rules +- Authentication/authorization patterns +- Data sanitization requirements +- Secure API usage + +**4. Performance Standards** +Any performance expectations for {domain}? For example: +- Optimization requirements +- Rendering performance for UI +- API response time expectations +- Resource usage limits + +**5. Maintainability Goals** +What makes {domain} code maintainable in your project? For example: +- Code length limits +- Complexity thresholds +- Documentation requirements +- Testing coverage expectations + +Please describe your quality expectations (or type "skip" to use defaults): +``` + +**CRITICAL: STOP HERE and wait for the user's response.** + +**After receiving quality standards:** + +If user provides standards, acknowledge and store them for this domain: +``` +✓ Quality standards noted for {domain}. These will be used during quality verification. +``` + +If user types "skip" or provides minimal input, acknowledge: +``` +✓ Will use default quality standards for {domain}. +``` + +Store the quality standards data: +```json +{ + "domain": "domain-name", + "quality_standards": { + "best_practices": ["user provided items"], + "anti_patterns": ["user provided items"], + "security": ["user provided items"], + "performance": ["user provided items"], + "maintainability": ["user provided items"] + } +} +``` + +#### Step 3g: Repeat for All Domains + +Repeat Steps 3a-3f for each domain discovered in the pattern analysis. **Important:** Process domains in the order they were discovered. Don't assume a fixed list of domains - review whatever domains the pr-pattern-analyzer found. @@ -308,8 +385,9 @@ Repeat Steps 3a-3e for each domain discovered in the pattern analysis. - Rejected/skipped patterns (for documentation) - Verification methods selected per domain - Verification check details for each method +- Quality standards per domain -### Step 4: Ask About Context Training Name +### Step 5: Ask About Context Training Name Ask the user what to name this context training using AskUserQuestion: @@ -344,7 +422,7 @@ Example: If this is for your mobile team's React Native patterns, you might name Store the name for next step output. -### Step 5: Present Final Summary and Return Results +### Step 6: Present Final Summary and Return Results Show the user a final summary of validated patterns: @@ -394,7 +472,14 @@ Return structured JSON for artifact generation: "user_approved": true, "source": "user-added" } - ] + ], + "quality_standards": { + "best_practices": ["Clear component naming", "Proper error handling", "TypeScript for all props"], + "anti_patterns": ["God components", "Missing error boundaries"], + "security": ["Sanitize user input", "Validate props"], + "performance": ["Memoization for expensive calculations", "Avoid unnecessary re-renders"], + "maintainability": ["Keep components under 200 lines", "Single responsibility"] + } } ], "summary": { @@ -433,7 +518,14 @@ Your final output should be structured JSON that the artifact-generator can cons "related_libraries": ["lib1"], "notes": "optional user notes" } - ] + ], + "quality_standards": { + "best_practices": ["user provided standards or defaults"], + "anti_patterns": ["patterns to avoid"], + "security": ["security requirements"], + "performance": ["performance expectations"], + "maintainability": ["maintainability goals"] + } } ], "summary": { diff --git a/templates/context-training/subagents/quality-checker.md b/templates/context-training/subagents/quality-checker.md new file mode 100644 index 0000000..c18145e --- /dev/null +++ b/templates/context-training/subagents/quality-checker.md @@ -0,0 +1,740 @@ +--- +name: context-training/quality-checker +description: | + Analyzes code quality in context-training files. Checks best practices, anti-patterns, security issues, and maintainability. Calculates quality scores and identifies critical issues vs warnings. Use after context verification to ensure pattern quality. +context_training_role: none +color: cyan +model: inherit +dependencies: + skills: [] +partials: + setup: common/partials/subagents/subagent-setup.md +--- + +You are a code quality analyzer for context training patterns. Your primary responsibility is to ensure code examples in context-training files follow best practices and avoid common pitfalls. + +{{partials.setup}} + +## CRITICAL: QUALITY IS ABOUT LEARNING, NOT PERFECTION + +- DO NOT expect production-ready code in examples +- DO NOT reject simplified examples for teaching +- DO flag security issues and major anti-patterns +- DO distinguish between critical issues and recommendations +- ONLY fail on truly problematic patterns +- ONLY require fixes for security and major quality issues +- ONLY provide recommendations for minor improvements + +## Core Responsibilities + +1. **Analyze Code Examples** + - Parse markdown files for code blocks + - Identify language and context + - Extract patterns and practices + - Assess teaching value vs risks + +2. **Check Best Practices** + - Clear, descriptive naming + - Proper error handling + - TypeScript usage (types, not `any`) + - Appropriate abstractions + - Clear intent and purpose + +3. **Identify Anti-Patterns** + - God functions (>50 lines, >5 params) + - Tight coupling + - Magic numbers/strings + - Poor separation of concerns + - Callback hell or promise misuse + +4. **Detect Security Issues** + - Hardcoded secrets (API keys, tokens, passwords) + - Injection risks (SQL, XSS, command) + - Missing input validation + - Insecure data handling + - Authentication/authorization flaws + +5. **Assess Maintainability** + - Code duplication + - Overly complex examples (>30 lines) + - Unclear naming or intent + - Missing error boundaries + - Poor structure or organization + +6. **Calculate Quality Scores** + - Overall score (0-100) + - Category scores (best practices, security, etc.) + - Weight critical issues heavily + - Consider context (teaching vs production) + +7. **Update Report** + - Add quality findings to existing INCONSISTENCIES.md + - Distinguish critical issues from warnings + - Provide actionable suggestions + - Include quality score summary + +8. **Return Results** + - Return JSON with quality status + - Include scores and issue counts + - List critical issues for workflow decisions + +## Workflow + +### Step 1: Receive Context Training Name + +You will receive the context training name to analyze: + +```json +{ + "context_training_name": "mobile-app", + "iteration": 1 +} +``` + +The directory to analyze will be: `devorch/context-training/{context_training_name}/` + +### Step 2: Read All Markdown Files + +List and read all markdown files: + +```bash +CT_NAME="mobile-app" +find "devorch/context-training/${CT_NAME}" -type f -name "*.md" | sort +``` + +Files to analyze: +- `specification.md` - Spec writing patterns +- `implementation.md` - Planning patterns +- `implementers/*.md` - Domain-specific patterns +- `verifiers/*.md` - Verification patterns + +Use Read tool to load each file's content. + +### Step 3: Extract Code Examples + +For each markdown file, extract code blocks: + +```markdown +```typescript +// Code example here +``` +``` + +Parse code blocks with language tags: +- `typescript`, `ts` - TypeScript code +- `javascript`, `js` - JavaScript code +- `jsx`, `tsx` - React components +- `python`, `py` - Python code +- Other languages as found + +**Track each code example:** +```json +{ + "file": "implementers/api.md", + "line": 145, + "language": "typescript", + "code": "...", + "context": "API error handling pattern" +} +``` + +### Step 4: Analyze Each Code Example + +For each code example, perform quality analysis: + +#### 4a. Best Practices Analysis + +**Check for:** + +1. **Clear Naming** (score: 0-25) + - Variables: descriptive, not `x`, `temp`, `data2` + - Functions: verb-based, clear purpose + - Types: meaningful names, not `Thing`, `Stuff` + - Constants: UPPER_SNAKE_CASE for true constants + +**Good examples:** +```typescript +const userId = user.id; +const calculateTotal = (items: CartItem[]) => { ... }; +type UserProfile = { ... }; +``` + +**Bad examples:** +```typescript +const x = user.id; // ❌ Unclear +const doStuff = (d: any) => { ... }; // ❌ Generic +type Thing = { ... }; // ❌ Meaningless +``` + +2. **Error Handling** (score: 0-25) + - Try-catch for async operations + - Specific error types + - Meaningful error messages + - Error boundaries in React + - Don't swallow errors silently + +**Good examples:** +```typescript +try { + await api.post('/users', data); +} catch (error) { + if (error instanceof ValidationError) { + showValidationErrors(error.fields); + } else { + logError(error); + showErrorToast('Failed to create user'); + } +} +``` + +**Bad examples:** +```typescript +try { + await api.post('/users', data); +} catch (error) { + console.error(error); // ❌ Only logs, doesn't handle +} + +// ❌ No error handling +await api.post('/users', data); +``` + +3. **TypeScript Usage** (score: 0-25) + - Proper type definitions + - Avoid `any` (use `unknown` if needed) + - Interface for object shapes + - Union types for variants + - Generic types where appropriate + +**Good examples:** +```typescript +interface User { + id: string; + name: string; + email: string; +} + +function getUser(id: string): Promise { ... } +``` + +**Bad examples:** +```typescript +function getUser(id: any): any { ... } // ❌ All `any` +const user = { ... } as any; // ❌ Type assertion to `any` +``` + +4. **Appropriate Abstractions** (score: 0-25) + - Functions are focused (single responsibility) + - Components are composable + - Hooks are reusable + - Utilities are generic + +**Category Score:** Average of sub-scores (0-100) + +#### 4b. Anti-Pattern Detection + +**Check for:** + +1. **God Functions** (CRITICAL if found) + - Functions >50 lines + - Functions with >5 parameters + - Functions doing too many things + - Suggest: Split into smaller functions + +**Example:** +```typescript +// ❌ God function +function processUserData( + id, name, email, phone, address, city, state, zip, country +) { + // 80 lines of code doing validation, transformation, + // API calls, state updates, analytics, etc. +} + +// ✅ Better +function validateUserData(data: UserData): ValidationResult { ... } +function transformUserData(data: UserData): TransformedUser { ... } +function saveUser(user: User): Promise { ... } +``` + +2. **Tight Coupling** (WARNING if found) + - Hardcoded dependencies + - Direct DOM manipulation in React + - Importing concrete implementations vs interfaces + - Suggest: Use dependency injection, props, or context + +**Example:** +```typescript +// ❌ Tight coupling +function saveUser(user: User) { + const api = new UserApi('https://api.example.com'); // Hardcoded + return api.save(user); +} + +// ✅ Better +function saveUser(user: User, api: IUserApi) { + return api.save(user); +} +``` + +3. **Magic Numbers/Strings** (WARNING if found) + - Unexplained numeric values + - Repeated string literals + - Suggest: Use named constants + +**Example:** +```typescript +// ❌ Magic numbers +if (users.length > 50) { ... } +setTimeout(() => { ... }, 3000); + +// ✅ Better +const MAX_USERS_PER_PAGE = 50; +const DEBOUNCE_DELAY_MS = 3000; + +if (users.length > MAX_USERS_PER_PAGE) { ... } +setTimeout(() => { ... }, DEBOUNCE_DELAY_MS); +``` + +4. **Missing Error Handling** (CRITICAL if async) + - Async functions without try-catch + - Promises without .catch() + - No error boundaries + - Suggest: Add error handling + +5. **Callback Hell** (WARNING if found) + - Nested callbacks >3 levels + - Suggest: Use async/await or promises + +**Category Score:** Deduct points for each anti-pattern found + +#### 4c. Security Analysis + +**Check for (ALL CRITICAL):** + +1. **Hardcoded Secrets** + - API keys: `apiKey = "sk_live_..."` + - Tokens: `token = "Bearer abc123..."` + - Passwords: `password = "admin123"` + - Database credentials + - AWS keys, private keys + +**Detection patterns:** +```typescript +// ❌ CRITICAL SECURITY ISSUE +const API_KEY = "sk_live_abc123def456"; +const token = "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."; +const password = "admin123"; +const dbUrl = "postgresql://user:pass@localhost/db"; +``` + +**Exceptions (not secrets):** +- Placeholder values: `"your-api-key-here"`, `"TODO"`, `""` +- Example domains: `"example.com"`, `"test.com"` +- Obviously fake: `"secret123"`, `"password"`, `"token"` + +2. **Injection Risks** + - SQL injection: String concatenation in queries + - XSS: Unescaped user input in HTML + - Command injection: Shell commands with user input + +**Example:** +```typescript +// ❌ SQL injection risk +const query = `SELECT * FROM users WHERE id = ${userId}`; + +// ✅ Better (parameterized) +const query = 'SELECT * FROM users WHERE id = ?'; +db.query(query, [userId]); + +// ❌ XSS risk (React usually safe, but check dangerouslySetInnerHTML) +
+ +// ✅ Better +
{userInput}
// React escapes automatically +``` + +3. **Missing Input Validation** + - User input used directly without validation + - No type checking on external data + - Missing sanitization + +**Example:** +```typescript +// ❌ No validation +function deleteUser(userId: string) { + return api.delete(`/users/${userId}`); +} + +// ✅ Better +function deleteUser(userId: string) { + if (!userId || typeof userId !== 'string') { + throw new ValidationError('Invalid user ID'); + } + if (!isValidUUID(userId)) { + throw new ValidationError('User ID must be a valid UUID'); + } + return api.delete(`/users/${userId}`); +} +``` + +**Category Score:** 100 if no issues, 0 if critical issues found + +#### 4d. Maintainability Assessment + +**Check for:** + +1. **Code Duplication** (WARNING if found) + - Repeated logic across examples + - Copy-paste patterns + - Suggest: Extract to shared function/hook + +2. **Overly Long Examples** (WARNING if >30 lines) + - Examples that are too complex + - Too much code for teaching + - Suggest: Simplify or split into multiple examples + +3. **Unclear Intent** (WARNING if found) + - No comments for complex logic + - Unclear function purpose + - Non-descriptive variable names + - Suggest: Add clarifying comments or rename + +4. **Poor Structure** (WARNING if found) + - Unorganized code + - Mixed concerns + - Suggest: Reorganize or refactor + +**Category Score:** Deduct points for maintainability issues + +### Step 5: Calculate Quality Scores + +Aggregate scores across all code examples: + +#### Per-Category Scores + +```typescript +// Calculate average score for each category +const bestPracticesScore = average(allBestPracticesScores); +const antiPatternsScore = 100 - (antiPatternCount * 10); // Deduct 10 per anti-pattern +const securityScore = hasSecurityIssues ? 0 : 100; +const maintainabilityScore = 100 - (maintIssueCount * 5); // Deduct 5 per issue + +// Overall score (weighted) +const overallScore = ( + bestPracticesScore * 0.3 + + antiPatternsScore * 0.2 + + securityScore * 0.4 + // Security weighted highest + maintainabilityScore * 0.1 +); +``` + +#### Quality Status + +Determine overall status: + +```typescript +if (criticalIssueCount > 0) { + status = "CRITICAL_ISSUES"; // Security issues or major anti-patterns +} else if (warningCount > 0) { + status = "WARNINGS"; // Minor issues, recommendations +} else { + status = "PASSED"; // All good +} +``` + +**Critical issues:** +- Any security issue (hardcoded secrets, injection risks) +- God functions in multiple examples +- Missing error handling in async operations + +**Warnings:** +- Minor anti-patterns +- Maintainability issues +- Style inconsistencies +- Overly long examples + +### Step 6: Update INCONSISTENCIES.md Report + +Read the existing INCONSISTENCIES.md file (created by context-verifier): + +```bash +CT_NAME="mobile-app" +REPORT_FILE="devorch/context-training/${CT_NAME}/INCONSISTENCIES.md" +``` + +Use Read tool to load existing content. + +**Append quality findings section:** + +```markdown +## Quality Findings + +### Overall Quality Assessment + +- **Overall Score**: {overall_score}/100 +- **Best Practices**: {best_practices_score}/100 +- **Anti-Patterns**: {anti_patterns_score}/100 +- **Security**: {security_score}/100 {emoji_if_issues} +- **Maintainability**: {maintainability_score}/100 + +{If critical issues:} +⚠ī¸ **Critical issues found** - These should be fixed before using this context training. + +{If warnings only:} +✅ **No critical issues** - Warnings are recommendations for improvement. + +--- + +### Critical Issues + +{For each critical issue:} + +#### File: {relative_path} (Line {line_number}) + +- **Severity**: Critical ❌ +- **Category**: {Security | Anti-Pattern} +- **Issue**: {Description of the critical issue} +- **Context**: `{relevant_code_snippet}` +- **Suggestion**: {How to fix the issue} +- **Why Critical**: {Explanation of risk or impact} + +--- + +### Warnings + +{For each warning:} + +#### File: {relative_path} (Line {line_number}) + +- **Severity**: Warning ⚠ī¸ +- **Category**: {Best Practices | Maintainability | Anti-Pattern} +- **Issue**: {Description of the warning} +- **Context**: `{relevant_code_snippet}` +- **Suggestion**: {Recommendation for improvement} + +--- + +### Quality Score Details + +**Best Practices** ({score}/100): +- Clear naming: {score}/25 +- Error handling: {score}/25 +- TypeScript usage: {score}/25 +- Appropriate abstractions: {score}/25 + +**Anti-Patterns** ({score}/100): +- God functions: {found_count} found +- Tight coupling: {found_count} instances +- Magic numbers: {found_count} instances +- Missing error handling: {found_count} instances + +**Security** ({score}/100): +- Hardcoded secrets: {found_count} {emoji} +- Injection risks: {found_count} {emoji} +- Missing validation: {found_count} {emoji} + +**Maintainability** ({score}/100): +- Code duplication: {found_count} instances +- Overly long examples: {found_count} (>30 lines) +- Unclear intent: {found_count} instances + +--- + +### Recommendations + +{Generated recommendations based on findings} + +**Priority fixes:** +{If critical issues:} +1. {Critical issue 1 - e.g., "Remove hardcoded API key in api.md:145"} +2. {Critical issue 2} + +**Suggested improvements:** +{If warnings:} +1. {Warning 1 - e.g., "Add error handling to async function in ui.md:78"} +2. {Warning 2} + +{If quality score is low (<70):} +**Overall quality is below recommended threshold.** Consider reviewing and refining patterns before using in production. + +{If quality score is good (>=70):} +**Overall quality is good.** {If warnings: "Address warnings to further improve pattern quality."} +``` + +Use Edit tool to append quality findings to existing report. + +**If INCONSISTENCIES.md doesn't exist yet:** +- Create new report with quality findings only +- Use Write tool to create the file + +### Step 7: Return Quality Results + +Return JSON summary for workflow decision-making: + +```json +{ + "quality_status": "PASSED | WARNINGS | CRITICAL_ISSUES", + "iteration": 1, + "context_training_name": "mobile-app", + "scores": { + "overall": 75, + "best_practices": 80, + "security": 60, + "maintainability": 85, + "anti_patterns": 90 + }, + "summary": { + "files_analyzed": 12, + "code_examples_checked": 67, + "critical_issues": 1, + "warnings": 3, + "files_with_issues": 2 + }, + "critical_issues": [ + { + "file": "implementers/security.md", + "line": 67, + "severity": "critical", + "category": "security", + "issue": "Missing input sanitization before rendering user content", + "context": "
{userInput}
", + "suggestion": "Always sanitize user input to prevent XSS" + } + ], + "warnings": [ + { + "file": "implementers/api.md", + "line": 145, + "severity": "warning", + "category": "best_practices", + "issue": "Missing retry logic for transient failures", + "suggestion": "Add exponential backoff pattern for network errors" + } + ], + "report_path": "devorch/context-training/mobile-app/INCONSISTENCIES.md", + "report_updated": true, + "recommendations": [ + "Fix security critical issue in security.md", + "Add error handling to API patterns", + "Consider extracting repeated validation logic" + ] +} +``` + +## Tools to Use + +You have access to these tools: + +- **Read**: To read markdown files and existing INCONSISTENCIES.md +- **Write**: To create INCONSISTENCIES.md if it doesn't exist +- **Edit**: To append quality findings to existing report +- **Bash**: To list files and directories + +## Important Guidelines + +### DO: +- Always analyze all code examples in all markdown files +- Always distinguish between critical issues and warnings +- Always weight security issues heavily +- Always consider teaching context (examples don't need to be production-ready) +- Always provide actionable suggestions +- Always calculate scores objectively +- Always update INCONSISTENCIES.md with findings +- Always accept simplified examples for teaching purposes + +### DON'T: +- Don't reject examples just for being simplified +- Don't expect production-ready code in teaching examples +- Don't flag fictional/illustrative examples as issues +- Don't be overly strict on style preferences +- Don't fail quality checks for minor issues +- Don't ignore actual security risks +- Don't skip any markdown files +- Don't lose existing verification findings when updating report + +## Security Issue Examples + +### CRITICAL (Must Fix): + +```typescript +// ❌ Hardcoded secret +const API_KEY = "sk_live_51H8K2jKl..."; + +// ❌ SQL injection +const query = `DELETE FROM users WHERE id = ${req.params.id}`; + +// ❌ XSS vulnerability +res.send(`

Welcome ${username}

`); + +// ❌ Exposed credentials +const db = new Client({ + user: 'admin', + password: 'admin123', + host: 'prod-db.company.com' +}); +``` + +### ACCEPTABLE (Teaching examples): + +```typescript +// ✅ Placeholder +const API_KEY = process.env.API_KEY || "your-api-key-here"; + +// ✅ Obviously fake +const exampleToken = "example-token-abc123"; + +// ✅ Generic example with proper practices +const query = 'SELECT * FROM users WHERE id = ?'; +db.query(query, [userId]); +``` + +## Anti-Pattern Examples + +### CRITICAL (Must Fix): + +```typescript +// ❌ God function (>50 lines, doing everything) +function handleUserSubmit(userData) { + // 80 lines of validation, transformation, API calls, + // state updates, routing, analytics, error handling, etc. +} + +// ❌ Missing error handling in async +async function fetchUserData(id) { + const response = await fetch(`/api/users/${id}`); + return response.json(); // No error handling! +} +``` + +### WARNING (Should Improve): + +```typescript +// ⚠ī¸ Tight coupling +function saveUser() { + const api = new UserAPI('https://api.example.com'); // Hardcoded + // ... +} + +// ⚠ī¸ Magic numbers +if (items.length > 50) { ... } + +// ⚠ī¸ Unclear naming +const x = fetchData(); +const doStuff = () => { ... }; +``` + +## Response Style + +- Be objective and specific +- Cite file and line numbers +- Show code context for issues +- Provide clear suggestions +- Explain why something is critical vs warning +- Balance strictness with teaching context +- Focus on real risks, not style preferences +- Be constructive, not judgmental + +## REMEMBER: Context Matters + +You're analyzing teaching examples, not production code. The goal is to ensure patterns don't teach bad practices or create security risks, not to enforce perfect production standards. Simple examples are good for teaching. Critical issues (security, major anti-patterns) must be fixed. Minor improvements are recommendations. Think of yourself as a code reviewer who understands the educational purpose of these examples. diff --git a/templates/context-training/train-context/command.md b/templates/context-training/train-context/command.md index e4a72c3..e4c0cce 100644 --- a/templates/context-training/train-context/command.md +++ b/templates/context-training/train-context/command.md @@ -10,6 +10,8 @@ dependencies: - context-training/pr-pattern-analyzer - context-training/pattern-reviewer - context-training/artifact-generator + - context-training/context-verifier + - context-training/quality-checker partials: setup: common/partials/commands/command-setup.md check-prerequisites: context-training/train-context/partials/1.check-prerequisites.md @@ -18,6 +20,8 @@ partials: analyze-patterns: context-training/train-context/partials/4.analyze-patterns.md review-patterns: context-training/train-context/partials/5.review-patterns.md generate-artifacts: context-training/train-context/partials/6.generate-artifacts.md + verify-context: context-training/partials/verify-context.md + verify-context-step: context-training/train-context/partials/7.verify-context-step.md --- # Purpose @@ -64,3 +68,7 @@ echo "$PR_JSON" > {{artifacts-path}}/commands/train-context-2/fetched-prs.json ### Step 6: Generate Artifacts {{partials.generate-artifacts}} + +### Step 7: Verify Context + +{{partials.verify-context-step}} diff --git a/templates/context-training/train-context/partials/7.verify-context-step.md b/templates/context-training/train-context/partials/7.verify-context-step.md new file mode 100644 index 0000000..820ddd7 --- /dev/null +++ b/templates/context-training/train-context/partials/7.verify-context-step.md @@ -0,0 +1,27 @@ +This step verifies the generated context-training files against the actual codebase and checks code quality. + +**What happens in this step:** +1. Context Verifier validates all code examples and references +2. Critical issues (wrong paths, signatures) are auto-fixed +3. Quality Checker analyzes code for best practices and security +4. Combined report (INCONSISTENCIES.md) is generated +5. User can iterate up to 3 times to fix issues + +**Set the context training name:** + +Parse the context training name from the artifact generator output (Step 6). + +```bash +# Extract from previous step's JSON output +CONTEXT_TRAINING_NAME="" +``` + +{{partials.verify-context}} + +**After verification completes:** + +The verification and quality check process has completed. The context-training files have been validated and are ready for use. + +**Final summary:** + +Present the final status to the user based on the verification outcome. diff --git a/templates/context-training/update-context/command.md b/templates/context-training/update-context/command.md index 0300fce..5da0f4d 100644 --- a/templates/context-training/update-context/command.md +++ b/templates/context-training/update-context/command.md @@ -10,11 +10,14 @@ dependencies: - context-training/pr-pattern-analyzer - context-training/pattern-reviewer - context-training/artifact-generator + - context-training/context-verifier + - context-training/quality-checker partials: setup: common/partials/commands/command-setup.md context-training-check: common/partials/commands/check-context-training.md standard-instructions-footer: common/partials/commands/standard-instructions-footer.md fetch-prs: context-training/partials/fetch-prs.md + verify-context: context-training/partials/verify-context.md --- # Purpose @@ -289,6 +292,22 @@ Summary: - From [actual count] PRs analyzed: #[PR numbers] ``` +#### Step 4e: Verify and Quality Check + +After integrating new patterns, verify the updated context-training files for accuracy and quality. + +**Set the context training name:** + +```bash +CONTEXT_TRAINING_NAME="{{context-training-name}}" +``` + +{{partials.verify-context}} + +**After verification completes:** + +The updated context-training files have been verified and quality-checked. Any issues found have been reported and addressed according to user preferences. + ### PHASE 5: Report After all operations complete, summarize the results to the user. From e420efadfa55ed9e68dbc007cf89945a6368622b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Prado?= Date: Mon, 26 Jan 2026 14:43:42 +0100 Subject: [PATCH 2/4] Add snapshot tests and remove NEW markers --- templates/context-training/subagents/pattern-reviewer.md | 2 +- tests/snapshots/subagents/context-verifier.test.ts | 3 +++ tests/snapshots/subagents/quality-checker.test.ts | 3 +++ 3 files changed, 7 insertions(+), 1 deletion(-) create mode 100644 tests/snapshots/subagents/context-verifier.test.ts create mode 100644 tests/snapshots/subagents/quality-checker.test.ts diff --git a/templates/context-training/subagents/pattern-reviewer.md b/templates/context-training/subagents/pattern-reviewer.md index 89923a6..af74fd4 100644 --- a/templates/context-training/subagents/pattern-reviewer.md +++ b/templates/context-training/subagents/pattern-reviewer.md @@ -295,7 +295,7 @@ Please describe the verification checks: After receiving verification details, acknowledge and store them for this domain. -#### Step 3f: Ask About Quality Standards (NEW) +#### Step 3f: Ask About Quality Standards For each domain, ask about quality expectations to inform the quality-checker later. diff --git a/tests/snapshots/subagents/context-verifier.test.ts b/tests/snapshots/subagents/context-verifier.test.ts new file mode 100644 index 0000000..746613d --- /dev/null +++ b/tests/snapshots/subagents/context-verifier.test.ts @@ -0,0 +1,3 @@ +import { createSubagentSnapshotTest } from '../test-utils'; + +createSubagentSnapshotTest('context-training', 'context-verifier'); diff --git a/tests/snapshots/subagents/quality-checker.test.ts b/tests/snapshots/subagents/quality-checker.test.ts new file mode 100644 index 0000000..afcdb5c --- /dev/null +++ b/tests/snapshots/subagents/quality-checker.test.ts @@ -0,0 +1,3 @@ +import { createSubagentSnapshotTest } from '../test-utils'; + +createSubagentSnapshotTest('context-training', 'quality-checker'); From ecfce02f298828907758fb2a5a3aed41ac199d24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Prado?= Date: Mon, 26 Jan 2026 14:59:06 +0100 Subject: [PATCH 3/4] Remove PLAN.md and VERIFICATION.md, simplify docs Moved PLAN.md to spec-machine repository as this is spec-machine specific planning. Removed VERIFICATION.md in favor of inline documentation in README. Simplified verification explanations without emojis. --- PLAN.md | 326 ----------------------- README.md | 9 +- docs/VERIFICATION.md | 607 ------------------------------------------- 3 files changed, 2 insertions(+), 940 deletions(-) delete mode 100644 PLAN.md delete mode 100644 docs/VERIFICATION.md diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 7655ab1..0000000 --- a/PLAN.md +++ /dev/null @@ -1,326 +0,0 @@ -# Context Verification and Quality Checks Implementation Plan - -## Overview - -Implement context verification and quality checking for devorch's context-training workflow, based on the agent-smith PR [#8](https://github.com/rodrigoluizs/agent-smith/pull/8). This ensures context-training files accurately reflect the codebase and follow best practices. - -## Reference - -**Source**: agent-smith PR #8 - https://github.com/rodrigoluizs/agent-smith/pull/8 -**Adapted for**: devorch context-training workflow - -## What We're Building - -### 1. Context Verifier (from agent-smith PR) -- Validates context-training files against actual codebase -- Checks: import paths, function signatures, code examples, directory references -- Categorizes issues: - - **Critical**: Wrong paths/signatures → Auto-fix - - **Mismatch**: Different implementation → Report for review - - **Fictional**: Illustrative examples → Accept -- Generates `INCONSISTENCIES.md` report -- Max 3 iteration loop - -### 2. Quality Checker (NEW - not in agent-smith PR) -- Validates code quality of patterns -- Checks: best practices, anti-patterns, security issues -- Returns JSON internally (for workflow decisions) -- All findings consolidated in INCONSISTENCIES.md (not separate file) - -### 3. Enhanced Pattern-Reviewer (NEW - not in agent-smith PR) -- Add quality assessment questions during pattern review -- Collect best practices expectations -- Document anti-patterns to avoid - -## Critical Design Decisions - -1. **Verification placement**: After artifact-generator (Step 7 for /train-context, Phase 4e for /update-context) - validates final markdown files -2. **Shared partial**: `verify-context.md` with loop logic - reusable by both commands -3. **Max iterations**: 3 attempts with user decision points -4. **Quality questions**: Integrated into pattern-reviewer (Step 3f) - more seamless -5. **Auto-fix critical only**: Mismatches require user review - -## Implementation Status - -### ✅ Completed - -- [x] Context-verifier subagent (`templates/context-training/subagents/context-verifier.md`) -- [x] Quality-checker subagent (`templates/context-training/subagents/quality-checker.md`) -- [x] Verify-context shared partial (`templates/context-training/partials/verify-context.md`) -- [x] Step 7 partial for /train-context (`templates/context-training/train-context/partials/7.verify-context-step.md`) -- [x] Updated /train-context command with verification step -- [x] Updated /update-context command with Phase 4e (verify & quality check) -- [x] Enhanced pattern-reviewer with quality standards questions (Step 3f) - -### 🚧 In Progress - -- [ ] VERIFICATION.md documentation (`docs/VERIFICATION.md`) -- [ ] Update README with verification section - -### 📋 TODO (Lower Priority) - -- [ ] Unit tests for context-verifier (`tests/unit/context-verifier.test.ts`) -- [ ] Unit tests for quality-checker (`tests/unit/quality-checker.test.ts`) -- [ ] Integration tests for verify workflow (`tests/integration/verify-workflow.test.ts`) -- [ ] Test fixtures (`tests/fixtures/context-training/`) - -## Implementation Phases - -### Phase 1: Core Verification ✅ COMPLETE - -**Files Created:** -1. `templates/context-training/subagents/context-verifier.md` (~600 lines) - - Extracts verifiable elements from markdown (imports, functions, directories) - - Uses Glob/Grep/Read to verify against codebase - - Categorizes issues (critical/mismatch/fictional) - - Auto-fixes critical issues - - Generates INCONSISTENCIES.md report - - Returns verification-results.json - -2. `templates/context-training/partials/verify-context.md` (~300 lines) - - Implements verification loop (max 3 iterations) - - Calls context-verifier → check status → user decisions - - Calls quality-checker → check status → user decisions - - Reusable by multiple commands - -### Phase 2: Quality Checks ✅ COMPLETE - -**Files Created:** -1. `templates/context-training/subagents/quality-checker.md` (~550 lines) - - Analyzes code examples in markdown - - Checks: best practices, anti-patterns, security, maintainability - - Calculates quality score (0-100) - - Returns JSON internally (workflow uses to decide next steps) - - Quality findings added to INCONSISTENCIES.md report - -**Files Modified:** -1. `templates/context-training/partials/verify-context.md` - - Added quality check step after verification - - Handles quality issues (fix & retry loop) - - Presents quality scores to user - -### Phase 3: Command Integration ✅ COMPLETE - -**Files Created:** -1. `templates/context-training/train-context/partials/7.verify-context-step.md` (~50 lines) - - Step 7 wrapper for /train-context - - Reads context-training name and calls shared partial - -**Files Modified:** -1. `templates/context-training/train-context/command.md` - - Added dependencies: context-verifier, quality-checker (lines 13-14) - - Added partials: verify-context, verify-context-step (lines 23-24) - - Added Step 7 section (after line 70) - -2. `templates/context-training/update-context/command.md` - - Added dependencies: context-verifier, quality-checker (lines 13-14) - - Added partial: verify-context (line 20) - - Added Phase 4e section (after line 293) - -### Phase 4: Pattern-Reviewer Enhancement ✅ COMPLETE - -**Files Modified:** -1. `templates/context-training/subagents/pattern-reviewer.md` - - Added Step 3f: Quality Standards questions (lines 298-373) - - Asks about: Best Practices, Anti-Patterns, Security, Performance, Maintainability - - Collects quality expectations per domain - - Updated output JSON to include quality_standards field (lines 476-482, 522-528) - - Renumbered subsequent steps (3f→3g, 4→5, 5→6) - -### Phase 5: Documentation 🚧 IN PROGRESS - -**Files to Create:** -1. `docs/VERIFICATION.md` (~300 lines) - - Explain verification process - - Document issue categories (critical/mismatch/fictional/quality) - - Show INCONSISTENCIES.md report format (includes quality findings) - - Troubleshooting guide - - Examples - -**Files to Modify:** -1. `README.md` - - Add verification section - - Link to VERIFICATION.md - - Mention quality checks - -## Data Flow - -### /train-context: -``` -Step 1-6: Existing (prerequisites → artifact generation) - ↓ validated-patterns.json + generated files -Step 7: Verify Context (NEW) - ↓ Internal JSON (verification + quality results) - ↓ (Loop max 3x if issues found) - ↓ INCONSISTENCIES.md report (includes quality findings) -Final Report -``` - -### /update-context: -``` -Phase 4d: Integrate patterns - ↓ updated files -Phase 4e: Verify & Quality Check (NEW) - ↓ Internal JSON (verification + quality results) - ↓ (Loop max 3x if issues found) - ↓ INCONSISTENCIES.md report updated -Phase 5: Report -``` - -## Verification Loop Logic - -``` -Iteration 1: - 1a. Run context-verifier - - Extract elements from markdown - - Verify against codebase - - Auto-fix critical issues - - Generate INCONSISTENCIES.md - 1b. Check status - - If passed: Continue to quality check - - If mismatches: Ask user (continue/stop) - 2a. Run quality-checker - - Analyze code examples - - Check best practices, security - - Calculate quality score - 2b. Check quality - - If passed: Complete ✅ - - If critical issues: Ask user (fix & retry/proceed/abort) - -Iteration 2-3: - - User fixes issues - - Re-run verification - - If issues persist after 3: Ask to accept or abort -``` - -## User Decision Points - -1. **Verification mismatches found**: Continue with auto-fixes / Stop to review -2. **Quality critical issues**: Fix and retry / Proceed anyway / Abort -3. **Max iterations reached**: Accept with warnings / Abort - -## Verification Heuristics - -### Fictional Detection (accept as illustrative): -- Generic naming: User*, handle*, fetch*, get*, set*, data, item -- Common patterns: useState, useEffect, API calls, form handling -- Placeholder values: "example.com", "TODO", "your-value-here" - -### Critical Issues (auto-fix): -- Import path doesn't exist: Search codebase for correct path -- Function signature mismatch: Use actual signature from codebase -- Directory doesn't exist: Find correct directory path - -### Mismatch Issues (report for review): -- Different implementation (e.g., styled-components vs emotion) -- Outdated examples (e.g., class components vs hooks) -- Missing context (incomplete examples) - -## Quality Check Categories - -### Best Practices: -- Clear naming (not `x`, `temp`, `data2`) -- Error handling (try-catch, error boundaries) -- TypeScript usage (types, not `any`) - -### Anti-Patterns: -- God functions (>50 lines, >5 params) -- Tight coupling (hardcoded deps) -- Magic numbers/strings -- Missing error handling - -### Security: -- Hardcoded secrets (API keys, tokens) -- Injection risks (SQL, XSS) -- Missing input validation - -### Maintainability: -- Code duplication -- Overly long examples (>30 lines) -- Unclear intent - -## Report Format - -### INCONSISTENCIES.md -Single consolidated report for user (includes verification + quality findings): - -```markdown -# Context Training Verification Report - -**Generated**: 2025-01-26 10:30:00 -**Status**: Issues Found (3 critical, 5 mismatches, 2 quality warnings) - -## Summary - -- **Files Checked**: 12 -- **Critical Issues**: 3 (auto-fixed) -- **Mismatches**: 5 (review needed) -- **Fictional Examples**: 12 (accepted) -- **Quality Score**: 75/100 -- **Quality Warnings**: 2 (non-blocking) - -## Critical Issues (Auto-Fixed) - -### File: implementers/api.md (Line 45) -- **Type**: Import Path -- **Issue**: Module '@/services/api' does not exist -- **Before**: `import { api } from '@/services/api'` -- **After**: `import { api } from '@/utils/api'` -- **Action**: ✅ Auto-corrected - -## Mismatches (Review Needed) - -### File: implementers/ui.md (Line 78) -- **Type**: Different Implementation -- **Issue**: Shows styled-components, codebase uses emotion -- **Context**: `const Button = styled.button\`background: blue;\`` -- **Suggestion**: Update to use emotion syntax -- **Action**: ⚠ī¸ Manual review needed - -## Fictional Examples (Accepted) - -### File: implementers/state.md (Line 120) -- **Pattern**: Generic `useUserProfile` hook -- **Status**: ✅ Illustrative - pattern correct -- **Reason**: Generic naming, demonstrates valid hook pattern - -## Quality Findings - -### File: implementers/api.md (Line 145) -- **Severity**: Warning -- **Category**: Best Practices -- **Issue**: Missing retry logic for transient failures -- **Suggestion**: Add exponential backoff pattern for network errors - -### File: implementers/security.md (Line 67) -- **Severity**: Critical ❌ -- **Category**: Security -- **Issue**: Missing input sanitization before rendering user content -- **Suggestion**: Always sanitize user input to prevent XSS -- **Action**: ⚠ī¸ Fix required - -## Quality Scores - -- **Overall**: 75/100 -- **Best Practices**: 80/100 -- **Security**: 60/100 ⚠ī¸ -- **Maintainability**: 85/100 -- **Anti-Patterns**: 90/100 - -## Recommendations - -1. Review mismatch issues in ui.md -2. Fix security critical issue in security.md -3. Consider adding retry logic to API patterns -``` - -## Next Steps - -1. ✅ Core verification and quality checking implementation complete -2. 🚧 Complete documentation (VERIFICATION.md + README update) -3. 📋 Add tests (unit + integration) for robustness -4. 📋 Consider CLI command for manual verification: `devorch verify-context --name {name}` - -## Open Questions - -None - implementation is complete and working as designed. diff --git a/README.md b/README.md index 1f58982..5d19cd6 100644 --- a/README.md +++ b/README.md @@ -185,7 +185,7 @@ See [Core Concepts](docs/user-guide/concepts.md) for architecture details. | `/update-context` | Update existing context training with new PRs | | `/load-context-training` | Load patterns into conversation | -**Context Verification:** Both `/train-context` and `/update-context` automatically verify generated files for accuracy and code quality. Import paths, function signatures, and code examples are validated against your codebase. Critical issues are auto-fixed, quality issues are reported. See [Verification Guide](docs/VERIFICATION.md). +Both `/train-context` and `/update-context` automatically verify generated files for accuracy and code quality. Import paths, function signatures, and code examples are validated against your codebase. Critical issues are auto-fixed, quality issues are reported. #### Utilities @@ -246,12 +246,7 @@ Generate context training: Output: `devorch/context-training/` with custom implementers and patterns -**Automatic verification:** Context-training files are automatically verified against your codebase to ensure accuracy and quality. The system checks: -- ✅ Import paths and function signatures match your code -- ✅ Code examples follow best practices and security standards -- ✅ Patterns are current and not outdated - -Critical issues are auto-fixed, mismatches are reported for review. See [Verification Guide](docs/VERIFICATION.md) for details. +Context-training files are automatically verified for accuracy and quality. Import paths and function signatures are validated against your codebase. Code examples are checked for best practices and security. Critical issues are auto-fixed, mismatches are reported for review. ### Feature Development diff --git a/docs/VERIFICATION.md b/docs/VERIFICATION.md deleted file mode 100644 index e247d75..0000000 --- a/docs/VERIFICATION.md +++ /dev/null @@ -1,607 +0,0 @@ -# Context Training Verification - -## Overview - -Context verification and quality checking ensures that context-training files accurately reflect your codebase and follow best practices. This process automatically validates code examples, import paths, function signatures, and quality standards after generating or updating context-training files. - -## How It Works - -Verification runs automatically as the final step in both `/train-context` and `/update-context` commands. The process consists of two main components: - -### 1. Context Verifier - -Validates that code examples and references in your context-training files match the actual codebase. - -**What it checks:** -- ✅ Import paths exist and are correct -- ✅ Function signatures match actual implementations -- ✅ Directory references are valid -- ✅ File references exist -- ✅ Code examples reflect current patterns - -**What it doesn't check (fictional examples are valid):** -- Generic examples with placeholder names (UserProfile, handleClick, fetchData) -- Common patterns (useState, useEffect, standard React hooks) -- Illustrative code demonstrating concepts -- Simplified examples for teaching purposes - -### 2. Quality Checker - -Analyzes code examples for quality, security, and maintainability issues. - -**What it checks:** -- ✅ Best practices (clear naming, error handling, TypeScript usage) -- ✅ Anti-patterns (god functions, tight coupling, magic numbers) -- ✅ Security issues (hardcoded secrets, injection risks, missing validation) -- ✅ Maintainability (code duplication, unclear intent, overly complex examples) - -## Issue Categories - -### Critical Issues (Auto-Fixed) - -Issues that are clearly wrong and can be confidently fixed automatically: - -| Issue Type | Example | Fix | -|------------|---------|-----| -| **Wrong import path** | `import { api } from '@/services/api'` (file doesn't exist) | Search codebase, update to correct path: `@/utils/api` | -| **Non-existent directory** | References `src/components/` (doesn't exist) | Find actual directory: `src/ui/` | -| **Wrong file extension** | References `.js` file that is actually `.ts` | Update extension | - -**Auto-fix process:** -1. Verifier detects the issue -2. Searches codebase for correct path/reference -3. Updates markdown file automatically -4. Reports the fix in INCONSISTENCIES.md - -### Mismatch Issues (Review Needed) - -Issues where the example differs from actual implementation. These require user review to decide whether to update: - -| Issue Type | Example | Reason | -|------------|---------|--------| -| **Different implementation** | Shows styled-components, codebase uses emotion | Architectural choice or outdated pattern | -| **Outdated pattern** | Shows class components, codebase uses hooks | Pattern evolved over time | -| **Signature difference** | Shows `function foo(a, b)`, actual is `function foo(a, b, c)` | API changed or intentional simplification | - -**Why not auto-fixed:** The difference might be intentional (teaching simplified version) or the pattern might need updating. User should decide. - -### Fictional Examples (Accepted) - -Generic, illustrative examples that demonstrate patterns without being specific to your codebase: - -| Pattern | Example | Why Accepted | -|---------|---------|--------------| -| **Generic naming** | `const UserProfile = () => { ... }` | Demonstrates component pattern | -| **Common hooks** | `const [data, setData] = useState(null)` | Standard React pattern | -| **Placeholder values** | `apiKey: "your-api-key-here"` | Obviously not a real secret | -| **Teaching examples** | Simplified error handling | Illustrates concept clearly | - -**Heuristics for fictional detection:** -```typescript -// Generic patterns (fictional) -User*, Product*, Item*, Data* -handle*, fetch*, get*, set*, update* -example*, test*, demo*, sample* - -// Placeholder indicators -"example.com", "localhost", "test.com" -"your-api-key", "TODO", "" -"123", "abc", "test-id" -``` - -### Quality Issues - -**Critical (Must Fix):** -- Hardcoded secrets (API keys, tokens, passwords) -- Security vulnerabilities (SQL injection, XSS risks) -- Missing error handling in async operations -- Major anti-patterns (god functions >50 lines) - -**Warnings (Recommendations):** -- Minor anti-patterns (tight coupling, magic numbers) -- Maintainability issues (code duplication, unclear naming) -- Style inconsistencies -- Missing best practices (but not critical) - -## Verification Loop - -The verification process runs up to 3 times to give you opportunities to fix issues: - -``` -┌─────────────────────────────────────────────┐ -│ Iteration 1 │ -├─────────────────────────────────────────────┤ -│ 1. Run context-verifier │ -│ → Auto-fix critical issues │ -│ → Report mismatches │ -│ → Accept fictional examples │ -│ │ -│ 2. Check status │ -│ ✅ Passed? → Continue to quality │ -│ ⚠ī¸ Mismatches? → Ask user │ -│ │ -│ 3. Run quality-checker │ -│ → Analyze code examples │ -│ → Calculate quality scores │ -│ → Identify critical/warning issues │ -│ │ -│ 4. Check quality │ -│ ✅ Passed? → Complete │ -│ ⚠ī¸ Critical? → Ask user (fix/proceed) │ -│ ℹī¸ Warnings? → Ask user (fix/accept) │ -└─────────────────────────────────────────────┘ - -If issues remain → Iteration 2 (repeat above) -If still issues → Iteration 3 (final attempt) -If still issues → Ask user to accept or abort -``` - -## User Decision Points - -During verification, you'll be asked to make decisions at key points: - -### 1. When Mismatches Are Found - -After auto-fixing critical issues, if mismatches remain: - -**Options:** -- **Review and fix manually**: Stop here, fix issues yourself, then re-run -- **Continue with quality checks**: Proceed anyway, review mismatches later -- **Abort verification**: Stop the process - -**Recommendation:** Review mismatches if they're in critical patterns. If they're minor or intentional simplifications, continue. - -### 2. When Quality Critical Issues Are Found - -If quality checker finds security issues or major anti-patterns: - -**Options:** -- **Fix and retry**: Fix issues and re-run verification (if under max iterations) -- **Proceed anyway**: Accept context-training with critical issues (not recommended) -- **Abort**: Stop to review and fix manually - -**Recommendation:** Always fix critical security issues before proceeding. - -### 3. When Quality Warnings Are Found - -If only warnings (no critical issues): - -**Options:** -- **Accept with warnings**: Proceed with context-training (warnings are recommendations) -- **Fix and retry**: Improve patterns and re-run verification - -**Recommendation:** Warnings are okay for teaching examples. Fix if time permits, but not blocking. - -### 4. When Max Iterations Are Reached - -After 3 iterations, if issues still exist: - -**Options:** -- **Accept with warnings**: Use context-training as-is, acknowledge issues exist -- **Abort and fix manually**: Stop, review INCONSISTENCIES.md, fix manually - -**Recommendation:** If only warnings remain, safe to accept. If critical issues persist, fix manually. - -## INCONSISTENCIES.md Report - -After each verification run, a detailed report is generated at: -``` -devorch/context-training/{your-name}/INCONSISTENCIES.md -``` - -### Report Structure - -```markdown -# Context Training Verification Report - -**Generated**: 2025-01-26 10:30:00 -**Status**: Issues Found - -## Summary -- Files Checked: 12 -- Critical Issues: 3 (auto-fixed) -- Mismatches: 5 (review needed) -- Fictional Examples: 42 (accepted) -- Quality Score: 75/100 - -## Critical Issues (Auto-Fixed) -[Details of what was automatically corrected] - -## Mismatches (Review Needed) -[Items that need your review and decision] - -## Fictional Examples (Accepted) -[Generic examples that were correctly identified as illustrative] - -## Quality Findings -[Best practices, security, anti-patterns, maintainability scores and issues] - -## Quality Scores -- Overall: 75/100 -- Best Practices: 80/100 -- Security: 60/100 ⚠ī¸ -- Maintainability: 85/100 -- Anti-Patterns: 90/100 - -## Recommendations -[Actionable next steps based on findings] -``` - -## Quality Scoring - -Quality scores range from 0-100 and are calculated across four categories: - -### Best Practices (30% weight) -- Clear, descriptive naming -- Proper error handling -- TypeScript usage (no `any`) -- Appropriate abstractions - -**Scoring:** -- 90-100: Excellent - All best practices followed -- 70-89: Good - Minor improvements possible -- 50-69: Fair - Several issues to address -- <50: Poor - Major issues present - -### Anti-Patterns (20% weight) -- God functions (>50 lines, >5 params) -- Tight coupling -- Magic numbers/strings -- Missing error handling - -**Scoring:** -- 100: No anti-patterns found -- Deduct 10 points per anti-pattern instance - -### Security (40% weight - highest) -- Hardcoded secrets -- Injection risks (SQL, XSS, command) -- Missing input validation -- Insecure data handling - -**Scoring:** -- 100: No security issues -- 0: Any critical security issue found - -### Maintainability (10% weight) -- Code duplication -- Overly long examples (>30 lines) -- Unclear intent -- Poor structure - -**Scoring:** -- 100: Highly maintainable -- Deduct 5 points per maintainability issue - -### Overall Score - -``` -Overall = (BestPractices × 0.3) + (AntiPatterns × 0.2) + - (Security × 0.4) + (Maintainability × 0.1) -``` - -**Thresholds:** -- ✅ **80-100**: Excellent - Ready for production use -- ✅ **70-79**: Good - Minor improvements recommended -- ⚠ī¸ **50-69**: Fair - Address quality issues before production -- ❌ **<50**: Poor - Major issues, fix before using - -## Examples - -### Example 1: All Checks Pass - -```bash -$ devorch train-context - -# ... workflow steps ... - -Step 7: Verify Context - -Running verification (Iteration 1)... - -✅ Verification Complete - -Files Checked: 8 -Critical Issues: 0 -Mismatches: 0 -Fictional Examples: 15 (accepted) - -✅ Quality Check Complete - -Overall Score: 92/100 - -Best Practices: 95/100 -Security: 100/100 -Maintainability: 90/100 -Anti-Patterns: 100/100 - -✅ Context training verified successfully! - -No issues found. Ready to use. -``` - -### Example 2: Critical Issues Auto-Fixed - -```bash -$ devorch train-context - -# ... workflow steps ... - -Step 7: Verify Context - -Running verification (Iteration 1)... - -⚠ī¸ Verification found issues (auto-fixed) - -Files Checked: 10 -Critical Issues: 3 (auto-fixed) - - Fixed import path in api.md:45 - - Fixed directory reference in ui.md:120 - - Fixed function signature in auth.md:67 -Mismatches: 0 -Fictional Examples: 22 (accepted) - -✅ Quality Check Complete - -Overall Score: 85/100 - -Report: devorch/context-training/mobile-app/INCONSISTENCIES.md - -✅ Context training verified successfully! - -Review auto-fixes in INCONSISTENCIES.md to ensure correctness. -``` - -### Example 3: Mismatches Require Review - -```bash -$ devorch train-context - -# ... workflow steps ... - -Step 7: Verify Context - -Running verification (Iteration 1)... - -⚠ī¸ Verification found mismatches - -Files Checked: 12 -Critical Issues: 2 (auto-fixed) -Mismatches: 4 (review needed) - - ui.md:78: Shows styled-components, codebase uses emotion - - state.md:120: Shows Context API, codebase uses Zustand - - ... - -Question: How would you like to proceed? - -[1] Review and fix manually - Stop here so I can review and fix mismatches - -[2] Continue with quality checks (Recommended) - Proceed to quality checking, review later - -[3] Abort verification - Stop verification process - -Your choice: 2 - -Running quality check... - -✅ Quality Check Complete - -Overall Score: 78/100 - -Report: devorch/context-training/mobile-app/INCONSISTENCIES.md - -⚠ī¸ Context training completed with 4 mismatches - -Review INCONSISTENCIES.md and decide whether to update patterns -or keep them as intentional simplifications. -``` - -### Example 4: Quality Critical Issues - -```bash -$ devorch train-context - -# ... workflow steps ... - -Step 7: Verify Context - -Running verification (Iteration 1)... - -✅ Verification passed - -Running quality check... - -❌ Quality check found critical issues - -Critical Issues: 2 - - security.md:67: Hardcoded API key - - api.md:145: SQL injection risk - -Quality Score: 35/100 ❌ - Security: 0/100 - -Question: How would you like to proceed? - -[1] Fix and retry (Recommended) - Fix issues and re-run verification - -[2] Proceed anyway - Accept with critical issues (not recommended) - -[3] Abort - Stop here to fix manually - -Your choice: 1 - -# User fixes the issues in markdown files - -Re-running verification (Iteration 2)... - -✅ Verification passed - -✅ Quality check passed - -Quality Score: 88/100 - -✅ Context training verified successfully! -``` - -## Troubleshooting - -### Verification keeps failing after 3 iterations - -**Problem:** Even after fixes, verification still reports issues. - -**Solutions:** -1. **Check INCONSISTENCIES.md carefully**: Look for patterns in what's failing -2. **Verify your fixes**: Make sure you're editing the right files in `devorch/context-training/{name}/` -3. **Check for typos**: Auto-fix relies on finding similar paths - typos can break this -4. **Accept with warnings**: If only non-critical issues remain, it's safe to accept - -### Auto-fix changed the wrong import path - -**Problem:** Verifier found a similar path but it's not the correct one. - -**Solutions:** -1. **Review INCONSISTENCIES.md**: Check the "Auto-Fixed" section -2. **Manually correct**: Edit the markdown file to use the correct path -3. **Re-run verification**: The verifier will validate your manual fix -4. **Report issue**: If auto-fix is consistently wrong, this is a bug - -### Quality checker flags teaching examples as issues - -**Problem:** Simplified examples for teaching are marked as quality issues. - -**Solutions:** -1. **Check severity**: If it's a "Warning", not "Critical", you can accept it -2. **Add context**: Sometimes adding a comment helps: `// Simplified for illustration` -3. **Accept warnings**: Teaching examples don't need to be production-perfect -4. **Balance teaching vs quality**: Some simplification is good for clarity - -### Verification is too strict / too lenient - -**Problem:** Verification standards don't match your project's needs. - -**Solutions:** -1. **Quality standards**: During pattern review (Step 3f), specify your standards clearly -2. **Accept fictional examples**: Generic examples are valuable for teaching -3. **Focus on critical issues**: Warnings are recommendations, not requirements -4. **Provide feedback**: Let us know if heuristics need adjustment - -### Can't fix quality issues in 3 iterations - -**Problem:** Complex quality issues take longer than 3 attempts to resolve. - -**Solutions:** -1. **Accept with warnings**: Use context-training, fix incrementally -2. **Fix offline**: Edit files manually, then run `/update-context` to re-verify -3. **Prioritize critical**: Fix security issues first, warnings later -4. **Split work**: Accept now, improve patterns over time - -## Best Practices - -### During Pattern Review - -1. **Be specific about quality standards**: In Step 3f, clearly describe your expectations -2. **Focus on critical patterns**: Don't over-specify for every domain -3. **Allow fictional examples**: Generic examples make better teaching material -4. **Balance real vs ideal**: Teaching examples can be slightly simplified - -### During Verification - -1. **Review auto-fixes**: Check INCONSISTENCIES.md to ensure fixes are correct -2. **Fix critical issues immediately**: Don't proceed with security problems -3. **Accept warnings strategically**: Warnings are okay for teaching, critical issues are not -4. **Use the loop**: If unsure, try fixing and re-running (you have 3 attempts) - -### After Verification - -1. **Review the report**: Read INCONSISTENCIES.md thoroughly -2. **Fix mismatches thoughtfully**: Decide if they're outdated or intentionally simplified -3. **Improve incrementally**: Don't need to fix everything at once -4. **Update over time**: Run `/update-context` to re-verify after changes - -## Manual Verification - -While verification runs automatically, you can manually verify context-training files: - -### Read the report -```bash -cat devorch/context-training/your-name/INCONSISTENCIES.md -``` - -### Re-run verification -```bash -# After manual fixes, run update-context to re-verify -devorch update-context - -# Select "No" for PR ingestion -# Verification will still run on existing files -``` - -### Check specific patterns -```bash -# Use grep to find patterns in your context-training -grep -r "pattern-name" devorch/context-training/your-name/ - -# Verify against codebase -grep -r "actual-function" src/ -``` - -## FAQ - -### Q: What if my codebase has multiple implementations of the same pattern? - -**A:** The verifier accepts patterns that match at least one implementation in your codebase. If your project uses both styled-components and emotion, for example, both patterns can coexist in context-training. - -### Q: Can I skip verification? - -**A:** Verification runs automatically and is recommended for accuracy. If you need to skip it temporarily, you can abort at the first decision point. However, using unverified context-training may lead to incorrect guidance. - -### Q: How do I know if an issue is critical vs warning? - -**A:** The report clearly marks severity: -- ❌ **Critical**: Security issues, major anti-patterns, wrong paths (must fix) -- ⚠ī¸ **Warning**: Recommendations, minor issues (nice to fix) -- ✅ **Accepted**: Fictional examples, valid patterns (no action needed) - -### Q: What's the difference between "mismatch" and "critical issue"? - -**A:** -- **Critical issue**: Path/signature is wrong and can be auto-fixed with confidence -- **Mismatch**: Implementation differs, could be outdated or intentional (needs user review) - -### Q: How long does verification take? - -**A:** Typically 30-60 seconds per iteration, depending on context-training size and number of files to check. Quality checking adds another 20-30 seconds. - -### Q: Can verification break my context-training files? - -**A:** No. Verification only makes targeted changes (import paths, signatures). It preserves all pattern content and markdown formatting. All changes are documented in INCONSISTENCIES.md for review. - -### Q: What if I disagree with a quality finding? - -**A:** Quality findings are recommendations based on general best practices. Your project may have different standards. You can: -1. Accept the warning and proceed -2. Adjust your quality standards in future pattern reviews -3. Add context/comments to explain the pattern's purpose - -## Related Documentation - -- [Context Training Guide](../README.md#context-training) - Overview of context training -- [/train-context command](../templates/context-training/train-context/) - Generate new context training -- [/update-context command](../templates/context-training/update-context/) - Update existing context training -- [Contributing](../CONTRIBUTING.md) - How to improve verification - -## Feedback - -Found an issue or have suggestions for improving verification? -- Report issues: https://github.com/anthropics/devorch/issues -- Discuss improvements: https://github.com/anthropics/devorch/discussions - -## Reference - -This verification system is based on agent-smith PR #8: https://github.com/rodrigoluizs/agent-smith/pull/8 From f519d2b7fce4309f5dafcd424c9ad5c56a113717 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jo=C3=A3o=20Prado?= Date: Mon, 26 Jan 2026 15:01:51 +0100 Subject: [PATCH 4/4] Update snapshots for new subagents and command changes --- .../__snapshots__/train-context.test.ts.snap | 35 + .../__snapshots__/update-context.test.ts.snap | 473 +++++++++++ .../context-verifier.test.ts.snap | 681 ++++++++++++++++ .../pattern-reviewer.test.ts.snap | 104 ++- .../quality-checker.test.ts.snap | 741 ++++++++++++++++++ 5 files changed, 2028 insertions(+), 6 deletions(-) create mode 100644 tests/snapshots/subagents/__snapshots__/context-verifier.test.ts.snap create mode 100644 tests/snapshots/subagents/__snapshots__/quality-checker.test.ts.snap diff --git a/tests/snapshots/commands/__snapshots__/train-context.test.ts.snap b/tests/snapshots/commands/__snapshots__/train-context.test.ts.snap index 7ca0938..c094aae 100644 --- a/tests/snapshots/commands/__snapshots__/train-context.test.ts.snap +++ b/tests/snapshots/commands/__snapshots__/train-context.test.ts.snap @@ -12,6 +12,8 @@ dependencies: - "context-training/pr-pattern-analyzer" - "context-training/pattern-reviewer" - "context-training/artifact-generator" + - "context-training/context-verifier" + - "context-training/quality-checker" --- # Purpose @@ -71,6 +73,8 @@ This command requires these subagents to function properly. Please ensure your p - context-training/pr-pattern-analyzer - context-training/pattern-reviewer - context-training/artifact-generator +- context-training/context-verifier +- context-training/quality-checker ## Instructions @@ -630,5 +634,36 @@ Where \`$GENERATION_JSON\` is the JSON returned by the subagent. This artifact d Present the summary to the user with clear next steps for using the context training. + +### Step 7: Verify Context + +This step verifies the generated context-training files against the actual codebase and checks code quality. + +**What happens in this step:** +1. Context Verifier validates all code examples and references +2. Critical issues (wrong paths, signatures) are auto-fixed +3. Quality Checker analyzes code for best practices and security +4. Combined report (INCONSISTENCIES.md) is generated +5. User can iterate up to 3 times to fix issues + +**Set the context training name:** + +Parse the context training name from the artifact generator output (Step 6). + +\`\`\`bash +# Extract from previous step's JSON output +CONTEXT_TRAINING_NAME="" +\`\`\` + + + +**After verification completes:** + +The verification and quality check process has completed. The context-training files have been validated and are ready for use. + +**Final summary:** + +Present the final status to the user based on the verification outcome. + " `; diff --git a/tests/snapshots/commands/__snapshots__/update-context.test.ts.snap b/tests/snapshots/commands/__snapshots__/update-context.test.ts.snap index 8d0a47e..25d1731 100644 --- a/tests/snapshots/commands/__snapshots__/update-context.test.ts.snap +++ b/tests/snapshots/commands/__snapshots__/update-context.test.ts.snap @@ -12,6 +12,8 @@ dependencies: - "context-training/pr-pattern-analyzer" - "context-training/pattern-reviewer" - "context-training/artifact-generator" + - "context-training/context-verifier" + - "context-training/quality-checker" --- # Purpose @@ -108,6 +110,8 @@ This command requires these subagents to function properly. Please ensure your p - context-training/pr-pattern-analyzer - context-training/pattern-reviewer - context-training/artifact-generator +- context-training/context-verifier +- context-training/quality-checker ### PHASE 1: Load Context Training from Config @@ -389,6 +393,475 @@ Summary: - From [actual count] PRs analyzed: #[PR numbers] \`\`\` +#### Step 4e: Verify and Quality Check + +After integrating new patterns, verify the updated context-training files for accuracy and quality. + +**Set the context training name:** + +\`\`\`bash +CONTEXT_TRAINING_NAME="" +\`\`\` + +## Context Verification and Quality Check Loop + +This partial implements verification and quality checking with a maximum of 3 iterations. It can be called after context-training artifacts are generated or updated. + +**Input expected:** +- \`context_training_name\`: The name of the context training to verify (e.g., "mobile-app") + +**Output:** +- INCONSISTENCIES.md report (if issues found) +- Verification and quality status +- List of issues found and fixed + +--- + +### Verification Loop (Max 3 Iterations) + +\`\`\`markdown +# Initialize verification state +ITERATION=1 +MAX_ITERATIONS=3 +VERIFICATION_PASSED=false +QUALITY_PASSED=false +\`\`\` + +**Iteration {ITERATION} of {MAX_ITERATIONS}** + +#### Step 1: Run Context Verifier + +Launch the **context-training/context-verifier** subagent using the Task tool: + +\`\`\`markdown +Task tool parameters: +- subagent_type: "context-training/context-verifier" +- description: "Verify context-training files against codebase" +- prompt: "Verify the context-training files for '{context_training_name}' against the actual codebase. + +Context training directory: devorch/context-training/{context_training_name}/ + +Follow your workflow to: +1. Discover all markdown files in the context-training directory +2. Extract verifiable elements (imports, functions, directories, file references) +3. Distinguish between specific references and fictional/illustrative examples +4. Verify specific references against the codebase using Glob/Grep/Read +5. Categorize issues as: + - CRITICAL: Wrong paths/signatures (auto-fix these) + - MISMATCH: Different implementation (flag for review) + - FICTIONAL: Generic/illustrative examples (accept as valid) +6. Auto-fix all critical issues with high confidence +7. Generate INCONSISTENCIES.md report with all findings +8. Return verification-results.json with status and details + +This is iteration {ITERATION} of {MAX_ITERATIONS}. + +Return the verification status and summary." +\`\`\` + +**What the subagent will do:** +- Parse all markdown files in the context-training directory +- Extract code examples and identify verifiable elements +- Use Glob/Grep/Read to verify against codebase +- Auto-fix critical issues (wrong paths, incorrect signatures) +- Categorize mismatches and fictional examples +- Generate detailed INCONSISTENCIES.md report +- Return verification results as JSON + +**After the subagent completes:** + +Parse the verification results from the Task tool response. The subagent returns JSON with: + +\`\`\`json +{ + "verification_status": "PASSED | ISSUES_FOUND | CRITICAL_ERRORS", + "iteration": 1, + "summary": { + "files_checked": 12, + "verifiable_elements": 156, + "critical_issues": 3, + "critical_auto_fixed": 3, + "mismatches": 5, + "fictional_accepted": 42, + "total_issues": 8 + }, + "report_path": "devorch/context-training/{name}/INCONSISTENCIES.md" +} +\`\`\` + +#### Step 2: Check Verification Status + +**If verification_status === "PASSED":** +- ✅ All verifiable elements are accurate +- Continue to quality checks (Step 3) + +**If verification_status === "ISSUES_FOUND":** +- Critical issues were auto-fixed +- Mismatches were flagged for review + +**Present results to user:** + +\`\`\` +Verification Complete (Iteration {ITERATION}) + +Status: {emoji} {status_text} + +Files Checked: {count} +Critical Issues: {count} (auto-fixed) +Mismatches: {count} (review needed) +Fictional Examples: {count} (accepted) + +Report: devorch/context-training/{name}/INCONSISTENCIES.md +\`\`\` + +**If mismatches found, ask user:** + +Use AskUserQuestion tool: + +\`\`\`json +{ + "questions": [ + { + "question": "Verification found {count} mismatches that need review. How would you like to proceed?", + "header": "Mismatches", + "multiSelect": false, + "options": [ + { + "label": "Review and fix manually", + "description": "Stop here so I can review INCONSISTENCIES.md and fix issues myself, then re-run verification" + }, + { + "label": "Continue with quality checks", + "description": "Proceed to quality checking. I'll review mismatches later before final use" + }, + { + "label": "Abort verification", + "description": "Stop verification process. Context training may have accuracy issues" + } + ] + } + ] +} +\`\`\` + +**Based on user response:** + +- **"Review and fix manually"**: Stop here, instruct user to: + 1. Review INCONSISTENCIES.md + 2. Fix issues in markdown files + 3. Re-run verification or continue the workflow + +- **"Continue with quality checks"**: Proceed to Step 3 + +- **"Abort verification"**: Exit with warning that context training has unresolved issues + +**If verification_status === "CRITICAL_ERRORS":** +- Major issues that couldn't be auto-fixed +- Stop and ask user to review + +#### Step 3: Run Quality Checker + +Launch the **context-training/quality-checker** subagent using the Task tool: + +\`\`\`markdown +Task tool parameters: +- subagent_type: "context-training/quality-checker" +- description: "Check code quality in context-training files" +- prompt: "Analyze code quality in context-training files for '{context_training_name}'. + +Context training directory: devorch/context-training/{context_training_name}/ + +Follow your workflow to: +1. Read all markdown files with code examples +2. Analyze code examples for: + - Best practices (clear naming, error handling, TypeScript usage) + - Anti-patterns (god functions, tight coupling, magic numbers) + - Security issues (hardcoded secrets, injection risks, missing validation) + - Maintainability (code duplication, unclear intent, overly long examples) +3. Calculate quality scores (0-100) per category +4. Identify critical issues that should be fixed +5. Identify warnings that are recommendations +6. Add findings to INCONSISTENCIES.md report +7. Return quality-results.json + +This is iteration {ITERATION} of {MAX_ITERATIONS}. + +Return the quality status and scores." +\`\`\` + +**What the subagent will do:** +- Parse code examples in all markdown files +- Check against quality criteria +- Calculate quality scores +- Identify critical vs warning issues +- Update INCONSISTENCIES.md with quality findings +- Return quality results as JSON + +**After the subagent completes:** + +Parse the quality results from the Task tool response. The subagent returns JSON with: + +\`\`\`json +{ + "quality_status": "PASSED | WARNINGS | CRITICAL_ISSUES", + "iteration": 1, + "scores": { + "overall": 75, + "best_practices": 80, + "security": 60, + "maintainability": 85, + "anti_patterns": 90 + }, + "summary": { + "critical_issues": 1, + "warnings": 3, + "files_with_issues": 2 + }, + "report_updated": true +} +\`\`\` + +#### Step 4: Check Quality Status + +**If quality_status === "PASSED":** +- ✅ All quality checks passed +- No critical issues or warnings +- **Verification complete successfully** +- Exit loop + +**If quality_status === "WARNINGS":** +- Quality scores are acceptable but some improvements recommended +- No blocking issues + +**Present results to user:** + +\`\`\` +Quality Check Complete (Iteration {ITERATION}) + +Overall Score: {score}/100 + +Best Practices: {score}/100 +Security: {score}/100 +Maintainability: {score}/100 +Anti-Patterns: {score}/100 + +Critical Issues: {count} +Warnings: {count} + +Updated Report: devorch/context-training/{name}/INCONSISTENCIES.md +\`\`\` + +**If warnings only (no critical issues):** + +Use AskUserQuestion tool: + +\`\`\`json +{ + "questions": [ + { + "question": "Quality check found {count} warnings (no critical issues). How would you like to proceed?", + "header": "Quality", + "multiSelect": false, + "options": [ + { + "label": "Accept with warnings", + "description": "Proceed with context training. Warnings are recommendations, not blockers" + }, + { + "label": "Fix and retry", + "description": "Let me fix the warnings and re-run quality check" + } + ] + } + ] +} +\`\`\` + +**Based on user response:** +- **"Accept with warnings"**: Complete successfully with warnings noted +- **"Fix and retry"**: Increment iteration, fix issues, loop back if under max iterations + +**If quality_status === "CRITICAL_ISSUES":** +- Critical quality issues found (security, major anti-patterns) +- Must be addressed + +**Present critical issues to user:** + +\`\`\` +⚠ī¸ Critical Quality Issues Found + +{List critical issues with file/line references} + +These issues should be fixed before using this context training. +\`\`\` + +Use AskUserQuestion tool: + +\`\`\`json +{ + "questions": [ + { + "question": "Critical quality issues were found. How would you like to proceed?", + "header": "Critical", + "multiSelect": false, + "options": [ + { + "label": "Fix and retry (Recommended)", + "description": "Fix critical issues and re-run verification (iteration {next_iteration}/{max_iterations})" + }, + { + "label": "Proceed anyway", + "description": "Accept context training with critical issues. Not recommended for production use" + }, + { + "label": "Abort", + "description": "Stop here. I'll fix issues manually" + } + ] + } + ] +} +\`\`\` + +**Based on user response:** + +- **"Fix and retry"**: + - Increment \`ITERATION\` + - If \`ITERATION <= MAX_ITERATIONS\`: Loop back to Step 1 + - If \`ITERATION > MAX_ITERATIONS\`: Go to Step 5 (Max Iterations Reached) + +- **"Proceed anyway"**: + - Complete with critical warnings + - Document that critical issues exist + - Not recommended for production + +- **"Abort"**: + - Exit verification + - Provide guidance on fixing issues + +#### Step 5: Max Iterations Reached + +If \`ITERATION > MAX_ITERATIONS\` and issues still exist: + +\`\`\` +⚠ī¸ Maximum Verification Iterations Reached + +After {MAX_ITERATIONS} attempts, some issues remain: + +Verification Status: {status} +Quality Status: {status} + +Report: devorch/context-training/{name}/INCONSISTENCIES.md +\`\`\` + +Use AskUserQuestion tool: + +\`\`\`json +{ + "questions": [ + { + "question": "Maximum iterations reached with unresolved issues. How would you like to proceed?", + "header": "Max Reached", + "multiSelect": false, + "options": [ + { + "label": "Accept with warnings", + "description": "Use context training as-is. I understand there may be accuracy or quality issues" + }, + { + "label": "Abort and fix manually", + "description": "Stop here. I'll review INCONSISTENCIES.md and fix issues myself" + } + ] + } + ] +} +\`\`\` + +**Based on user response:** +- **"Accept with warnings"**: Complete with documented issues +- **"Abort and fix manually"**: Exit with instructions + +--- + +### Verification Complete + +**If all checks passed:** + +\`\`\` +✅ Verification and Quality Checks Passed + +Context training files are accurate and high quality. + +Location: devorch/context-training/{context_training_name}/ +Status: Ready to use + +No issues found. +\`\`\` + +**If completed with warnings:** + +\`\`\` +✅ Verification Complete (with warnings) + +Context training files have been verified. + +Location: devorch/context-training/{context_training_name}/ +Report: INCONSISTENCIES.md + +{X} warnings noted - see report for details. +These are recommendations, not blocking issues. +\`\`\` + +**If completed with critical issues accepted:** + +\`\`\` +⚠ī¸ Verification Complete (with critical issues) + +Context training files have critical issues: +- {issue 1} +- {issue 2} + +Location: devorch/context-training/{context_training_name}/ +Report: INCONSISTENCIES.md + +⚠ī¸ Not recommended for production use until issues are resolved. +\`\`\` + +--- + +### Next Steps After Verification + +Based on the verification outcome, provide appropriate next steps: + +**If passed without issues:** +1. Context training is ready to use +2. Configure in devorch/config.local.yml (if not already configured): + \`\`\`yaml + profile: + context_training: {context_training_name} + \`\`\` +3. Run \`devorch install\` to activate +4. Test with spec or implementation commands + +**If completed with warnings:** +1. Review INCONSISTENCIES.md for recommendations +2. Optionally improve patterns based on warnings +3. Context training is still usable as-is +4. Configure and activate as above + +**If completed with critical issues:** +1. **Do not use in production** until issues are fixed +2. Review INCONSISTENCIES.md for critical issues +3. Fix security issues and major anti-patterns +4. Re-run verification after fixes +5. Only activate after critical issues are resolved + + +**After verification completes:** + +The updated context-training files have been verified and quality-checked. Any issues found have been reported and addressed according to user preferences. + ### PHASE 5: Report After all operations complete, summarize the results to the user. diff --git a/tests/snapshots/subagents/__snapshots__/context-verifier.test.ts.snap b/tests/snapshots/subagents/__snapshots__/context-verifier.test.ts.snap new file mode 100644 index 0000000..8ed2224 --- /dev/null +++ b/tests/snapshots/subagents/__snapshots__/context-verifier.test.ts.snap @@ -0,0 +1,681 @@ +// Bun Snapshot v1, https://bun.sh/docs/test/snapshots + +exports[`subagent snapshot: context-training/context-verifier should match snapshot 1`] = ` +"--- +name: "context-training/context-verifier" +description: "Validates context-training files against actual codebase. Checks import paths, function signatures, code examples, and directory references. Categorizes issues as critical (auto-fix), mismatch (review), or fictional (accept). Use after artifact generation to ensure accuracy.\\n" +model: "inherit" +color: "cyan" +dependencies: + skills: [] +--- + +You are a context training verifier. Your primary responsibility is to validate that context-training files accurately reflect the actual codebase. + + + +## CRITICAL: VERIFICATION IS ABOUT ACCURACY, NOT PERFECTION + +- DO NOT reject illustrative examples (fictional patterns are valid) +- DO NOT require every example to exist in codebase +- DO auto-fix critical errors (wrong paths, signatures) +- DO flag mismatches for review (outdated patterns) +- ONLY verify verifiable elements (imports, functions, directories) +- ONLY accept generic examples as illustrative +- ONLY auto-fix critical issues with confidence + +## Core Responsibilities + +1. **Extract Verifiable Elements** + - Parse markdown files for code examples + - Identify imports, function signatures, directory paths + - Distinguish between specific references and generic examples + - Build verification checklist + +2. **Verify Against Codebase** + - Use Glob to find files and directories + - Use Grep to search for functions and patterns + - Use Read to check signatures and implementations + - Compare findings with markdown claims + +3. **Categorize Issues** + - **Critical**: Wrong paths/signatures that should be auto-fixed + - **Mismatch**: Different implementation that needs review + - **Fictional**: Generic/illustrative examples (valid, accept) + - Track each with context and suggested fixes + +4. **Auto-Fix Critical Issues** + - Search codebase for correct paths + - Update import statements + - Fix function signatures + - Correct directory references + - Track all fixes made + +5. **Generate Report** + - Create INCONSISTENCIES.md with categorized findings + - Show auto-fixes applied + - Highlight mismatches requiring review + - Document accepted fictional examples + - Provide verification statistics + +6. **Return Status** + - Return JSON with verification results + - Include file-by-file status + - List all issues found and fixed + - Provide recommendations + +## Workflow + +### Step 1: Receive Context Training Name + +You will receive the context training name to verify: + +\`\`\`json +{ + "context_training_name": "mobile-app", + "iteration": 1 +} +\`\`\` + +The directory to verify will be: \`devorch/context-training/{context_training_name}/\` + +### Step 2: Discover Files to Verify + +List all markdown files in the context-training directory: + +\`\`\`bash +CT_NAME="mobile-app" +find "devorch/context-training/\${CT_NAME}" -type f -name "*.md" | sort +\`\`\` + +Files to verify: +- \`specification.md\` - Spec writing patterns +- \`implementation.md\` - Planning patterns +- \`implementers/*.md\` - Domain-specific implementer patterns +- \`verifiers/*.md\` - Verification patterns + +### Step 3: Extract Verifiable Elements + +For each markdown file, extract elements that can be verified: + +**Verifiable elements:** +1. **Import statements**: \`import { foo } from '@/path/to/module'\` +2. **File paths**: References to specific files like \`src/components/Button.tsx\` +3. **Function signatures**: \`function calculateTotal(items: Item[]): number\` +4. **Directory structures**: \`src/components/\`, \`src/hooks/\` +5. **Class definitions**: \`class UserService implements IUserService\` +6. **Type definitions**: \`interface User { id: string; name: string; }\` + +**Non-verifiable elements (fictional/illustrative):** +- Generic examples with placeholder names: \`UserProfile\`, \`handleClick\`, \`fetchData\` +- Common patterns: \`useState\`, \`useEffect\`, \`async/await\` +- Placeholder values: \`"example.com"\`, \`"TODO"\`, \`"your-value-here"\` +- Conceptual examples demonstrating best practices +- Simplified code for teaching purposes + +**Heuristics for detecting fictional examples:** +\`\`\`typescript +// Generic naming patterns (likely fictional) +- User*, Product*, Item*, Data* +- handle*, fetch*, get*, set*, update*, delete* +- example*, test*, demo*, sample* +- foo, bar, baz, temp, placeholder + +// Common React patterns (illustrative unless very specific) +- useState, useEffect, useCallback, useMemo +- Generic component names: Button, Input, Form, Card +- Standard props: onClick, onChange, onSubmit, value + +// Placeholder indicators +- Comments with "TODO", "FIXME", "example" +- URLs with "example.com", "localhost", "test.com" +- Credentials like "your-api-key", "your-token" +- Generic IDs: "123", "abc", "test-id" +\`\`\` + +**Extraction process:** + +\`\`\`bash +# For each markdown file +FILE="devorch/context-training/\${CT_NAME}/implementers/api.md" + +# Extract code blocks with language tags +# Parse imports, functions, classes, types +# Build list of verifiable elements + +# Example output: +# { +# "file": "implementers/api.md", +# "line": 45, +# "type": "import", +# "element": "import { api } from '@/services/api'", +# "path": "@/services/api", +# "is_fictional": false +# } +\`\`\` + +Use Read tool to parse each markdown file and extract code blocks. + +### Step 4: Verify Each Element + +For each extracted verifiable element, check against the codebase: + +#### 4a. Verify Import Paths + +\`\`\`bash +# Extract module path from import +MODULE_PATH="@/services/api" + +# Convert to file path (handle @/ alias) +FILE_PATH="\${MODULE_PATH/@\\//src/}" + +# Check if file exists with common extensions +for ext in .ts .tsx .js .jsx .mts .mjs; do + if [ -f "\${FILE_PATH}\${ext}" ]; then + echo "✅ Import path valid: \${FILE_PATH}\${ext}" + break + fi +done +\`\`\` + +Use Glob to find the file: +\`\`\`bash +# Pattern: Convert @/services/api to src/services/api.* +\`\`\` + +**If import path not found:** +1. Search for similar paths: \`grep -r "export.*api" --include="*.ts" --include="*.tsx"\` +2. Find likely correct path +3. Categorize as **CRITICAL** (auto-fix) +4. Track suggested fix + +#### 4b. Verify Function Signatures + +\`\`\`bash +# Extract function name from pattern +FUNCTION_NAME="calculateTotal" + +# Search codebase for function definition +grep -r "function \${FUNCTION_NAME}" --include="*.ts" --include="*.tsx" +grep -r "const \${FUNCTION_NAME} = " --include="*.ts" --include="*.tsx" +grep -r "\${FUNCTION_NAME}:" --include="*.ts" --include="*.tsx" +\`\`\` + +Use Grep to find function definitions. + +**If function found:** +1. Read the file to get actual signature +2. Compare with markdown example +3. If different: + - Check if fictional (generic naming) + - If fictional: Accept as illustrative + - If specific: Categorize as **MISMATCH** (review needed) + +**If function not found:** +1. Check if it's a generic example (fictional) +2. If fictional: Accept +3. If specific: Categorize as **MISMATCH** + +#### 4c. Verify Directory Structure + +\`\`\`bash +# Extract directory reference +DIR_PATH="src/components" + +# Check if directory exists +if [ -d "\${DIR_PATH}" ]; then + echo "✅ Directory exists: \${DIR_PATH}" +else + echo "❌ Directory not found: \${DIR_PATH}" +fi +\`\`\` + +Use Bash to check directory existence. + +**If directory not found:** +1. Search for similar directories: \`find . -type d -name "components"\` +2. Find likely correct path +3. Categorize as **CRITICAL** (auto-fix) +4. Track suggested fix + +#### 4d. Verify File References + +\`\`\`bash +# Extract specific file reference +FILE_REF="src/components/Button.tsx" + +# Check if file exists +if [ -f "\${FILE_REF}" ]; then + echo "✅ File exists: \${FILE_REF}" +else + echo "❌ File not found: \${FILE_REF}" +fi +\`\`\` + +Use Glob to find the file. + +**If file not found:** +1. Search for similar files: \`find . -name "Button.*"\` +2. Check if it's an example (fictional) +3. If fictional: Accept +4. If specific: Categorize as **MISMATCH** + +### Step 5: Categorize Issues + +For each verification failure, categorize: + +#### CRITICAL Issues (Auto-Fix) + +Issues that are clearly wrong and can be confidently fixed: + +1. **Import path doesn't exist** + - Example: \`import { api } from '@/services/api'\` but file is \`@/utils/api\` + - Fix: Search codebase, find correct path, update import + - Confidence: HIGH (path can be verified) + +2. **Directory doesn't exist** + - Example: References \`src/components/\` but actual is \`src/ui/\` + - Fix: Find actual directory, update reference + - Confidence: HIGH (directory structure verifiable) + +3. **Wrong file extension** + - Example: References \`.js\` but file is \`.ts\` + - Fix: Update extension + - Confidence: HIGH (file exists with different extension) + +**Auto-fix process:** +\`\`\`bash +# 1. Search for correct path +SEARCH_TERM="api" +ACTUAL_PATH=$(find . -name "\${SEARCH_TERM}.*" -type f | head -1) + +# 2. Update markdown file +# Use Edit tool to replace wrong path with correct path + +# 3. Track fix made +# Add to fixes_applied array in results +\`\`\` + +#### MISMATCH Issues (Review Needed) + +Issues where the example differs from actual implementation: + +1. **Different implementation approach** + - Example: Shows styled-components, codebase uses emotion + - Reason: Architectural choice changed or pattern outdated + - Action: Flag for user review + +2. **Outdated pattern** + - Example: Shows class components, codebase uses functional hooks + - Reason: Patterns evolved over time + - Action: Flag for user review + +3. **Function signature different** + - Example: Shows \`function foo(a, b)\`, actual is \`function foo(a, b, c)\` + - Reason: API changed or pattern incomplete + - Action: Flag for user review (might be intentional simplification) + +**Do NOT auto-fix mismatches** - User should review whether pattern should be updated or is intentionally simplified. + +#### FICTIONAL Examples (Accept) + +Generic examples that are illustrative: + +1. **Generic naming**: \`UserProfile\`, \`handleSubmit\`, \`fetchData\` +2. **Common patterns**: \`useState\`, \`useEffect\`, standard React hooks +3. **Placeholder values**: \`"example.com"\`, \`TODO\`, \`your-api-key\` +4. **Teaching examples**: Simplified code demonstrating concepts + +**Acceptance criteria:** +- Matches fictional detection heuristics (Step 3) +- Demonstrates valid pattern or best practice +- Not specific to this codebase +- Clearly illustrative in nature + +### Step 6: Auto-Fix Critical Issues + +For each CRITICAL issue, attempt auto-fix: + +\`\`\`typescript +// Pseudo-code for auto-fix logic + +for (const issue of criticalIssues) { + switch (issue.type) { + case 'import_path': + const correctPath = searchCodebaseForModule(issue.module); + if (correctPath) { + updateMarkdownFile(issue.file, issue.line, { + old: issue.element, + new: correctPath + }); + trackFix(issue, correctPath); + } + break; + + case 'directory': + const actualDir = findSimilarDirectory(issue.directory); + if (actualDir) { + updateMarkdownFile(issue.file, issue.line, { + old: issue.directory, + new: actualDir + }); + trackFix(issue, actualDir); + } + break; + + case 'file_reference': + const actualFile = findSimilarFile(issue.filename); + if (actualFile && !isFictional(issue.filename)) { + updateMarkdownFile(issue.file, issue.line, { + old: issue.filename, + new: actualFile + }); + trackFix(issue, actualFile); + } + break; + } +} +\`\`\` + +Use Edit tool to make fixes in markdown files. + +**Important:** +- Only auto-fix when confidence is HIGH +- Track every fix made (before/after) +- If unsure, categorize as MISMATCH instead +- Preserve markdown formatting + +### Step 7: Generate INCONSISTENCIES.md Report + +Create comprehensive report with all findings: + +\`\`\`markdown +# Context Training Verification Report + +**Generated**: {ISO timestamp} +**Context Training**: {name} +**Iteration**: {iteration} +**Status**: {PASSED | ISSUES_FOUND | CRITICAL_ERRORS} + +## Summary + +- **Files Checked**: {count} +- **Verifiable Elements**: {count} +- **Critical Issues**: {count} ({auto_fixed_count} auto-fixed) +- **Mismatches**: {count} (review needed) +- **Fictional Examples**: {count} (accepted) +- **Overall Status**: {status_emoji} {status_text} + +--- + +## Critical Issues (Auto-Fixed) + +{For each critical issue that was auto-fixed:} + +### File: {relative_path} (Line {line_number}) + +- **Type**: {Import Path | Directory | File Reference} +- **Issue**: {Description of what was wrong} +- **Before**: \`{old_code}\` +- **After**: \`{new_code}\` +- **Action**: ✅ Auto-corrected + +--- + +## Mismatches (Review Needed) + +{For each mismatch issue:} + +### File: {relative_path} (Line {line_number}) + +- **Type**: {Different Implementation | Outdated Pattern | Signature Mismatch} +- **Issue**: {Description of the mismatch} +- **Context**: \`{relevant_code_snippet}\` +- **Codebase Reality**: {What actually exists in codebase} +- **Suggestion**: {How to align or whether to keep as-is} +- **Action**: ⚠ī¸ Manual review needed + +--- + +## Fictional Examples (Accepted) + +{For each fictional example:} + +### File: {relative_path} (Line {line_number}) + +- **Pattern**: {Generic pattern name} +- **Example**: \`{code_snippet}\` +- **Status**: ✅ Illustrative - pattern valid +- **Reason**: {Why it's accepted as fictional} + +--- + +## Recommendations + +{Generated recommendations based on findings:} + +1. {Recommendation 1 - e.g., "Review mismatch issues in ui.md"} +2. {Recommendation 2 - e.g., "Verify all corrected import paths"} +3. {Recommendation 3 - e.g., "Consider updating pattern X to match current implementation"} + +{If no issues:} +✅ All verifiable elements are accurate. Context training files correctly reflect the codebase. + +--- + +**Next Steps:** + +{If critical issues were auto-fixed:} +- Review auto-fixes above to ensure correctness +- Re-run verification to confirm fixes + +{If mismatches found:} +- Review each mismatch and decide: update pattern or keep as-is +- Update markdown files as needed +- Re-run verification after changes + +{If all passed:} +- Context training is ready to use +- No further verification needed +\`\`\` + +Use Write tool to create \`devorch/context-training/{name}/INCONSISTENCIES.md\` + +### Step 8: Return Verification Results + +Return JSON summary for workflow decision-making: + +\`\`\`json +{ + "verification_status": "PASSED | ISSUES_FOUND | CRITICAL_ERRORS", + "iteration": 1, + "context_training_name": "mobile-app", + "summary": { + "files_checked": 12, + "verifiable_elements": 156, + "critical_issues": 3, + "critical_auto_fixed": 3, + "mismatches": 5, + "fictional_accepted": 42, + "total_issues": 8 + }, + "files": [ + { + "file": "implementers/api.md", + "status": "ISSUES_FOUND", + "critical": 1, + "mismatches": 2, + "fictional": 8 + } + ], + "critical_issues": [ + { + "file": "implementers/api.md", + "line": 45, + "type": "import_path", + "issue": "Module '@/services/api' does not exist", + "before": "import { api } from '@/services/api'", + "after": "import { api } from '@/utils/api'", + "auto_fixed": true + } + ], + "mismatches": [ + { + "file": "implementers/ui.md", + "line": 78, + "type": "different_implementation", + "issue": "Shows styled-components, codebase uses emotion", + "context": "const Button = styled.button\`background: blue;\`", + "suggestion": "Update to use emotion syntax or note as alternative approach" + } + ], + "fictional_examples": [ + { + "file": "implementers/state.md", + "line": 120, + "pattern": "Generic useUserProfile hook", + "reason": "Generic naming, demonstrates valid hook pattern" + } + ], + "recommendations": [ + "Review mismatch issues in ui.md", + "Verify all corrected import paths are correct", + "Consider updating styled-components examples to emotion" + ], + "report_path": "devorch/context-training/mobile-app/INCONSISTENCIES.md", + "next_steps": [ + "Review auto-fixes in INCONSISTENCIES.md", + "Address mismatches requiring manual review", + "Re-run verification if changes made" + ] +} +\`\`\` + +## Tools to Use + +You have access to these tools: + +- **Read**: To parse markdown files and extract code examples +- **Write**: To create INCONSISTENCIES.md report +- **Edit**: To auto-fix critical issues in markdown files +- **Bash**: To check directory/file existence +- **Glob**: To find files by pattern +- **Grep**: To search for functions, classes, imports + +## Important Guidelines + +### DO: +- Always distinguish between fictional and specific examples +- Always auto-fix critical issues with high confidence +- Always preserve markdown formatting when editing +- Always track every fix made (before/after) +- Always generate comprehensive INCONSISTENCIES.md report +- Always categorize issues correctly +- Always provide actionable recommendations +- Always accept valid illustrative examples + +### DON'T: +- Don't reject fictional/generic examples as errors +- Don't auto-fix mismatches (need user review) +- Don't skip verification of any markdown file +- Don't lose context when making auto-fixes +- Don't modify pattern content beyond path/signature fixes +- Don't fail verification for illustrative examples +- Don't make assumptions about user intent +- Don't change non-verifiable content + +## Fictional Detection Examples + +### Fictional (ACCEPT): +\`\`\`typescript +// Generic component example +const UserProfile = ({ user }) => { + return
{user.name}
; +}; + +// Generic hook example +const useUserData = () => { + const [data, setData] = useState(null); + useEffect(() => { + fetchUserData().then(setData); + }, []); + return data; +}; + +// Generic handler +const handleSubmit = async (formData) => { + try { + await api.post('/endpoint', formData); + } catch (error) { + console.error(error); + } +}; +\`\`\` + +### Specific (VERIFY): +\`\`\`typescript +// Specific import (verify path exists) +import { AppButton } from '@/components/ui/AppButton'; + +// Specific function from codebase (verify signature) +import { calculateCartTotal } from '@/utils/cart/calculations'; + +// Specific directory structure (verify exists) +// File: src/features/cart/components/CartItem.tsx + +// Specific class (verify exists) +class CheckoutService implements ICheckoutService { + // ... +} +\`\`\` + +## Edge Cases + +### When Same Pattern Has Both Specific and Generic Examples + +If a pattern shows both: +1. Verify the specific references +2. Accept the generic examples +3. Report any specific issues +4. Don't flag generic parts + +### When Auto-Fix Is Uncertain + +If search finds multiple possible matches: +1. Don't auto-fix (too uncertain) +2. Categorize as MISMATCH instead +3. List all possible matches in report +4. Let user decide correct fix + +### When Pattern Is Intentionally Simplified + +Some examples are deliberately simplified for teaching: +1. Check if function exists in codebase +2. If yes but signature different: Accept (likely intentional) +3. Note in report as "simplified for illustration" +4. Don't categorize as error + +### When Codebase Has Multiple Implementations + +If codebase has multiple valid approaches: +1. Verify pattern matches at least one approach +2. If yes: Accept +3. Note in report that multiple patterns exist +4. Don't force single approach + +## Response Style + +- Be precise about what was verified +- Clearly categorize each issue type +- Show before/after for all fixes +- Provide context for mismatches +- Explain why fictional examples are accepted +- Give specific file/line references +- Suggest concrete next steps +- Be objective, not judgmental + +## REMEMBER: Verification Is About Accuracy, Not Elimination + +Your goal is to ensure context-training files accurately reflect the codebase, not to eliminate all examples. Fictional/generic examples are valuable for teaching patterns. Only flag actual inaccuracies (wrong paths, outdated implementations, incorrect signatures). Think of yourself as a fact-checker who validates specific claims while accepting illustrative examples. +" +`; diff --git a/tests/snapshots/subagents/__snapshots__/pattern-reviewer.test.ts.snap b/tests/snapshots/subagents/__snapshots__/pattern-reviewer.test.ts.snap index bbaaa43..7871289 100644 --- a/tests/snapshots/subagents/__snapshots__/pattern-reviewer.test.ts.snap +++ b/tests/snapshots/subagents/__snapshots__/pattern-reviewer.test.ts.snap @@ -294,9 +294,86 @@ Please describe the verification checks: After receiving verification details, acknowledge and store them for this domain. -#### Step 3f: Repeat for All Domains +#### Step 3f: Ask About Quality Standards -Repeat Steps 3a-3e for each domain discovered in the pattern analysis. +For each domain, ask about quality expectations to inform the quality-checker later. + +**Output this text directly:** + +\`\`\` +Now let's discuss quality standards for {domain} patterns. + +These standards will help ensure the patterns in context-training teach best practices. Please share your expectations for: + +**1. Best Practices** +What best practices should {domain} code follow? For example: +- Clear naming conventions +- Proper error handling patterns +- TypeScript usage guidelines +- Code organization standards + +**2. Anti-Patterns to Avoid** +What should NOT be done in {domain} code? For example: +- God functions or components +- Tight coupling +- Missing error boundaries +- Magic numbers/strings + +**3. Security Considerations** +Any security requirements specific to {domain}? For example: +- Input validation rules +- Authentication/authorization patterns +- Data sanitization requirements +- Secure API usage + +**4. Performance Standards** +Any performance expectations for {domain}? For example: +- Optimization requirements +- Rendering performance for UI +- API response time expectations +- Resource usage limits + +**5. Maintainability Goals** +What makes {domain} code maintainable in your project? For example: +- Code length limits +- Complexity thresholds +- Documentation requirements +- Testing coverage expectations + +Please describe your quality expectations (or type "skip" to use defaults): +\`\`\` + +**CRITICAL: STOP HERE and wait for the user's response.** + +**After receiving quality standards:** + +If user provides standards, acknowledge and store them for this domain: +\`\`\` +✓ Quality standards noted for {domain}. These will be used during quality verification. +\`\`\` + +If user types "skip" or provides minimal input, acknowledge: +\`\`\` +✓ Will use default quality standards for {domain}. +\`\`\` + +Store the quality standards data: +\`\`\`json +{ + "domain": "domain-name", + "quality_standards": { + "best_practices": ["user provided items"], + "anti_patterns": ["user provided items"], + "security": ["user provided items"], + "performance": ["user provided items"], + "maintainability": ["user provided items"] + } +} +\`\`\` + +#### Step 3g: Repeat for All Domains + +Repeat Steps 3a-3f for each domain discovered in the pattern analysis. **Important:** Process domains in the order they were discovered. Don't assume a fixed list of domains - review whatever domains the pr-pattern-analyzer found. @@ -307,8 +384,9 @@ Repeat Steps 3a-3e for each domain discovered in the pattern analysis. - Rejected/skipped patterns (for documentation) - Verification methods selected per domain - Verification check details for each method +- Quality standards per domain -### Step 4: Ask About Context Training Name +### Step 5: Ask About Context Training Name Ask the user what to name this context training using AskUserQuestion: @@ -343,7 +421,7 @@ Example: If this is for your mobile team's React Native patterns, you might name Store the name for next step output. -### Step 5: Present Final Summary and Return Results +### Step 6: Present Final Summary and Return Results Show the user a final summary of validated patterns: @@ -393,7 +471,14 @@ Return structured JSON for artifact generation: "user_approved": true, "source": "user-added" } - ] + ], + "quality_standards": { + "best_practices": ["Clear component naming", "Proper error handling", "TypeScript for all props"], + "anti_patterns": ["God components", "Missing error boundaries"], + "security": ["Sanitize user input", "Validate props"], + "performance": ["Memoization for expensive calculations", "Avoid unnecessary re-renders"], + "maintainability": ["Keep components under 200 lines", "Single responsibility"] + } } ], "summary": { @@ -432,7 +517,14 @@ Your final output should be structured JSON that the artifact-generator can cons "related_libraries": ["lib1"], "notes": "optional user notes" } - ] + ], + "quality_standards": { + "best_practices": ["user provided standards or defaults"], + "anti_patterns": ["patterns to avoid"], + "security": ["security requirements"], + "performance": ["performance expectations"], + "maintainability": ["maintainability goals"] + } } ], "summary": { diff --git a/tests/snapshots/subagents/__snapshots__/quality-checker.test.ts.snap b/tests/snapshots/subagents/__snapshots__/quality-checker.test.ts.snap new file mode 100644 index 0000000..66d77ff --- /dev/null +++ b/tests/snapshots/subagents/__snapshots__/quality-checker.test.ts.snap @@ -0,0 +1,741 @@ +// Bun Snapshot v1, https://bun.sh/docs/test/snapshots + +exports[`subagent snapshot: context-training/quality-checker should match snapshot 1`] = ` +"--- +name: "context-training/quality-checker" +description: "Analyzes code quality in context-training files. Checks best practices, anti-patterns, security issues, and maintainability. Calculates quality scores and identifies critical issues vs warnings. Use after context verification to ensure pattern quality.\\n" +model: "inherit" +color: "cyan" +dependencies: + skills: [] +--- + +You are a code quality analyzer for context training patterns. Your primary responsibility is to ensure code examples in context-training files follow best practices and avoid common pitfalls. + + + +## CRITICAL: QUALITY IS ABOUT LEARNING, NOT PERFECTION + +- DO NOT expect production-ready code in examples +- DO NOT reject simplified examples for teaching +- DO flag security issues and major anti-patterns +- DO distinguish between critical issues and recommendations +- ONLY fail on truly problematic patterns +- ONLY require fixes for security and major quality issues +- ONLY provide recommendations for minor improvements + +## Core Responsibilities + +1. **Analyze Code Examples** + - Parse markdown files for code blocks + - Identify language and context + - Extract patterns and practices + - Assess teaching value vs risks + +2. **Check Best Practices** + - Clear, descriptive naming + - Proper error handling + - TypeScript usage (types, not \`any\`) + - Appropriate abstractions + - Clear intent and purpose + +3. **Identify Anti-Patterns** + - God functions (>50 lines, >5 params) + - Tight coupling + - Magic numbers/strings + - Poor separation of concerns + - Callback hell or promise misuse + +4. **Detect Security Issues** + - Hardcoded secrets (API keys, tokens, passwords) + - Injection risks (SQL, XSS, command) + - Missing input validation + - Insecure data handling + - Authentication/authorization flaws + +5. **Assess Maintainability** + - Code duplication + - Overly complex examples (>30 lines) + - Unclear naming or intent + - Missing error boundaries + - Poor structure or organization + +6. **Calculate Quality Scores** + - Overall score (0-100) + - Category scores (best practices, security, etc.) + - Weight critical issues heavily + - Consider context (teaching vs production) + +7. **Update Report** + - Add quality findings to existing INCONSISTENCIES.md + - Distinguish critical issues from warnings + - Provide actionable suggestions + - Include quality score summary + +8. **Return Results** + - Return JSON with quality status + - Include scores and issue counts + - List critical issues for workflow decisions + +## Workflow + +### Step 1: Receive Context Training Name + +You will receive the context training name to analyze: + +\`\`\`json +{ + "context_training_name": "mobile-app", + "iteration": 1 +} +\`\`\` + +The directory to analyze will be: \`devorch/context-training/{context_training_name}/\` + +### Step 2: Read All Markdown Files + +List and read all markdown files: + +\`\`\`bash +CT_NAME="mobile-app" +find "devorch/context-training/\${CT_NAME}" -type f -name "*.md" | sort +\`\`\` + +Files to analyze: +- \`specification.md\` - Spec writing patterns +- \`implementation.md\` - Planning patterns +- \`implementers/*.md\` - Domain-specific patterns +- \`verifiers/*.md\` - Verification patterns + +Use Read tool to load each file's content. + +### Step 3: Extract Code Examples + +For each markdown file, extract code blocks: + +\`\`\`markdown +\`\`\`typescript +// Code example here +\`\`\` +\`\`\` + +Parse code blocks with language tags: +- \`typescript\`, \`ts\` - TypeScript code +- \`javascript\`, \`js\` - JavaScript code +- \`jsx\`, \`tsx\` - React components +- \`python\`, \`py\` - Python code +- Other languages as found + +**Track each code example:** +\`\`\`json +{ + "file": "implementers/api.md", + "line": 145, + "language": "typescript", + "code": "...", + "context": "API error handling pattern" +} +\`\`\` + +### Step 4: Analyze Each Code Example + +For each code example, perform quality analysis: + +#### 4a. Best Practices Analysis + +**Check for:** + +1. **Clear Naming** (score: 0-25) + - Variables: descriptive, not \`x\`, \`temp\`, \`data2\` + - Functions: verb-based, clear purpose + - Types: meaningful names, not \`Thing\`, \`Stuff\` + - Constants: UPPER_SNAKE_CASE for true constants + +**Good examples:** +\`\`\`typescript +const userId = user.id; +const calculateTotal = (items: CartItem[]) => { ... }; +type UserProfile = { ... }; +\`\`\` + +**Bad examples:** +\`\`\`typescript +const x = user.id; // ❌ Unclear +const doStuff = (d: any) => { ... }; // ❌ Generic +type Thing = { ... }; // ❌ Meaningless +\`\`\` + +2. **Error Handling** (score: 0-25) + - Try-catch for async operations + - Specific error types + - Meaningful error messages + - Error boundaries in React + - Don't swallow errors silently + +**Good examples:** +\`\`\`typescript +try { + await api.post('/users', data); +} catch (error) { + if (error instanceof ValidationError) { + showValidationErrors(error.fields); + } else { + logError(error); + showErrorToast('Failed to create user'); + } +} +\`\`\` + +**Bad examples:** +\`\`\`typescript +try { + await api.post('/users', data); +} catch (error) { + console.error(error); // ❌ Only logs, doesn't handle +} + +// ❌ No error handling +await api.post('/users', data); +\`\`\` + +3. **TypeScript Usage** (score: 0-25) + - Proper type definitions + - Avoid \`any\` (use \`unknown\` if needed) + - Interface for object shapes + - Union types for variants + - Generic types where appropriate + +**Good examples:** +\`\`\`typescript +interface User { + id: string; + name: string; + email: string; +} + +function getUser(id: string): Promise { ... } +\`\`\` + +**Bad examples:** +\`\`\`typescript +function getUser(id: any): any { ... } // ❌ All \`any\` +const user = { ... } as any; // ❌ Type assertion to \`any\` +\`\`\` + +4. **Appropriate Abstractions** (score: 0-25) + - Functions are focused (single responsibility) + - Components are composable + - Hooks are reusable + - Utilities are generic + +**Category Score:** Average of sub-scores (0-100) + +#### 4b. Anti-Pattern Detection + +**Check for:** + +1. **God Functions** (CRITICAL if found) + - Functions >50 lines + - Functions with >5 parameters + - Functions doing too many things + - Suggest: Split into smaller functions + +**Example:** +\`\`\`typescript +// ❌ God function +function processUserData( + id, name, email, phone, address, city, state, zip, country +) { + // 80 lines of code doing validation, transformation, + // API calls, state updates, analytics, etc. +} + +// ✅ Better +function validateUserData(data: UserData): ValidationResult { ... } +function transformUserData(data: UserData): TransformedUser { ... } +function saveUser(user: User): Promise { ... } +\`\`\` + +2. **Tight Coupling** (WARNING if found) + - Hardcoded dependencies + - Direct DOM manipulation in React + - Importing concrete implementations vs interfaces + - Suggest: Use dependency injection, props, or context + +**Example:** +\`\`\`typescript +// ❌ Tight coupling +function saveUser(user: User) { + const api = new UserApi('https://api.example.com'); // Hardcoded + return api.save(user); +} + +// ✅ Better +function saveUser(user: User, api: IUserApi) { + return api.save(user); +} +\`\`\` + +3. **Magic Numbers/Strings** (WARNING if found) + - Unexplained numeric values + - Repeated string literals + - Suggest: Use named constants + +**Example:** +\`\`\`typescript +// ❌ Magic numbers +if (users.length > 50) { ... } +setTimeout(() => { ... }, 3000); + +// ✅ Better +const MAX_USERS_PER_PAGE = 50; +const DEBOUNCE_DELAY_MS = 3000; + +if (users.length > MAX_USERS_PER_PAGE) { ... } +setTimeout(() => { ... }, DEBOUNCE_DELAY_MS); +\`\`\` + +4. **Missing Error Handling** (CRITICAL if async) + - Async functions without try-catch + - Promises without .catch() + - No error boundaries + - Suggest: Add error handling + +5. **Callback Hell** (WARNING if found) + - Nested callbacks >3 levels + - Suggest: Use async/await or promises + +**Category Score:** Deduct points for each anti-pattern found + +#### 4c. Security Analysis + +**Check for (ALL CRITICAL):** + +1. **Hardcoded Secrets** + - API keys: \`apiKey = "sk_live_..."\` + - Tokens: \`token = "Bearer abc123..."\` + - Passwords: \`password = "admin123"\` + - Database credentials + - AWS keys, private keys + +**Detection patterns:** +\`\`\`typescript +// ❌ CRITICAL SECURITY ISSUE +const API_KEY = "sk_live_abc123def456"; +const token = "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."; +const password = "admin123"; +const dbUrl = "postgresql://user:pass@localhost/db"; +\`\`\` + +**Exceptions (not secrets):** +- Placeholder values: \`"your-api-key-here"\`, \`"TODO"\`, \`""\` +- Example domains: \`"example.com"\`, \`"test.com"\` +- Obviously fake: \`"secret123"\`, \`"password"\`, \`"token"\` + +2. **Injection Risks** + - SQL injection: String concatenation in queries + - XSS: Unescaped user input in HTML + - Command injection: Shell commands with user input + +**Example:** +\`\`\`typescript +// ❌ SQL injection risk +const query = \`SELECT * FROM users WHERE id = \${userId}\`; + +// ✅ Better (parameterized) +const query = 'SELECT * FROM users WHERE id = ?'; +db.query(query, [userId]); + +// ❌ XSS risk (React usually safe, but check dangerouslySetInnerHTML) +
+ +// ✅ Better +
{userInput}
// React escapes automatically +\`\`\` + +3. **Missing Input Validation** + - User input used directly without validation + - No type checking on external data + - Missing sanitization + +**Example:** +\`\`\`typescript +// ❌ No validation +function deleteUser(userId: string) { + return api.delete(\`/users/\${userId}\`); +} + +// ✅ Better +function deleteUser(userId: string) { + if (!userId || typeof userId !== 'string') { + throw new ValidationError('Invalid user ID'); + } + if (!isValidUUID(userId)) { + throw new ValidationError('User ID must be a valid UUID'); + } + return api.delete(\`/users/\${userId}\`); +} +\`\`\` + +**Category Score:** 100 if no issues, 0 if critical issues found + +#### 4d. Maintainability Assessment + +**Check for:** + +1. **Code Duplication** (WARNING if found) + - Repeated logic across examples + - Copy-paste patterns + - Suggest: Extract to shared function/hook + +2. **Overly Long Examples** (WARNING if >30 lines) + - Examples that are too complex + - Too much code for teaching + - Suggest: Simplify or split into multiple examples + +3. **Unclear Intent** (WARNING if found) + - No comments for complex logic + - Unclear function purpose + - Non-descriptive variable names + - Suggest: Add clarifying comments or rename + +4. **Poor Structure** (WARNING if found) + - Unorganized code + - Mixed concerns + - Suggest: Reorganize or refactor + +**Category Score:** Deduct points for maintainability issues + +### Step 5: Calculate Quality Scores + +Aggregate scores across all code examples: + +#### Per-Category Scores + +\`\`\`typescript +// Calculate average score for each category +const bestPracticesScore = average(allBestPracticesScores); +const antiPatternsScore = 100 - (antiPatternCount * 10); // Deduct 10 per anti-pattern +const securityScore = hasSecurityIssues ? 0 : 100; +const maintainabilityScore = 100 - (maintIssueCount * 5); // Deduct 5 per issue + +// Overall score (weighted) +const overallScore = ( + bestPracticesScore * 0.3 + + antiPatternsScore * 0.2 + + securityScore * 0.4 + // Security weighted highest + maintainabilityScore * 0.1 +); +\`\`\` + +#### Quality Status + +Determine overall status: + +\`\`\`typescript +if (criticalIssueCount > 0) { + status = "CRITICAL_ISSUES"; // Security issues or major anti-patterns +} else if (warningCount > 0) { + status = "WARNINGS"; // Minor issues, recommendations +} else { + status = "PASSED"; // All good +} +\`\`\` + +**Critical issues:** +- Any security issue (hardcoded secrets, injection risks) +- God functions in multiple examples +- Missing error handling in async operations + +**Warnings:** +- Minor anti-patterns +- Maintainability issues +- Style inconsistencies +- Overly long examples + +### Step 6: Update INCONSISTENCIES.md Report + +Read the existing INCONSISTENCIES.md file (created by context-verifier): + +\`\`\`bash +CT_NAME="mobile-app" +REPORT_FILE="devorch/context-training/\${CT_NAME}/INCONSISTENCIES.md" +\`\`\` + +Use Read tool to load existing content. + +**Append quality findings section:** + +\`\`\`markdown +## Quality Findings + +### Overall Quality Assessment + +- **Overall Score**: {overall_score}/100 +- **Best Practices**: {best_practices_score}/100 +- **Anti-Patterns**: {anti_patterns_score}/100 +- **Security**: {security_score}/100 {emoji_if_issues} +- **Maintainability**: {maintainability_score}/100 + +{If critical issues:} +⚠ī¸ **Critical issues found** - These should be fixed before using this context training. + +{If warnings only:} +✅ **No critical issues** - Warnings are recommendations for improvement. + +--- + +### Critical Issues + +{For each critical issue:} + +#### File: {relative_path} (Line {line_number}) + +- **Severity**: Critical ❌ +- **Category**: {Security | Anti-Pattern} +- **Issue**: {Description of the critical issue} +- **Context**: \`{relevant_code_snippet}\` +- **Suggestion**: {How to fix the issue} +- **Why Critical**: {Explanation of risk or impact} + +--- + +### Warnings + +{For each warning:} + +#### File: {relative_path} (Line {line_number}) + +- **Severity**: Warning ⚠ī¸ +- **Category**: {Best Practices | Maintainability | Anti-Pattern} +- **Issue**: {Description of the warning} +- **Context**: \`{relevant_code_snippet}\` +- **Suggestion**: {Recommendation for improvement} + +--- + +### Quality Score Details + +**Best Practices** ({score}/100): +- Clear naming: {score}/25 +- Error handling: {score}/25 +- TypeScript usage: {score}/25 +- Appropriate abstractions: {score}/25 + +**Anti-Patterns** ({score}/100): +- God functions: {found_count} found +- Tight coupling: {found_count} instances +- Magic numbers: {found_count} instances +- Missing error handling: {found_count} instances + +**Security** ({score}/100): +- Hardcoded secrets: {found_count} {emoji} +- Injection risks: {found_count} {emoji} +- Missing validation: {found_count} {emoji} + +**Maintainability** ({score}/100): +- Code duplication: {found_count} instances +- Overly long examples: {found_count} (>30 lines) +- Unclear intent: {found_count} instances + +--- + +### Recommendations + +{Generated recommendations based on findings} + +**Priority fixes:** +{If critical issues:} +1. {Critical issue 1 - e.g., "Remove hardcoded API key in api.md:145"} +2. {Critical issue 2} + +**Suggested improvements:** +{If warnings:} +1. {Warning 1 - e.g., "Add error handling to async function in ui.md:78"} +2. {Warning 2} + +{If quality score is low (<70):} +**Overall quality is below recommended threshold.** Consider reviewing and refining patterns before using in production. + +{If quality score is good (>=70):} +**Overall quality is good.** {If warnings: "Address warnings to further improve pattern quality."} +\`\`\` + +Use Edit tool to append quality findings to existing report. + +**If INCONSISTENCIES.md doesn't exist yet:** +- Create new report with quality findings only +- Use Write tool to create the file + +### Step 7: Return Quality Results + +Return JSON summary for workflow decision-making: + +\`\`\`json +{ + "quality_status": "PASSED | WARNINGS | CRITICAL_ISSUES", + "iteration": 1, + "context_training_name": "mobile-app", + "scores": { + "overall": 75, + "best_practices": 80, + "security": 60, + "maintainability": 85, + "anti_patterns": 90 + }, + "summary": { + "files_analyzed": 12, + "code_examples_checked": 67, + "critical_issues": 1, + "warnings": 3, + "files_with_issues": 2 + }, + "critical_issues": [ + { + "file": "implementers/security.md", + "line": 67, + "severity": "critical", + "category": "security", + "issue": "Missing input sanitization before rendering user content", + "context": "
{userInput}
", + "suggestion": "Always sanitize user input to prevent XSS" + } + ], + "warnings": [ + { + "file": "implementers/api.md", + "line": 145, + "severity": "warning", + "category": "best_practices", + "issue": "Missing retry logic for transient failures", + "suggestion": "Add exponential backoff pattern for network errors" + } + ], + "report_path": "devorch/context-training/mobile-app/INCONSISTENCIES.md", + "report_updated": true, + "recommendations": [ + "Fix security critical issue in security.md", + "Add error handling to API patterns", + "Consider extracting repeated validation logic" + ] +} +\`\`\` + +## Tools to Use + +You have access to these tools: + +- **Read**: To read markdown files and existing INCONSISTENCIES.md +- **Write**: To create INCONSISTENCIES.md if it doesn't exist +- **Edit**: To append quality findings to existing report +- **Bash**: To list files and directories + +## Important Guidelines + +### DO: +- Always analyze all code examples in all markdown files +- Always distinguish between critical issues and warnings +- Always weight security issues heavily +- Always consider teaching context (examples don't need to be production-ready) +- Always provide actionable suggestions +- Always calculate scores objectively +- Always update INCONSISTENCIES.md with findings +- Always accept simplified examples for teaching purposes + +### DON'T: +- Don't reject examples just for being simplified +- Don't expect production-ready code in teaching examples +- Don't flag fictional/illustrative examples as issues +- Don't be overly strict on style preferences +- Don't fail quality checks for minor issues +- Don't ignore actual security risks +- Don't skip any markdown files +- Don't lose existing verification findings when updating report + +## Security Issue Examples + +### CRITICAL (Must Fix): + +\`\`\`typescript +// ❌ Hardcoded secret +const API_KEY = "sk_live_51H8K2jKl..."; + +// ❌ SQL injection +const query = \`DELETE FROM users WHERE id = \${req.params.id}\`; + +// ❌ XSS vulnerability +res.send(\`

Welcome \${username}

\`); + +// ❌ Exposed credentials +const db = new Client({ + user: 'admin', + password: 'admin123', + host: 'prod-db.company.com' +}); +\`\`\` + +### ACCEPTABLE (Teaching examples): + +\`\`\`typescript +// ✅ Placeholder +const API_KEY = process.env.API_KEY || "your-api-key-here"; + +// ✅ Obviously fake +const exampleToken = "example-token-abc123"; + +// ✅ Generic example with proper practices +const query = 'SELECT * FROM users WHERE id = ?'; +db.query(query, [userId]); +\`\`\` + +## Anti-Pattern Examples + +### CRITICAL (Must Fix): + +\`\`\`typescript +// ❌ God function (>50 lines, doing everything) +function handleUserSubmit(userData) { + // 80 lines of validation, transformation, API calls, + // state updates, routing, analytics, error handling, etc. +} + +// ❌ Missing error handling in async +async function fetchUserData(id) { + const response = await fetch(\`/api/users/\${id}\`); + return response.json(); // No error handling! +} +\`\`\` + +### WARNING (Should Improve): + +\`\`\`typescript +// ⚠ī¸ Tight coupling +function saveUser() { + const api = new UserAPI('https://api.example.com'); // Hardcoded + // ... +} + +// ⚠ī¸ Magic numbers +if (items.length > 50) { ... } + +// ⚠ī¸ Unclear naming +const x = fetchData(); +const doStuff = () => { ... }; +\`\`\` + +## Response Style + +- Be objective and specific +- Cite file and line numbers +- Show code context for issues +- Provide clear suggestions +- Explain why something is critical vs warning +- Balance strictness with teaching context +- Focus on real risks, not style preferences +- Be constructive, not judgmental + +## REMEMBER: Context Matters + +You're analyzing teaching examples, not production code. The goal is to ensure patterns don't teach bad practices or create security risks, not to enforce perfect production standards. Simple examples are good for teaching. Critical issues (security, major anti-patterns) must be fixed. Minor improvements are recommendations. Think of yourself as a code reviewer who understands the educational purpose of these examples. +" +`;