[syntax-error-quality] Syntax Error Quality Report: Engine validation errors lack file location; schema errors lack examples

[github-actions]

📊 Error Message Quality Analysis

Analysis Date: 2026-03-04
Test Cases: 3
Average Score: 69.7/100
Status: ⚠️ Needs Improvement (Average score 69.7 is below threshold of 70; critical consistency pattern identified)

---

Executive Summary

Three workflows were tested with different categories of syntax errors by analyzing the compiler source code pipeline end-to-end. The overall quality is near the acceptable threshold (69.7/100), but a critical consistency gap was found: engine validation errors (engine: copiilot) lack file path and line:col location information that all other error types (YAML syntax, schema validation) provide. This breaks IDE click-to-navigate and forces developers to manually hunt for the offending line.

Key Findings:

• Strengths: YAML syntax errors include source context with visual ^ pointers; schema errors include precise line locations; engine errors have excellent "Did you mean?" suggestions with examples and docs links
• Weaknesses: Engine validation errors omit file:line:col: prefix; schema constraint errors provide no valid-value examples; YAML errors translate parser jargon well but offer no corrected-syntax examples
• Critical Issue: Inconsistency between error formats — engine validation errors bypass formatCompilerError() and reach the user as plain strings without location context

---

Test Case Results

Test Case 1: Invalid YAML Syntax — Score: 78/100 ✅

Test Configuration

Source workflow: dev.md (42 lines, simple workflow)
Error Type: Category A — Frontmatter YAML syntax
Error Introduced: Line 10: engine copilot (missing colon after key)

Reconstructed Compiler Output (from source analysis)

.../test-1.md:10:8: error: Invalid YAML syntax: expected 'key: value' format (did you forget a colon after the key?)
    9 | strict: false
>  10 | engine copilot
      |        ^
```

**Pipeline traced**: `yaml.Unmarshal` fails → `FormatYAMLError()` formats with `yaml.FormatError(err, false, true)` → `adjustLineNumbersInFormattedError(offset=1)` → `translateYAMLMessage("non-map value is specified")` → `createFrontmatterError()` emits VSCode-compatible format with source context.

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 22/25 | Excellent |
| Actionability | 18/25 | Good |
| Context | 19/20 | Excellent |
| Examples | 5/15 | Poor |
| Consistency | 14/15 | Excellent |
| **Total** | **78/100** | **Good** |

#### Strengths
- ✅ `file:line:col: error:` format is IDE-parseable and VS Code–compatible
- ✅ YAML jargon translated to plain English ("non-map value is specified" → "did you forget a colon after the key?")
- ✅ Source lines shown with `^` pointer at exact error position
- ✅ Consistent format with all other YAML/schema errors

#### Weaknesses
- ⚠️ No example of correct syntax (`engine: copilot`)
- ⚠️ `^` points at column 8 (space before "copilot") instead of the start of the key `engine`
- ⚠️ No link to workflow syntax documentation

#### Improvement Suggestions

1. **Add corrected syntax example**:
   ```
   Did you mean:
   engine: copilot
   ```

2. **Adjust column pointer** to highlight the full key word instead of mid-token position

3. **Link to frontmatter reference docs** when a YAML key is unrecognised

</details>

<details>
<summary><b>Test Case 2: Invalid Engine Name</b> — Score: 64/100 ⚠️ <b>[Critical Pattern]</b></summary>

#### Test Configuration

**Source workflow**: `issue-triage-agent.md` (88 lines, medium complexity)
**Error Type**: Category B — Configuration error (typo in engine name)
**Error Introduced**: Added `engine: copiilot` (extra `i` — typo)

#### Reconstructed Compiler Output (from source analysis)

```
invalid engine: copiilot. Valid engines are: claude, codex, copilot, gemini.

Did you mean: copilot?

Example:
engine: copilot

See: https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/engines.md
```

**Pipeline traced**: YAML parses OK → `validateEngine("copiilot")` in `engine_validation.go` → finds closest match via `parser.FindClosestMatches` → builds rich error string → returned directly without `formatCompilerError()` → no `file:line:col:` prefix added.

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 18/25 | Good |
| Actionability | 23/25 | Excellent |
| Context | 4/20 | Poor |
| Examples | 14/15 | Excellent |
| Consistency | 5/15 | Poor |
| **Total** | **64/100** | **Acceptable** |

#### Strengths
- ✅ "Did you mean: copilot?" suggestion is outstanding developer experience
- ✅ Lists all valid engine names
- ✅ Provides concrete `Example: engine: copilot`
- ✅ Links directly to engine documentation

#### Weaknesses
- ❌ **No `file:line:col:` prefix** — developer cannot click to navigate in IDE
- ❌ No source code context showing which line is wrong
- ❌ Completely different visual format from Test 1 and Test 3 errors (inconsistency)
- ❌ Error bypasses `formatCompilerError()` entirely (`compiler_orchestrator_engine.go:218` returns the raw error)

#### Improvement Suggestions

1. **Wrap engine validation error through `formatCompilerError()`**:

   ```go
   // compiler_orchestrator_engine.go — around line 216-218
   if err := c.validateEngine(engineSetting); err != nil {
       orchestratorEngineLog.Printf("Engine validation failed: %v", err)
   -   return nil, err
   +   return nil, formatCompilerError(filePath, "error", err.Error(), nil)
   }
   ```
   This would produce:
   ```
   .github/workflows/issue-triage-agent.md:9:1: error: invalid engine: copiilot. Valid engines are: ...

   Did you mean: copilot?

   Example:
   engine: copilot
   ```

2. **Pass `filePath` into the orchestrator engine function** so the error can be location-tagged

</details>

<details>
<summary><b>Test Case 3: Negative Timeout Value</b> — Score: 67/100 ⚠️</summary>

#### Test Configuration

**Source workflow**: `repository-quality-improver.md` (568 lines, complex)
**Error Type**: Category C — Semantic / out-of-range value
**Error Introduced**: Changed `timeout-minutes: 20` to `timeout-minutes: -10`

#### Reconstructed Compiler Output (from source analysis)

```
.../test-3.md:31:1: error: must be at least 1 (got -10)
  29 |     max: 1
  30 |     close-older-discussions: true
  31 | timeout-minutes: -10
     | ^^^^^^^^^^^^^^^
  32 | strict: true

Pipeline traced: YAML parses OK → validateWithSchemaAndLocation() in schema_compiler.go → jsonschema minimum: got -10, want 1 → translateSchemaConstraintMessage() → "must be at least 1 (got -10)" → LocateJSONPathInYAMLWithAdditionalProperties finds timeout-minutes at line 31 → console.FormatError() renders with source context.

Evaluation Scores

Dimension	Score	Rating
Clarity	20/25	Good
Actionability	14/25	Minimal
Context	16/20	Good
Examples	4/15	Poor
Consistency	13/15	Good
Total	67/100	Acceptable

Strengths

• ✅ file:line:col: error: format with working IDE navigation
• ✅ "must be at least 1 (got -10)" clearly states both constraint and actual value
• ✅ Source context with ^ underline of the key name
• ✅ Consistent with schema error format

Weaknesses

• ⚠️ "must be at least 1" says the minimum but not a practical range or common value
• ⚠️ No example: timeout-minutes: 30 would immediately clarify
• ⚠️ No documentation link to timeout configuration reference

Improvement Suggestions

1. Add practical example to range constraint errors:

   must be at least 1 (got -10)
   
   Example:
   timeout-minutes: 30   # 30 minutes is a common choice
   

2. Enhance translateSchemaConstraintMessage() in schema_errors.go to include field-contextual examples when a constraint is violated

3. Add a documentation link to the frontmatter reference for range-constrained fields

---

Overall Statistics

Metric	Value
Tests Run	3
Average Score	69.7/100
Good (70-84)	1
Acceptable (55-69)	2
Poor (<55)	0

Quality Assessment: ⚠️ Needs Improvement — Average score 69.7 is just below the 70 threshold. One test case (engine validation) has a critical format inconsistency. Two test cases (engine, timeout) lack examples.

---

Priority Improvement Recommendations

🔴 High Priority — Engine Validation Error Lacks File Location

Problem: validateEngine() in engine_validation.go returns a plain string error that is returned directly from compiler_orchestrator_engine.go (line 218) without going through formatCompilerError(). All other error types (YAML parse errors, schema constraint errors) include file:line:col: error: prefix. Engine errors do not.

Impact: Developer cannot click to navigate in VS Code/JetBrains. Must manually search the file for engine:.

Fix location: pkg/workflow/compiler_orchestrator_engine.go ~line 216–218

// Current (no location info):
if err := c.validateEngine(engineSetting); err != nil {
    return nil, err
}

// Suggested (adds file:line:col prefix):
if err := c.validateEngine(engineSetting); err != nil {
    return nil, formatCompilerError(workflowData.MarkdownPath, "error", err.Error(), nil)
}

🟡 Medium Priority — Schema Constraint Errors Lack Examples

Problem: Range constraint errors like must be at least 1 (got -10) do not include a practical example of a valid value.

Fix location: pkg/parser/schema_errors.go → translateSchemaConstraintMessage() or formatSchemaFailureDetail() in schema_compiler.go

Suggested enhancement:

// In translateSchemaConstraintMessage, add example hint:
return fmt.Sprintf("must be at least %s (got %s)\n\nExample:\n%s: %s", m[2], m[1], fieldName, m[2])

🟡 Medium Priority — YAML Syntax Errors Lack Corrected Syntax Examples

Problem: When YAML syntax errors are translated (e.g., "did you forget a colon after the key?"), no corrected-syntax example is shown.

Fix location: pkg/workflow/frontmatter_error.go → createFrontmatterError() — append a generic hint after the source context.

---

Implementation Files

File	Change
pkg/workflow/compiler_orchestrator_engine.go	Wrap validateEngine error in formatCompilerError()
pkg/parser/schema_errors.go	Enhance translateSchemaConstraintMessage() with field examples
pkg/workflow/frontmatter_error.go	Add corrected-syntax hint after YAML source context

---

Success Metrics

1. All three error types produce file:line:col: error: prefix (currently Test 2 does not)
2. Range constraint errors include at least one valid-value example
3. YAML syntax error "missing colon" hint includes corrected form

---

References:

• Workflow run §22682258339
• Analyzed files: pkg/workflow/engine_validation.go, pkg/workflow/frontmatter_error.go, pkg/parser/schema_compiler.go, pkg/parser/schema_errors.go, pkg/console/console.go

Generated by Daily Syntax Error Quality Check · ◷

• expires on Mar 7, 2026, 6:13 PM UTC