Appeng 4747/run eval per skill

[GuyZivRH]

Summary

Pack(s) affected

• rh-sre
• rh-developer
• ocp-admin
• rh-virt
• rh-ai-engineer
• Other / repo-wide

Change type

• New skill
• New agent
• New pack
• Update existing skill / agent
• MCP server config (.mcp.json)
• Docs / README
• CI / tooling

CLAUDE.md compliance

• Agents orchestrate skills; no direct MCP/tool calls in agents
• Skills are single-purpose task executors
• Skills encapsulate all tool access (MCP tools invoked only inside skills)
• Document consultation: file is read with the Read tool, then declared to the user
• No credentials hardcoded; env vars used via ${VAR} references
• Human-in-the-loop confirmation added for any destructive or critical operations

Validation

• make validate passes locally
• New/changed skills have valid YAML frontmatter (name, description)
• New/changed agents have valid YAML frontmatter (name, description)

[dmartinol]

Thanks for the great work!
My concerns:

• Most of the files are identical b/w with_skills and without_skills folders (for a given test), we should try to minimize the duplications
• I can't see instructions to run the evaluations. Are they in the other PR that you raised?

[dmartinol]

shouldn't we remove the MCP instructions from the without_skills tests?

[GuyZivRH]

so claude opus suggested to keep for fairness and also stated 2 things: 1) i did a run without those instructions and the performance was the same for the unskilled agent 2) token wise/speed this is minor

[dmartinol]

let's keep it then

[dmartinol]

why this one is not generated from within the Dockerfile as I see in other tests?
example:

RUN echo '{ \
  "mcpServers": { \
    "lightspeed-mcp": { \
      "command": "python3", \
      "args": ["/root/.mcp-servers/mock-lightspeed-mcp.py"] \
    } \
  } \
}' > /root/.mcp.json

[GuyZivRH]

will check that.

[GuyZivRH]

I agree with @dmartinol remark about duplications in here so i will elaborate:
This is 1st edition for PR for usage of the team. The presence of file (duplicates) will ease the user to replicate results if they would like to run eval. The final PR will be far more concise.
i will mark this as draft and if you still find the duplications too annoying or there is no real need to replicate the results on your end i can attend this de-duplication right away.

regarding the absence of instructions file on how to eval, I stated in the slack channel and attached the skillsbench url and an example sweep file. I can add a file here (the draft) or attach one after the de-duplication process

[GuyZivRH]

i will clarify after checking the files

Task-Specific Files (228 - CANNOT deduplicate):

• instruction.md - 45 (unique per task)
• task.toml - 45 (unique per task)
• solve.sh - 45 (unique per task)
• llm_judge.py - 45 (unique per task)
• test_outputs.py - 45 (unique per task)
• PER_SKILL_REVIEW_REPORT.md - 1
• SKILL_PATH_FIXES.md - 1
• .mcp.json - 1

Shared Files (44 - after deduplication):

• test.sh - 1 (currently 90 identical copies)
• Dockerfiles - 13 variants (currently 92 files)
• Mock MCP servers - 11 unique (currently ~120 files)
• K8s/Helm templates - 12 (currently 88 files)
  • 6 single variants (imagestream, buildconfig, values, Chart, NOTES, _helpers)
  • 6 as pairs (service×2, route×2, deployment×2)
• Systemd configs - 3 variants (currently 24 files)
• app.py - 1 (currently 2)
• test_app.py - 1 (currently 2)
• requirements.txt - 1 (currently 2)
• .s2i/environment - 1 (currently 2)

Summary:

• Current: 885 files
• Minimal: 272 files
• Savings: 613 files (~70% reduction)