holocron/plugins/kyberforge/docs/agentskillsio/evaluating-skills.md

topic, source_keys

topic	source_keys
agentskills-evaluating-skills	agentskills-evaluating-skills

Designing test cases

A test case has three parts: a prompt (realistic user message), expected output (human-readable success description), and optional input files.

Store test cases in evals/evals.json inside the skill directory:

{
  "skill_name": "csv-analyzer",
  "evals": [
    {
      "id": 1,
      "prompt": "I have a CSV of monthly sales data in data/sales_2025.csv. Can you find the top 3 months by revenue and make a bar chart?",
      "expected_output": "A bar chart showing the top 3 months by revenue, with labeled axes.",
      "files": ["evals/files/sales_2025.csv"]
    }
  ]
}

Tips for test prompts:

• Start with 2–3 test cases. Expand after seeing first results.
• Vary phrasing, formality, and detail level.
• Cover at least one edge case (malformed input, ambiguous request).
• Use realistic context: file paths, column names, personal context. "Process this data" tests nothing useful.

Don't define assertions yet — add them after seeing what the first run produces.

Workspace structure

Each iteration gets its own directory. Each test case gets with_skill/ and without_skill/ subdirectories:

csv-analyzer/
├── SKILL.md
└── evals/
    └── evals.json
csv-analyzer-workspace/
└── iteration-1/
    ├── eval-top-months-chart/
    │   ├── with_skill/
    │   │   ├── outputs/
    │   │   ├── timing.json
    │   │   └── grading.json
    │   └── without_skill/
    │       ├── outputs/
    │       ├── timing.json
    │       └── grading.json
    └── benchmark.json

The only file you author by hand is evals/evals.json. Other JSON files are produced during the eval process.

Running evals

Each run starts with a clean context — no leftover state. Run each eval with the skill and without it (or against a previous version as baseline).

When improving an existing skill, snapshot it before editing:

cp -r <skill-path> <workspace>/skill-snapshot/

Capture timing data when each run completes:

{
  "total_tokens": 84852,
  "duration_ms": 23332
}

In Claude Code, the task completion notification includes total_tokens and duration_ms — save immediately.

Writing assertions

Add assertions after seeing first-round outputs. Assertions are verifiable statements about what the output should contain:

Good assertions:

• "The output file is valid JSON" — programmatically verifiable
• "The bar chart has labeled axes" — specific and observable
• "The report includes at least 3 recommendations" — countable

Weak assertions:

• "The output is good" — too vague
• "The output uses exactly the phrase 'Total Revenue: $X'" — too brittle

Add assertions to evals.json:

"assertions": [
  "The output includes a bar chart image file",
  "The chart shows exactly 3 months",
  "Both axes are labeled",
  "The chart title or caption mentions revenue"
]

Grading outputs

Grade each assertion against actual outputs: PASS or FAIL with specific evidence. Evidence should quote or reference the output, not state an opinion.

{
  "assertion_results": [
    {
      "text": "Both axes are labeled",
      "passed": false,
      "evidence": "Y-axis is labeled 'Revenue ($)' but X-axis has no label"
    }
  ],
  "summary": { "passed": 3, "failed": 1, "total": 4, "pass_rate": 0.75 }
}

Grading principles:

• Require concrete evidence for PASS. Don't give the benefit of the doubt.
• Review the assertions themselves while grading — notice when assertions are always passing (too easy) or always failing (broken/too hard).

For holistic quality: try blind comparison — present both outputs to an LLM judge without revealing which came from which version. Complements assertion grading by catching differences that don't map to specific assertions.

Aggregating results

{
  "run_summary": {
    "with_skill": {
      "pass_rate": { "mean": 0.83, "stddev": 0.06 },
      "time_seconds": { "mean": 45.0 },
      "tokens": { "mean": 3800 }
    },
    "without_skill": {
      "pass_rate": { "mean": 0.33, "stddev": 0.10 },
      "time_seconds": { "mean": 32.0 },
      "tokens": { "mean": 2100 }
    },
    "delta": { "pass_rate": 0.50, "time_seconds": 13.0, "tokens": 1700 }
  }
}

The delta tells you what the skill costs (more time, more tokens) and what it buys (higher pass rate).

Analyzing patterns

• Remove assertions that always pass in both configurations. They inflate the with-skill pass rate without reflecting skill value.
• Investigate assertions that always fail in both. Either the assertion is broken or the task is too hard.
• Study assertions that pass with-skill but fail without. This is where the skill adds value — understand why.
• Tighten instructions when results are inconsistent. High stddev means flaky evals or ambiguous instructions. Add examples or more specific guidance.
• Check time and token outliers. If one eval takes 3× longer, read its execution transcript.

Human review

After grading, review actual outputs. Assertion grading only checks what you thought to write assertions for. Record specific feedback:

{
  "eval-top-months-chart": "Chart is missing axis labels and months are alphabetical not chronological.",
  "eval-clean-missing-emails": ""
}

"Missing axis labels" is actionable; "looks bad" is not. Empty feedback means the output looked fine.

Iterating on the skill

Three sources of signal:

• Failed assertions — specific gaps: missing step, unclear instruction, unhandled case
• Human feedback — broader quality issues: wrong approach, poorly structured output
• Execution transcripts — why things went wrong: ambiguous instructions, unproductive steps

Feed all three — with the current SKILL.md — to an LLM to propose changes. Guidelines for the LLM:

• Generalize from feedback (don't add narrow patches for specific examples)
• Keep the skill lean (fewer, better instructions outperform exhaustive rules)
• Explain the why (reasoning-based instructions work better than rigid directives)
• Bundle repeated work into scripts/ when agents reinvent the same logic each run

The loop

1. Give eval signals + current SKILL.md to an LLM; ask for improvements.
2. Review and apply the changes.
3. Rerun all test cases in a new iteration-<N+1>/ directory.
4. Grade, aggregate, and review with a human.
5. Repeat until satisfied, feedback is consistently empty, or improvement stops.