holocron/plugins/kyberforge/docs/agentskillsio/evaluating-skills.md

---
topic: agentskills-evaluating-skills
source_keys:
  - agentskills-evaluating-skills
---

## Designing test cases

A test case has three parts: a **prompt** (realistic user message), **expected output** (human-readable success description), and optional **input files**.

Store test cases in `evals/evals.json` inside the skill directory:

```json
{
  "skill_name": "csv-analyzer",
  "evals": [
    {
      "id": 1,
      "prompt": "I have a CSV of monthly sales data in data/sales_2025.csv. Can you find the top 3 months by revenue and make a bar chart?",
      "expected_output": "A bar chart showing the top 3 months by revenue, with labeled axes.",
      "files": ["evals/files/sales_2025.csv"]
    }
  ]
}
```

**Tips for test prompts:**
- Start with 2–3 test cases. Expand after seeing first results.
- Vary phrasing, formality, and detail level.
- Cover at least one edge case (malformed input, ambiguous request).
- Use realistic context: file paths, column names, personal context. "Process this data" tests nothing useful.

Don't define assertions yet — add them after seeing what the first run produces.

## Workspace structure

Each iteration gets its own directory. Each test case gets `with_skill/` and `without_skill/` subdirectories:

```
csv-analyzer/
├── SKILL.md
└── evals/
    └── evals.json
csv-analyzer-workspace/
└── iteration-1/
    ├── eval-top-months-chart/
    │   ├── with_skill/
    │   │   ├── outputs/
    │   │   ├── timing.json
    │   │   └── grading.json
    │   └── without_skill/
    │       ├── outputs/
    │       ├── timing.json
    │       └── grading.json
    └── benchmark.json
```

The only file you author by hand is `evals/evals.json`. Other JSON files are produced during the eval process.

## Running evals

Each run starts with a clean context — no leftover state. Run each eval **with the skill** and **without it** (or against a previous version as baseline).

When improving an existing skill, snapshot it before editing:
```bash
cp -r <skill-path> <workspace>/skill-snapshot/
```

Capture timing data when each run completes:
```json
{
  "total_tokens": 84852,
  "duration_ms": 23332
}
```

In Claude Code, the task completion notification includes `total_tokens` and `duration_ms` — save immediately.

## Writing assertions

Add assertions after seeing first-round outputs. Assertions are verifiable statements about what the output should contain:

Good assertions:
- `"The output file is valid JSON"` — programmatically verifiable
- `"The bar chart has labeled axes"` — specific and observable
- `"The report includes at least 3 recommendations"` — countable

Weak assertions:
- `"The output is good"` — too vague
- `"The output uses exactly the phrase 'Total Revenue: $X'"` — too brittle

Add assertions to `evals.json`:

```json
"assertions": [
  "The output includes a bar chart image file",
  "The chart shows exactly 3 months",
  "Both axes are labeled",
  "The chart title or caption mentions revenue"
]
```

## Grading outputs

Grade each assertion against actual outputs: PASS or FAIL with specific evidence. Evidence should quote or reference the output, not state an opinion.

```json
{
  "assertion_results": [
    {
      "text": "Both axes are labeled",
      "passed": false,
      "evidence": "Y-axis is labeled 'Revenue ($)' but X-axis has no label"
    }
  ],
  "summary": { "passed": 3, "failed": 1, "total": 4, "pass_rate": 0.75 }
}
```

Grading principles:
- **Require concrete evidence for PASS.** Don't give the benefit of the doubt.
- **Review the assertions themselves** while grading — notice when assertions are always passing (too easy) or always failing (broken/too hard).

For holistic quality: try blind comparison — present both outputs to an LLM judge without revealing which came from which version. Complements assertion grading by catching differences that don't map to specific assertions.

## Aggregating results

```json
{
  "run_summary": {
    "with_skill": {
      "pass_rate": { "mean": 0.83, "stddev": 0.06 },
      "time_seconds": { "mean": 45.0 },
      "tokens": { "mean": 3800 }
    },
    "without_skill": {
      "pass_rate": { "mean": 0.33, "stddev": 0.10 },
      "time_seconds": { "mean": 32.0 },
      "tokens": { "mean": 2100 }
    },
    "delta": { "pass_rate": 0.50, "time_seconds": 13.0, "tokens": 1700 }
  }
}
```

The `delta` tells you what the skill costs (more time, more tokens) and what it buys (higher pass rate).

## Analyzing patterns

- **Remove assertions that always pass in both configurations.** They inflate the with-skill pass rate without reflecting skill value.
- **Investigate assertions that always fail in both.** Either the assertion is broken or the task is too hard.
- **Study assertions that pass with-skill but fail without.** This is where the skill adds value — understand why.
- **Tighten instructions when results are inconsistent.** High `stddev` means flaky evals or ambiguous instructions. Add examples or more specific guidance.
- **Check time and token outliers.** If one eval takes 3× longer, read its execution transcript.

## Human review

After grading, review actual outputs. Assertion grading only checks what you thought to write assertions for. Record specific feedback:

```json
{
  "eval-top-months-chart": "Chart is missing axis labels and months are alphabetical not chronological.",
  "eval-clean-missing-emails": ""
}
```

"Missing axis labels" is actionable; "looks bad" is not. Empty feedback means the output looked fine.

## Iterating on the skill

Three sources of signal:
- **Failed assertions** — specific gaps: missing step, unclear instruction, unhandled case
- **Human feedback** — broader quality issues: wrong approach, poorly structured output
- **Execution transcripts** — why things went wrong: ambiguous instructions, unproductive steps

Feed all three — with the current `SKILL.md` — to an LLM to propose changes. Guidelines for the LLM:
- Generalize from feedback (don't add narrow patches for specific examples)
- Keep the skill lean (fewer, better instructions outperform exhaustive rules)
- Explain the why (reasoning-based instructions work better than rigid directives)
- Bundle repeated work into `scripts/` when agents reinvent the same logic each run

### The loop

1. Give eval signals + current `SKILL.md` to an LLM; ask for improvements.
2. Review and apply the changes.
3. Rerun all test cases in a new `iteration-<N+1>/` directory.
4. Grade, aggregate, and review with a human.
5. Repeat until satisfied, feedback is consistently empty, or improvement stops.