--- topic: agentskills-evaluating-skills source_keys: - agentskills-evaluating-skills --- ## Designing test cases A test case has three parts: a **prompt** (realistic user message), **expected output** (human-readable success description), and optional **input files**. Store test cases in `evals/evals.json` inside the skill directory: ```json { "skill_name": "csv-analyzer", "evals": [ { "id": 1, "prompt": "I have a CSV of monthly sales data in data/sales_2025.csv. Can you find the top 3 months by revenue and make a bar chart?", "expected_output": "A bar chart showing the top 3 months by revenue, with labeled axes.", "files": ["evals/files/sales_2025.csv"] } ] } ``` **Tips for test prompts:** - Start with 2–3 test cases. Expand after seeing first results. - Vary phrasing, formality, and detail level. - Cover at least one edge case (malformed input, ambiguous request). - Use realistic context: file paths, column names, personal context. "Process this data" tests nothing useful. Don't define assertions yet — add them after seeing what the first run produces. ## Workspace structure Each iteration gets its own directory. Each test case gets `with_skill/` and `without_skill/` subdirectories: ``` csv-analyzer/ ├── SKILL.md └── evals/ └── evals.json csv-analyzer-workspace/ └── iteration-1/ ├── eval-top-months-chart/ │ ├── with_skill/ │ │ ├── outputs/ │ │ ├── timing.json │ │ └── grading.json │ └── without_skill/ │ ├── outputs/ │ ├── timing.json │ └── grading.json └── benchmark.json ``` The only file you author by hand is `evals/evals.json`. Other JSON files are produced during the eval process. ## Running evals Each run starts with a clean context — no leftover state. Run each eval **with the skill** and **without it** (or against a previous version as baseline). When improving an existing skill, snapshot it before editing: ```bash cp -r /skill-snapshot/ ``` Capture timing data when each run completes: ```json { "total_tokens": 84852, "duration_ms": 23332 } ``` In Claude Code, the task completion notification includes `total_tokens` and `duration_ms` — save immediately. ## Writing assertions Add assertions after seeing first-round outputs. Assertions are verifiable statements about what the output should contain: Good assertions: - `"The output file is valid JSON"` — programmatically verifiable - `"The bar chart has labeled axes"` — specific and observable - `"The report includes at least 3 recommendations"` — countable Weak assertions: - `"The output is good"` — too vague - `"The output uses exactly the phrase 'Total Revenue: $X'"` — too brittle Add assertions to `evals.json`: ```json "assertions": [ "The output includes a bar chart image file", "The chart shows exactly 3 months", "Both axes are labeled", "The chart title or caption mentions revenue" ] ``` ## Grading outputs Grade each assertion against actual outputs: PASS or FAIL with specific evidence. Evidence should quote or reference the output, not state an opinion. ```json { "assertion_results": [ { "text": "Both axes are labeled", "passed": false, "evidence": "Y-axis is labeled 'Revenue ($)' but X-axis has no label" } ], "summary": { "passed": 3, "failed": 1, "total": 4, "pass_rate": 0.75 } } ``` Grading principles: - **Require concrete evidence for PASS.** Don't give the benefit of the doubt. - **Review the assertions themselves** while grading — notice when assertions are always passing (too easy) or always failing (broken/too hard). For holistic quality: try blind comparison — present both outputs to an LLM judge without revealing which came from which version. Complements assertion grading by catching differences that don't map to specific assertions. ## Aggregating results ```json { "run_summary": { "with_skill": { "pass_rate": { "mean": 0.83, "stddev": 0.06 }, "time_seconds": { "mean": 45.0 }, "tokens": { "mean": 3800 } }, "without_skill": { "pass_rate": { "mean": 0.33, "stddev": 0.10 }, "time_seconds": { "mean": 32.0 }, "tokens": { "mean": 2100 } }, "delta": { "pass_rate": 0.50, "time_seconds": 13.0, "tokens": 1700 } } } ``` The `delta` tells you what the skill costs (more time, more tokens) and what it buys (higher pass rate). ## Analyzing patterns - **Remove assertions that always pass in both configurations.** They inflate the with-skill pass rate without reflecting skill value. - **Investigate assertions that always fail in both.** Either the assertion is broken or the task is too hard. - **Study assertions that pass with-skill but fail without.** This is where the skill adds value — understand why. - **Tighten instructions when results are inconsistent.** High `stddev` means flaky evals or ambiguous instructions. Add examples or more specific guidance. - **Check time and token outliers.** If one eval takes 3× longer, read its execution transcript. ## Human review After grading, review actual outputs. Assertion grading only checks what you thought to write assertions for. Record specific feedback: ```json { "eval-top-months-chart": "Chart is missing axis labels and months are alphabetical not chronological.", "eval-clean-missing-emails": "" } ``` "Missing axis labels" is actionable; "looks bad" is not. Empty feedback means the output looked fine. ## Iterating on the skill Three sources of signal: - **Failed assertions** — specific gaps: missing step, unclear instruction, unhandled case - **Human feedback** — broader quality issues: wrong approach, poorly structured output - **Execution transcripts** — why things went wrong: ambiguous instructions, unproductive steps Feed all three — with the current `SKILL.md` — to an LLM to propose changes. Guidelines for the LLM: - Generalize from feedback (don't add narrow patches for specific examples) - Keep the skill lean (fewer, better instructions outperform exhaustive rules) - Explain the why (reasoning-based instructions work better than rigid directives) - Bundle repeated work into `scripts/` when agents reinvent the same logic each run ### The loop 1. Give eval signals + current `SKILL.md` to an LLM; ask for improvements. 2. Review and apply the changes. 3. Rerun all test cases in a new `iteration-/` directory. 4. Grade, aggregate, and review with a human. 5. Repeat until satisfied, feedback is consistently empty, or improvement stops.