holocron/.agents/skills/write-eval/SKILL.md

---
name: write-eval
description: Write or generate an eval.yaml test file for a skill. Use when the user wants to create evals, add test coverage, or says "write evals for this skill", "create eval.yaml for X", or "add tests for this skill". Do NOT use when the user wants to run existing evals, write unit tests for code, or debug test failures.
version: "1.0"
updated: 2026-05-17
when: invoked by explicit trigger ("write evals for this skill", "create eval.yaml for X") or implicit request for skill test coverage
metadata:
  category: factory
source:
  - repo: agentskills/agentskills
    commit: 2d3e01f590f68bee2cb76a3200823e93b2cc9eaa
    files:
      - docs/skill-creation/evaluating-skills.mdx  # evals schema, two-section workspace layout, assertion quality guidelines
    updated: 2026-05-17
  - repo: darkrishabh/agent-skills-eval
    commit: b60eebe3c6edaa917a284e13b9b0e9fa00f1c957
    files:
      - src/types.ts  # AgentSkillsEval interface — string-slug id, name field, prompt/expected_output/assertions structure
      - examples/basic-skill/evals/evals.json  # concrete schema example
    updated: 2026-05-17
  - repo: bmad-code-org/BMAD-METHOD
    commit: 71136bc6af77cbf507d3768494311d5b6ca95cc5
    files:
      - evals/bmm-skills/bmad-product-brief/triggers.json  # trigger classification dataset, should_trigger boolean pattern
      - evals/bmm-skills/bmad-product-brief/evals.json  # output test structure, boundary-enforcement negative test pattern
    updated: 2026-05-17
  - repo: mattpocock/skills
    commit: e74f0061bb67222181640effa98c675bdb2fdaa7
    files:
      - skills/engineering/tdd/SKILL.md  # behavioral test philosophy: test observable outputs through public interfaces
    updated: 2026-05-17
references:
  - https://agentskills.io/skill-creation/evaluating-skills
---

## Role

You are a test architect producing eval.yaml files that verify AI skill trigger behaviour and output quality.

## When to use / When not to use

**Use when:**
- User explicitly requests evals: "write evals for this skill", "create eval.yaml for X", "add tests for this skill"
- A new or refactored skill needs an eval file
- Existing eval coverage needs to be extended with additional test cases

**Do not use when:**
- User wants to run or execute existing evals
- User wants to write unit tests for application code (not a skill eval)
- User asks to debug or analyse failing eval results
- User asks to review or compare eval output

## Required inputs

- Target skill name — explicit or unambiguous from session context
- Target skill's SKILL.md — must be readable at `.agents/skills/<skill-name>/SKILL.md`
- Target skill's `metadata.category` — used to derive the output path

## Constraints

- Output path: `.agents/evals/<category>/<skill-name>/eval.yaml` — nested by category, not flat
- Every eval.yaml must contain all five required test types: ≥1 explicit trigger, ≥1 implicit trigger, ≥1 negative trigger, ≥2 deterministic output, ≥1 LLM-rubric quality
- Assertions must be specific and verifiable — "The output contains a trigger_tests section" not "The output is good"
- Assertions must be provider-agnostic — no tool-call assertions, no assumptions about the underlying model or runtime
- Show the test plan and wait for confirmation before writing any file
- On re-run (eval.yaml already exists): merge — classify proposed cases as NEW / IDENTICAL / CONFLICT; surface conflicts for human resolution before writing; do not silently overwrite
- Body ≤500 lines

## Process

1. **Identify the target skill.** If not explicit in the invocation, infer from session context. If ambiguous, ask before proceeding.

2. **Read the target SKILL.md** at `.agents/skills/<skill-name>/SKILL.md`. Extract:
   - `name`, `metadata.category` (for output path)
   - `description` (trigger description — source for explicit and implicit trigger test queries)
   - When / when not criteria (source for negative trigger test queries)
   - Required inputs and output format (source for deterministic output assertions)

3. **Check for an existing eval.yaml** at `.agents/evals/<category>/<skill-name>/eval.yaml`.
   - If it exists: read it and record all existing test IDs.

4. **Propose test cases** — one minimum per required type:

   **trigger_tests** — classify each query by whether the skill should activate:
   - ≥1 explicit trigger: a query using the skill's exact trigger phrase
   - ≥1 implicit trigger: a query describing the task without the trigger phrase; derive from the skill's purpose and use cases
   - ≥1 negative trigger: a query for an adjacent task the skill must NOT activate on; derive from the skill's when-not criteria; choose a case with surface similarity to the trigger

   **output_tests** — test what the skill produces:
   - ≥2 deterministic: assert on observable, machine-checkable properties of the output — required sections present, correct file path, schema compliance. Write as specific string conditions a reader could verify without inference.
   - ≥1 LLM-rubric: holistic quality assertions — conditions a judge evaluates from the full output. Test qualities that deterministic checks cannot capture: realism of trigger queries, specificity of assertions, boundary case coverage.

   For all assertions: write as verifiable conditions, not value judgements. Test boundary cases, not only happy paths. A good assertion survives internal refactoring of the skill.

5. **Classify proposed cases if an existing eval.yaml was found:**
   - **NEW** — ID not in existing file; safe to append
   - **IDENTICAL** — ID exists, content matches exactly; skip silently
   - **CONFLICT** — ID exists, content differs; display existing vs proposed side-by-side

6. **Present the full test plan.** Show each proposed case with its classification label (NEW / IDENTICAL / CONFLICT). For CONFLICT cases, ask the user to choose: keep existing, use proposed, or skip. Wait for confirmation before writing.

7. **Write eval.yaml.** Append NEW cases to the existing file (or write the full structure for a new file). Apply CONFLICT resolutions as chosen. Skip IDENTICAL cases.

## Output format

```yaml
skill_name: <name>

trigger_tests:
  - id: <string-slug>       # e.g. explicit-trigger-basic
    name: <display label>   # human-readable, e.g. "Explicit trigger — basic invocation"
    query: <exact user input text>
    should_trigger: true    # true for explicit and implicit; false for negative

output_tests:
  - id: <string-slug>
    name: <display label>
    type: deterministic     # or llm-rubric
    prompt: <user input to the skill>
    expected_output: <prose description of ideal output>
    assertions:
      - <specific, verifiable condition string>
```

## Failure handling

- **Target SKILL.md not found:** stop, report the path searched, do not guess or generate content from the skill name alone
- **`metadata.category` absent from SKILL.md:** ask for the category before computing the output path
- **All proposed cases conflict with existing file:** report the full conflict summary, wait for explicit direction — do not auto-resolve
- **Proposed test count below minimums:** flag which type is short before presenting the plan; do not proceed with a deficient eval

## Self-check

Verify before writing:

- [ ] All five test types present — ≥1 explicit, ≥1 implicit, ≥1 negative trigger; ≥2 deterministic, ≥1 LLM-rubric output
- [ ] trigger_tests: at least one `should_trigger: true` and at least one `should_trigger: false`
- [ ] All assertions are specific and verifiable — no vague quality claims
- [ ] Output path matches `.agents/evals/<category>/<skill-name>/eval.yaml`
- [ ] Test plan was presented and confirmed before the file was written
- [ ] CONFLICT cases were surfaced to the user and not silently resolved