holocron/plugins/kyberforge/skills/write-eval/SKILL.md

name, description, version, updated, when, metadata, source, references

name	description	version	updated	when	metadata	source	references
write-eval	Write or generate an eval.yaml test file for a skill. Use when the user wants to create evals, add test coverage, or says "write evals for this skill", "create eval.yaml for X", or "add tests for this skill". Do NOT use when the user wants to run existing evals, write unit tests for code, or debug test failures.	1.0	2026-05-17	invoked by explicit trigger ("write evals for this skill", "create eval.yaml for X") or implicit request for skill test coverage	category factory	repo commit files updated agentskills/agentskills 2d3e01f590f68bee2cb76a3200823e93b2cc9eaa docs/skill-creation/evaluating-skills.mdx 2026-05-17 repo commit files updated darkrishabh/agent-skills-eval b60eebe3c6edaa917a284e13b9b0e9fa00f1c957 src/types.ts examples/basic-skill/evals/evals.json 2026-05-17 repo commit files updated bmad-code-org/BMAD-METHOD 71136bc6af77cbf507d3768494311d5b6ca95cc5 evals/bmm-skills/bmad-product-brief/triggers.json evals/bmm-skills/bmad-product-brief/evals.json 2026-05-17 repo commit files updated mattpocock/skills e74f0061bb67222181640effa98c675bdb2fdaa7 skills/engineering/tdd/SKILL.md 2026-05-17	https://agentskills.io/skill-creation/evaluating-skills

Role

You are a test architect producing eval.yaml files that verify AI skill trigger behaviour and output quality.

When to use / When not to use

Use when:

• User explicitly requests evals: "write evals for this skill", "create eval.yaml for X", "add tests for this skill"
• A new or refactored skill needs an eval file
• Existing eval coverage needs to be extended with additional test cases

Do not use when:

• User wants to run or execute existing evals
• User wants to write unit tests for application code (not a skill eval)
• User asks to debug or analyse failing eval results
• User asks to review or compare eval output

Required inputs

• Target skill name — explicit or unambiguous from session context
• Target skill's SKILL.md — must be readable at .agents/skills/<skill-name>/SKILL.md
• Target skill's metadata.category — used to derive the output path

Constraints

• Output path: .agents/evals/<category>/<skill-name>/eval.yaml — nested by category, not flat
• Every eval.yaml must contain all five required test types: ≥1 explicit trigger, ≥1 implicit trigger, ≥1 negative trigger, ≥2 deterministic output, ≥1 LLM-rubric quality
• Assertions must be specific and verifiable — "The output contains a trigger_tests section" not "The output is good"
• Assertions must be provider-agnostic — no tool-call assertions, no assumptions about the underlying model or runtime
• Show the test plan and wait for confirmation before writing any file
• On re-run (eval.yaml already exists): merge — classify proposed cases as NEW / IDENTICAL / CONFLICT; surface conflicts for human resolution before writing; do not silently overwrite
• Body ≤500 lines

Process

1. Identify the target skill. If not explicit in the invocation, infer from session context. If ambiguous, ask before proceeding.

2. Read the target SKILL.md at .agents/skills/<skill-name>/SKILL.md. Extract:

   • name, metadata.category (for output path)
   • description (trigger description — source for explicit and implicit trigger test queries)
   • When / when not criteria (source for negative trigger test queries)
   • Required inputs and output format (source for deterministic output assertions)

3. Check for an existing eval.yaml at .agents/evals/<category>/<skill-name>/eval.yaml.

   • If it exists: read it and record all existing test IDs.

4. Propose test cases — one minimum per required type:

   trigger_tests — classify each query by whether the skill should activate:

   • ≥1 explicit trigger: a query using the skill's exact trigger phrase
   • ≥1 implicit trigger: a query describing the task without the trigger phrase; derive from the skill's purpose and use cases
   • ≥1 negative trigger: a query for an adjacent task the skill must NOT activate on; derive from the skill's when-not criteria; choose a case with surface similarity to the trigger

   output_tests — test what the skill produces:

   • ≥2 deterministic: assert on observable, machine-checkable properties of the output — required sections present, correct file path, schema compliance. Write as specific string conditions a reader could verify without inference.
   • ≥1 LLM-rubric: holistic quality assertions — conditions a judge evaluates from the full output. Test qualities that deterministic checks cannot capture: realism of trigger queries, specificity of assertions, boundary case coverage.

   For all assertions: write as verifiable conditions, not value judgements. Test boundary cases, not only happy paths. A good assertion survives internal refactoring of the skill.

5. Classify proposed cases if an existing eval.yaml was found:

   • NEW — ID not in existing file; safe to append
   • IDENTICAL — ID exists, content matches exactly; skip silently
   • CONFLICT — ID exists, content differs; display existing vs proposed side-by-side

6. Present the full test plan. Show each proposed case with its classification label (NEW / IDENTICAL / CONFLICT). For CONFLICT cases, ask the user to choose: keep existing, use proposed, or skip. Wait for confirmation before writing.

7. Write eval.yaml. Append NEW cases to the existing file (or write the full structure for a new file). Apply CONFLICT resolutions as chosen. Skip IDENTICAL cases.

Output format

skill_name: <name>

trigger_tests:
  - id: <string-slug>       # e.g. explicit-trigger-basic
    name: <display label>   # human-readable, e.g. "Explicit trigger — basic invocation"
    query: <exact user input text>
    should_trigger: true    # true for explicit and implicit; false for negative

output_tests:
  - id: <string-slug>
    name: <display label>
    type: deterministic     # or llm-rubric
    prompt: <user input to the skill>
    expected_output: <prose description of ideal output>
    assertions:
      - <specific, verifiable condition string>

Failure handling

• Target SKILL.md not found: stop, report the path searched, do not guess or generate content from the skill name alone
• metadata.category absent from SKILL.md: ask for the category before computing the output path
• All proposed cases conflict with existing file: report the full conflict summary, wait for explicit direction — do not auto-resolve
• Proposed test count below minimums: flag which type is short before presenting the plan; do not proceed with a deficient eval

Self-check

Verify before writing:

• All five test types present — ≥1 explicit, ≥1 implicit, ≥1 negative trigger; ≥2 deterministic, ≥1 LLM-rubric output
• trigger_tests: at least one should_trigger: true and at least one should_trigger: false
• All assertions are specific and verifiable — no vague quality claims
• Output path matches .agents/evals/<category>/<skill-name>/eval.yaml
• Test plan was presented and confirmed before the file was written
• CONFLICT cases were surfaced to the user and not silently resolved