chore(kyberforge): remove write-eval skill
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,141 +0,0 @@
|
||||
---
|
||||
name: write-eval
|
||||
description: Write or generate an eval.yaml test file for a skill. Use when the user wants to create evals, add test coverage, or says "write evals for this skill", "create eval.yaml for X", or "add tests for this skill". Do NOT use when the user wants to run existing evals, write unit tests for code, or debug test failures.
|
||||
version: "1.0"
|
||||
updated: 2026-05-17
|
||||
when: invoked by explicit trigger ("write evals for this skill", "create eval.yaml for X") or implicit request for skill test coverage
|
||||
metadata:
|
||||
category: factory
|
||||
source:
|
||||
- repo: agentskills/agentskills
|
||||
commit: 2d3e01f590f68bee2cb76a3200823e93b2cc9eaa
|
||||
files:
|
||||
- docs/skill-creation/evaluating-skills.mdx # evals schema, two-section workspace layout, assertion quality guidelines
|
||||
updated: 2026-05-17
|
||||
- repo: darkrishabh/agent-skills-eval
|
||||
commit: b60eebe3c6edaa917a284e13b9b0e9fa00f1c957
|
||||
files:
|
||||
- src/types.ts # AgentSkillsEval interface — string-slug id, name field, prompt/expected_output/assertions structure
|
||||
- examples/basic-skill/evals/evals.json # concrete schema example
|
||||
updated: 2026-05-17
|
||||
- repo: bmad-code-org/BMAD-METHOD
|
||||
commit: 71136bc6af77cbf507d3768494311d5b6ca95cc5
|
||||
files:
|
||||
- evals/bmm-skills/bmad-product-brief/triggers.json # trigger classification dataset, should_trigger boolean pattern
|
||||
- evals/bmm-skills/bmad-product-brief/evals.json # output test structure, boundary-enforcement negative test pattern
|
||||
updated: 2026-05-17
|
||||
- repo: mattpocock/skills
|
||||
commit: e74f0061bb67222181640effa98c675bdb2fdaa7
|
||||
files:
|
||||
- skills/engineering/tdd/SKILL.md # behavioral test philosophy: test observable outputs through public interfaces
|
||||
updated: 2026-05-17
|
||||
references:
|
||||
- https://agentskills.io/skill-creation/evaluating-skills
|
||||
---
|
||||
|
||||
## Role
|
||||
|
||||
You are a test architect producing eval.yaml files that verify AI skill trigger behaviour and output quality.
|
||||
|
||||
## When to use / When not to use
|
||||
|
||||
**Use when:**
|
||||
- User explicitly requests evals: "write evals for this skill", "create eval.yaml for X", "add tests for this skill"
|
||||
- A new or refactored skill needs an eval file
|
||||
- Existing eval coverage needs to be extended with additional test cases
|
||||
|
||||
**Do not use when:**
|
||||
- User wants to run or execute existing evals
|
||||
- User wants to write unit tests for application code (not a skill eval)
|
||||
- User asks to debug or analyse failing eval results
|
||||
- User asks to review or compare eval output
|
||||
|
||||
## Required inputs
|
||||
|
||||
- Target skill name — explicit or unambiguous from session context
|
||||
- Target skill's SKILL.md — must be readable at `.agents/skills/<skill-name>/SKILL.md`
|
||||
- Target skill's `metadata.category` — used to derive the output path
|
||||
|
||||
## Constraints
|
||||
|
||||
- Output path: `.agents/evals/<category>/<skill-name>/eval.yaml` — nested by category, not flat
|
||||
- Every eval.yaml must contain all five required test types: ≥1 explicit trigger, ≥1 implicit trigger, ≥1 negative trigger, ≥2 deterministic output, ≥1 LLM-rubric quality
|
||||
- Assertions must be specific and verifiable — "The output contains a trigger_tests section" not "The output is good"
|
||||
- Assertions must be provider-agnostic — no tool-call assertions, no assumptions about the underlying model or runtime
|
||||
- Show the test plan and wait for confirmation before writing any file
|
||||
- On re-run (eval.yaml already exists): merge — classify proposed cases as NEW / IDENTICAL / CONFLICT; surface conflicts for human resolution before writing; do not silently overwrite
|
||||
- Body ≤500 lines
|
||||
|
||||
## Process
|
||||
|
||||
1. **Identify the target skill.** If not explicit in the invocation, infer from session context. If ambiguous, ask before proceeding.
|
||||
|
||||
2. **Read the target SKILL.md** at `.agents/skills/<skill-name>/SKILL.md`. Extract:
|
||||
- `name`, `metadata.category` (for output path)
|
||||
- `description` (trigger description — source for explicit and implicit trigger test queries)
|
||||
- When / when not criteria (source for negative trigger test queries)
|
||||
- Required inputs and output format (source for deterministic output assertions)
|
||||
|
||||
3. **Check for an existing eval.yaml** at `.agents/evals/<category>/<skill-name>/eval.yaml`.
|
||||
- If it exists: read it and record all existing test IDs.
|
||||
|
||||
4. **Propose test cases** — one minimum per required type:
|
||||
|
||||
**trigger_tests** — classify each query by whether the skill should activate:
|
||||
- ≥1 explicit trigger: a query using the skill's exact trigger phrase
|
||||
- ≥1 implicit trigger: a query describing the task without the trigger phrase; derive from the skill's purpose and use cases
|
||||
- ≥1 negative trigger: a query for an adjacent task the skill must NOT activate on; derive from the skill's when-not criteria; choose a case with surface similarity to the trigger
|
||||
|
||||
**output_tests** — test what the skill produces:
|
||||
- ≥2 deterministic: assert on observable, machine-checkable properties of the output — required sections present, correct file path, schema compliance. Write as specific string conditions a reader could verify without inference.
|
||||
- ≥1 LLM-rubric: holistic quality assertions — conditions a judge evaluates from the full output. Test qualities that deterministic checks cannot capture: realism of trigger queries, specificity of assertions, boundary case coverage.
|
||||
|
||||
For all assertions: write as verifiable conditions, not value judgements. Test boundary cases, not only happy paths. A good assertion survives internal refactoring of the skill.
|
||||
|
||||
5. **Classify proposed cases if an existing eval.yaml was found:**
|
||||
- **NEW** — ID not in existing file; safe to append
|
||||
- **IDENTICAL** — ID exists, content matches exactly; skip silently
|
||||
- **CONFLICT** — ID exists, content differs; display existing vs proposed side-by-side
|
||||
|
||||
6. **Present the full test plan.** Show each proposed case with its classification label (NEW / IDENTICAL / CONFLICT). For CONFLICT cases, ask the user to choose: keep existing, use proposed, or skip. Wait for confirmation before writing.
|
||||
|
||||
7. **Write eval.yaml.** Append NEW cases to the existing file (or write the full structure for a new file). Apply CONFLICT resolutions as chosen. Skip IDENTICAL cases.
|
||||
|
||||
## Output format
|
||||
|
||||
```yaml
|
||||
skill_name: <name>
|
||||
|
||||
trigger_tests:
|
||||
- id: <string-slug> # e.g. explicit-trigger-basic
|
||||
name: <display label> # human-readable, e.g. "Explicit trigger — basic invocation"
|
||||
query: <exact user input text>
|
||||
should_trigger: true # true for explicit and implicit; false for negative
|
||||
|
||||
output_tests:
|
||||
- id: <string-slug>
|
||||
name: <display label>
|
||||
type: deterministic # or llm-rubric
|
||||
prompt: <user input to the skill>
|
||||
expected_output: <prose description of ideal output>
|
||||
assertions:
|
||||
- <specific, verifiable condition string>
|
||||
```
|
||||
|
||||
## Failure handling
|
||||
|
||||
- **Target SKILL.md not found:** stop, report the path searched, do not guess or generate content from the skill name alone
|
||||
- **`metadata.category` absent from SKILL.md:** ask for the category before computing the output path
|
||||
- **All proposed cases conflict with existing file:** report the full conflict summary, wait for explicit direction — do not auto-resolve
|
||||
- **Proposed test count below minimums:** flag which type is short before presenting the plan; do not proceed with a deficient eval
|
||||
|
||||
## Self-check
|
||||
|
||||
Verify before writing:
|
||||
|
||||
- [ ] All five test types present — ≥1 explicit, ≥1 implicit, ≥1 negative trigger; ≥2 deterministic, ≥1 LLM-rubric output
|
||||
- [ ] trigger_tests: at least one `should_trigger: true` and at least one `should_trigger: false`
|
||||
- [ ] All assertions are specific and verifiable — no vague quality claims
|
||||
- [ ] Output path matches `.agents/evals/<category>/<skill-name>/eval.yaml`
|
||||
- [ ] Test plan was presented and confirmed before the file was written
|
||||
- [ ] CONFLICT cases were surfaced to the user and not silently resolved
|
||||
@@ -1,85 +0,0 @@
|
||||
skill_name: write-eval
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-trigger-write-evals
|
||||
name: "Explicit trigger — write evals"
|
||||
query: "write evals for this skill"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-trigger-create-eval-yaml
|
||||
name: "Explicit trigger — create eval.yaml"
|
||||
query: "create eval.yaml for the tdd skill"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-trigger-test-coverage
|
||||
name: "Implicit trigger — test coverage request"
|
||||
query: "I need test coverage for the grill-me skill"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-trigger-run-evals
|
||||
name: "Negative — run evals (runner concern, not writer)"
|
||||
query: "run my evals"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-code-unit-tests
|
||||
name: "Negative — unit tests for application code"
|
||||
query: "write unit tests for my Python file"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-trigger-debug-failing-eval
|
||||
name: "Negative — debug failing eval"
|
||||
query: "my eval is failing, help me debug it"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: deterministic-correct-output-path
|
||||
name: "Deterministic — eval.yaml written to correct path"
|
||||
type: deterministic
|
||||
prompt: "write evals for the tdd skill"
|
||||
expected_output: "eval.yaml written to .agents/evals/implement/tdd/eval.yaml containing skill_name: tdd"
|
||||
assertions:
|
||||
- "Output references the path .agents/evals/implement/tdd/eval.yaml"
|
||||
- "Output file contains 'skill_name: tdd'"
|
||||
|
||||
- id: deterministic-all-five-types-present
|
||||
name: "Deterministic — eval.yaml contains all five required test types"
|
||||
type: deterministic
|
||||
prompt: "create eval.yaml for the grill-me skill"
|
||||
expected_output: "eval.yaml contains trigger_tests and output_tests sections with all five required test types represented"
|
||||
assertions:
|
||||
- "Output contains 'trigger_tests:'"
|
||||
- "Output contains 'output_tests:'"
|
||||
- "Output contains at least one entry with 'should_trigger: true'"
|
||||
- "Output contains at least one entry with 'should_trigger: false'"
|
||||
- "Output contains at least one entry with 'type: deterministic'"
|
||||
- "Output contains at least one entry with 'type: llm-rubric'"
|
||||
|
||||
- id: deterministic-plan-shown-before-write
|
||||
name: "Deterministic — test plan presented before file is written"
|
||||
type: deterministic
|
||||
prompt: "write evals for the diagnose skill"
|
||||
expected_output: "Skill presents each proposed test case with its id, type, and query before writing any file, then requests confirmation"
|
||||
assertions:
|
||||
- "Response presents each proposed test case individually — showing at minimum the query and test type — before any file is written"
|
||||
- "Response requests confirmation before proceeding to write"
|
||||
|
||||
- id: deterministic-merge-conflict-flagged
|
||||
name: "Deterministic — conflict flagged in plan on re-run with existing eval"
|
||||
type: deterministic
|
||||
prompt: "write evals for the tdd skill — eval.yaml already exists at .agents/evals/implement/tdd/eval.yaml with a test case id 'explicit-trigger-basic'"
|
||||
expected_output: "Skill identifies the existing eval.yaml, classifies the conflicting case as CONFLICT, and does not write until the user resolves it"
|
||||
assertions:
|
||||
- "Response indicates eval.yaml already exists at the target path"
|
||||
- "Response labels the conflicting test case as CONFLICT or equivalent"
|
||||
- "Response does not write the file before the user resolves the conflict"
|
||||
|
||||
- id: llm-rubric-assertion-quality
|
||||
name: "LLM rubric — assertions are specific and verifiable"
|
||||
type: llm-rubric
|
||||
prompt: "write evals for the write-skill skill"
|
||||
expected_output: "eval.yaml contains high-quality assertions that are specific, observable, and not vague"
|
||||
assertions:
|
||||
- "All assertions describe observable, verifiable conditions — not vague quality claims like 'output is good' or 'the response is helpful'"
|
||||
- "Trigger test queries reflect realistic user phrasings, not just the exact skill description verbatim"
|
||||
- "Negative trigger tests target adjacent tasks that share surface-level similarity with the skill's trigger"
|
||||
- "Deterministic assertions are machine-checkable without LLM inference — presence of strings, path patterns, required sections"
|
||||
Reference in New Issue
Block a user