Defame1297
3290b93640
Move test infrastructure (validate.bats, new-skill.bats) from scripts/ to tests/ — the spec defines scripts/ as executable code agents can run, so test files don't belong there. Add tests/README.md placeholders with bats-support dependency declaration. Update skill-audit to permit tests/ and flag other unlisted directories, add scripts/ purpose check, and add /skill-improve near-miss exclusion. Update skill-improve and skill-write to cover tests/ in directory lists, scaffold template, and authoring guidance. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
75 lines
4.1 KiB
Markdown
75 lines
4.1 KiB
Markdown
---
|
|
name: skill-improve
|
|
description: >
|
|
Apply evidence-based improvements to an existing skill. Use when the user
|
|
says "improve this skill", "fix this skill based on feedback", "apply these
|
|
audit findings", or "update the skill based on this grill output". Also use
|
|
when the user provides inline feedback about a skill's behavior and wants it
|
|
applied, or when a grill session, eval run, or audit has produced findings
|
|
the user wants acted on — even if they don't say "improve" explicitly. Do
|
|
not use when the user wants a read-only review only — use /skill-audit
|
|
instead. Do not use when no improvement signals are present in the
|
|
conversation or referenced files.
|
|
allowed-tools: Bash Read Write Edit
|
|
metadata:
|
|
category: factory
|
|
---
|
|
|
|
## Gotchas
|
|
|
|
- Patching per symptom is the default failure mode. Three eval failures may all trace to one missing instruction — always identify the root cause before editing.
|
|
- Do not create new scripts unless a signal explicitly calls for it. Writing scripts from scratch requires transcript analysis that is out of scope here; flag the opportunity as a suggestion instead.
|
|
|
|
## Step 1 — Verify inputs
|
|
|
|
Confirm the skill directory path exists and that at least one improvement signal is present in the conversation or a referenced file.
|
|
|
|
If the skill dir is missing, ask for it. If no signals are present, stop: "This skill applies existing signals to a skill. For a blind review without signals, use `/skill-audit` instead."
|
|
|
|
Signals can come from anywhere in the conversation or referenced files:
|
|
- Grill session output (most common predecessor in the factory sequence)
|
|
- `/skill-audit` findings (PASS/FAIL/SUGGESTION punch list)
|
|
- Eval assertion failures (grading.json, benchmark.json)
|
|
- Human feedback (feedback.json, inline in conversation, PR or issue comments)
|
|
- Session context describing what went wrong
|
|
|
|
## Step 2 — Gather and group signals
|
|
|
|
Read the current skill files (SKILL.md and any files in scripts/, references/, assets/, tests/). Then collect all signals from the conversation and any file paths the user has referenced.
|
|
|
|
Group signals by **root cause**, not symptom. Ask: "What single gap in the skill causes this cluster of failures?" One root cause → one fix. Do not make a separate edit for each symptom.
|
|
|
|
Example:
|
|
- Eval fails because output format is wrong
|
|
- Audit finding: no output template defined
|
|
- User feedback: "I always have to ask it to format the output"
|
|
- Root cause: SKILL.md has no output format specification → one fix: add an output template
|
|
|
|
## Step 3 — Announce planned changes
|
|
|
|
Before editing, state:
|
|
- Which root causes were identified and what evidence supports each
|
|
- Which files will be changed and what will change in each
|
|
|
|
Then proceed — edits are reversible via git, no approval checkpoint needed.
|
|
|
|
## Step 4 — Apply changes
|
|
|
|
Edit any file in the skill directory that the signals point to: SKILL.md, scripts/, references/, assets/, tests/, README.md.
|
|
|
|
**Generalize, don't patch.** Find the underlying gap, not the specific example that failed. A fix scoped only to the test cases you've seen will overfit and perform worse on new inputs.
|
|
|
|
**Keep it lean.** Remove instructions that aren't pulling their weight. For every sentence you add, ask: "Would the agent get this wrong without it?" A shorter, focused skill consistently outperforms an exhaustive one.
|
|
|
|
**Explain the why.** Reasoning-based instructions outperform rigid directives. If you find yourself writing a rule in all caps (ALWAYS/NEVER), reframe it: explain why the behavior matters so the agent can apply judgment in edge cases.
|
|
|
|
**Fix the root, not the symptom.** If a signal points to a script or reference file, edit that file directly rather than adding a workaround in SKILL.md.
|
|
|
|
**On scripts**: Fix and edit existing scripts freely when signals point to them.
|
|
|
|
## Step 5 — Validate and close
|
|
|
|
Run `/skill-audit` on the skill directory. Resolve any FAIL findings before considering the improvement complete.
|
|
|
|
If the skill has no `evals/` directory, note it after the audit: "No evals found — consider running `/write-eval` to add test coverage so future improvement cycles have quantitative signals to work from."
|