docs(kyberforge): add agentskillsio, agentsmd research docs and skill-write examples
- Add agentskillsio/ reference docs (8 topic files, agentskills- prefix stripped) - Add agentsmd/ reference docs (4 topic files) - Add skill-write examples: skill-creator (Anthropic), writing-great-skills (mattpocock), writing-skills (obra/superpowers) with canonical sources.md files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
100
plugins/kyberforge/docs/agentskillsio/optimizing-descriptions.md
Normal file
100
plugins/kyberforge/docs/agentskillsio/optimizing-descriptions.md
Normal file
@@ -0,0 +1,100 @@
|
||||
---
|
||||
topic: agentskills-optimizing-descriptions
|
||||
source_keys:
|
||||
- agentskills-optimizing-descriptions
|
||||
---
|
||||
|
||||
## How triggering works
|
||||
|
||||
At startup, agents load only the `name` and `description` of each available skill. When a user's task matches a description, the agent reads the full `SKILL.md` into context. The description carries the entire burden of triggering.
|
||||
|
||||
Important nuance: agents typically only consult skills for tasks that require knowledge or capabilities beyond what they can handle alone. A simple one-step request may not trigger a matching skill because the agent can handle it with basic tools. Specialized knowledge — an unfamiliar API, a domain-specific workflow, an uncommon format — is where a well-written description makes the difference.
|
||||
|
||||
## Writing effective descriptions
|
||||
|
||||
- **Use imperative phrasing.** "Use this skill when..." rather than "This skill does...". The agent is deciding whether to act, so tell it when to act.
|
||||
- **Focus on user intent, not implementation.** Describe what the user is trying to achieve, not the skill's internal mechanics.
|
||||
- **Err on the side of being pushy.** Explicitly list contexts where the skill applies, including cases where the user doesn't name the domain directly: "even if they don't explicitly mention 'CSV' or 'analysis.'"
|
||||
- **Keep it concise.** A few sentences to a short paragraph. Hard limit: 1024 characters (descriptions grow during optimization — check length).
|
||||
|
||||
## Designing trigger eval queries
|
||||
|
||||
Test triggering with a set of ~20 eval queries — realistic user prompts labeled `should_trigger: true/false`.
|
||||
|
||||
**Should-trigger queries (8–10):** vary along these axes:
|
||||
- *Phrasing*: formal, casual, abbreviations, typos
|
||||
- *Explicitness*: some name the domain directly ("analyze this CSV"), others don't ("my boss wants a chart from this data file")
|
||||
- *Detail*: terse prompts mixed with context-heavy ones (file paths, column names, backstory)
|
||||
- *Complexity*: single-step tasks alongside multi-step workflows
|
||||
|
||||
The most useful should-trigger queries are where the skill would help but the connection isn't obvious — these are where description wording makes the difference.
|
||||
|
||||
**Should-not-trigger queries (8–10):** use near-misses — queries that share keywords but need something different. Weak negative examples ("Write a fibonacci function") test nothing because there's no keyword overlap. Strong examples:
|
||||
|
||||
```
|
||||
# For a CSV analysis skill:
|
||||
"I need to update the formulas in my Excel budget spreadsheet"
|
||||
# shares "spreadsheet" concept, but needs Excel editing, not CSV analysis
|
||||
|
||||
"can you write a python script that reads a csv and uploads each row to postgres"
|
||||
# involves CSV, but the task is database ETL, not analysis
|
||||
```
|
||||
|
||||
Include realistic context in all queries: file paths, personal context ("my manager asked me to..."), specific column names, casual language.
|
||||
|
||||
## Testing trigger rates
|
||||
|
||||
Model behavior is nondeterministic. Run each query 3 times and compute a trigger rate (fraction of runs where the skill was invoked). A should-trigger query passes if its trigger rate is ≥0.5; should-not-trigger if <0.5.
|
||||
|
||||
Example script structure using Claude Code:
|
||||
|
||||
```bash
|
||||
check_triggered() {
|
||||
local query="$1"
|
||||
claude -p "$query" --output-format json 2>/dev/null \
|
||||
| jq -e --arg skill "$SKILL_NAME" \
|
||||
'any(.messages[].content[]; .type == "tool_use" and .name == "Skill" and .input.skill == $skill)' \
|
||||
> /dev/null 2>&1
|
||||
}
|
||||
```
|
||||
|
||||
## Train/validation split
|
||||
|
||||
Split your ~20 queries to avoid overfitting: ~60% train, ~40% validation. Both sets must have proportional should-trigger/should-not mixes. Use only the train set to guide changes; use the validation set only to check whether improvements generalize. Keep the split fixed across iterations.
|
||||
|
||||
## The optimization loop
|
||||
|
||||
1. Evaluate on both train and validation sets.
|
||||
2. Identify train-set failures: which should-trigger queries didn't? Which should-not-trigger queries did?
|
||||
3. Revise the description:
|
||||
- Should-trigger failures → description too narrow: broaden scope, add context about when the skill applies.
|
||||
- Should-not-trigger false positives → description too broad: add specificity about what the skill does *not* do.
|
||||
- Avoid adding specific keywords from failed queries — that's overfitting. Address the general category those queries represent.
|
||||
- If stuck after several iterations, try a structurally different framing rather than incremental tweaks.
|
||||
4. Repeat until train queries all pass or improvement stops.
|
||||
5. Select the best iteration by validation pass rate — not necessarily the last iteration.
|
||||
|
||||
Five iterations is usually enough. If not improving, the problem may be with the queries, not the description.
|
||||
|
||||
## Before and after
|
||||
|
||||
```yaml
|
||||
# Before
|
||||
description: Process CSV files.
|
||||
|
||||
# After
|
||||
description: >
|
||||
Analyze CSV and tabular data files — compute summary statistics,
|
||||
add derived columns, generate charts, and clean messy data. Use this
|
||||
skill when the user has a CSV, TSV, or Excel file and wants to
|
||||
explore, transform, or visualize the data, even if they don't
|
||||
explicitly mention "CSV" or "analysis."
|
||||
```
|
||||
|
||||
The improved version is more specific about capabilities (stats, derived columns, charts, cleaning) and broader about applicability (CSV, TSV, Excel; even without explicit keywords).
|
||||
|
||||
## Applying the result
|
||||
|
||||
1. Update the `description` field in `SKILL.md` frontmatter.
|
||||
2. Verify it's under 1024 characters.
|
||||
3. Run 5–10 fresh queries (never part of the optimization process) as a final generalization check.
|
||||
Reference in New Issue
Block a user