holocron/plugins/kyberforge/docs/agentskillsio/optimizing-descriptions.md

topic, source_keys

topic	source_keys
agentskills-optimizing-descriptions	agentskills-optimizing-descriptions

How triggering works

At startup, agents load only the name and description of each available skill. When a user's task matches a description, the agent reads the full SKILL.md into context. The description carries the entire burden of triggering.

Important nuance: agents typically only consult skills for tasks that require knowledge or capabilities beyond what they can handle alone. A simple one-step request may not trigger a matching skill because the agent can handle it with basic tools. Specialized knowledge — an unfamiliar API, a domain-specific workflow, an uncommon format — is where a well-written description makes the difference.

Writing effective descriptions

• Use imperative phrasing. "Use this skill when..." rather than "This skill does...". The agent is deciding whether to act, so tell it when to act.
• Focus on user intent, not implementation. Describe what the user is trying to achieve, not the skill's internal mechanics.
• Err on the side of being pushy. Explicitly list contexts where the skill applies, including cases where the user doesn't name the domain directly: "even if they don't explicitly mention 'CSV' or 'analysis.'"
• Keep it concise. A few sentences to a short paragraph. Hard limit: 1024 characters (descriptions grow during optimization — check length).

Designing trigger eval queries

Test triggering with a set of ~20 eval queries — realistic user prompts labeled should_trigger: true/false.

Should-trigger queries (8–10): vary along these axes:

• Phrasing: formal, casual, abbreviations, typos
• Explicitness: some name the domain directly ("analyze this CSV"), others don't ("my boss wants a chart from this data file")
• Detail: terse prompts mixed with context-heavy ones (file paths, column names, backstory)
• Complexity: single-step tasks alongside multi-step workflows

The most useful should-trigger queries are where the skill would help but the connection isn't obvious — these are where description wording makes the difference.

Should-not-trigger queries (8–10): use near-misses — queries that share keywords but need something different. Weak negative examples ("Write a fibonacci function") test nothing because there's no keyword overlap. Strong examples:

# For a CSV analysis skill:
"I need to update the formulas in my Excel budget spreadsheet"
# shares "spreadsheet" concept, but needs Excel editing, not CSV analysis

"can you write a python script that reads a csv and uploads each row to postgres"
# involves CSV, but the task is database ETL, not analysis

Include realistic context in all queries: file paths, personal context ("my manager asked me to..."), specific column names, casual language.

Testing trigger rates

Model behavior is nondeterministic. Run each query 3 times and compute a trigger rate (fraction of runs where the skill was invoked). A should-trigger query passes if its trigger rate is ≥0.5; should-not-trigger if <0.5.

Example script structure using Claude Code:

check_triggered() {
  local query="$1"
  claude -p "$query" --output-format json 2>/dev/null \
    | jq -e --arg skill "$SKILL_NAME" \
      'any(.messages[].content[]; .type == "tool_use" and .name == "Skill" and .input.skill == $skill)' \
      > /dev/null 2>&1
}

Train/validation split

Split your ~20 queries to avoid overfitting: ~60% train, ~40% validation. Both sets must have proportional should-trigger/should-not mixes. Use only the train set to guide changes; use the validation set only to check whether improvements generalize. Keep the split fixed across iterations.

The optimization loop

1. Evaluate on both train and validation sets.
2. Identify train-set failures: which should-trigger queries didn't? Which should-not-trigger queries did?
3. Revise the description:
   • Should-trigger failures → description too narrow: broaden scope, add context about when the skill applies.
   • Should-not-trigger false positives → description too broad: add specificity about what the skill does not do.
   • Avoid adding specific keywords from failed queries — that's overfitting. Address the general category those queries represent.
   • If stuck after several iterations, try a structurally different framing rather than incremental tweaks.
4. Repeat until train queries all pass or improvement stops.
5. Select the best iteration by validation pass rate — not necessarily the last iteration.

Five iterations is usually enough. If not improving, the problem may be with the queries, not the description.

Before and after

# Before
description: Process CSV files.

# After
description: >
  Analyze CSV and tabular data files — compute summary statistics,
  add derived columns, generate charts, and clean messy data. Use this
  skill when the user has a CSV, TSV, or Excel file and wants to
  explore, transform, or visualize the data, even if they don't
  explicitly mention "CSV" or "analysis."

The improved version is more specific about capabilities (stats, derived columns, charts, cleaning) and broader about applicability (CSV, TSV, Excel; even without explicit keywords).

Applying the result

1. Update the description field in SKILL.md frontmatter.
2. Verify it's under 1024 characters.
3. Run 5–10 fresh queries (never part of the optimization process) as a final generalization check.