feat(kyberforge): ground skill-audit qualitative rubric in agentskillsio spec
Add references/description-quality.md and references/body-discipline.md to skill-audit — condensed, rubric-focused extracts from the agentskills.io specification docs. Both files are loaded conditionally via progressive disclosure triggers added to Step 3 (Description and Body discipline dimensions), so the agent consults the spec source when a finding is borderline rather than relying solely on inline heuristics developed during the skill-write authoring cycle. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -23,4 +23,6 @@ Provide the path to the skill directory to audit when invoking.
|
||||
|------|---------|
|
||||
| `SKILL.md` | Skill instructions for agents |
|
||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||
| `references/description-quality.md` | Spec-grounded rubric for description auditing — loaded when a finding is borderline |
|
||||
| `references/body-discipline.md` | Spec-grounded rubric for body discipline auditing — loaded when padding vs necessity is unclear |
|
||||
| `README.md` | This file |
|
||||
|
||||
@@ -38,6 +38,8 @@ Work through each dimension. Cite file and line number for every finding.
|
||||
- **Near-miss exclusions**: are "Do not use when..." clauses present if a near-miss skill could steal activations?
|
||||
- **Length**: under 1024 characters?
|
||||
|
||||
If a description finding is borderline or the distinction between PASS and FAIL is unclear, read `references/description-quality.md`.
|
||||
|
||||
### Body discipline
|
||||
|
||||
For each sentence in the body, apply: *"Would the agent get this wrong without this sentence?"* Flag any that answer "no" as padding.
|
||||
@@ -46,6 +48,8 @@ For each sentence in the body, apply: *"Would the agent get this wrong without t
|
||||
- **Why rationale**: include/exclude rules explain why, not just what
|
||||
- **Control calibration**: prescriptive for fragile or critical sequences; flexible where multiple approaches are valid
|
||||
|
||||
If uncertain whether a sentence is padding or whether a control decision is correctly calibrated, read `references/body-discipline.md`.
|
||||
|
||||
### Patterns
|
||||
|
||||
Check each pattern is appropriate and correctly formed:
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# Body Discipline Reference
|
||||
|
||||
Source: agentskills.io — skill-authoring
|
||||
|
||||
## The core test
|
||||
|
||||
For every sentence in the body, ask: **"Would the agent get this wrong without this instruction?"**
|
||||
|
||||
If no — cut it. The agent already knows it from general training. Adding it wastes tokens and dilutes the signal of what matters.
|
||||
|
||||
## What belongs in the body
|
||||
|
||||
Include content the agent lacks:
|
||||
- Project-specific conventions and domain procedures it cannot infer
|
||||
- Non-obvious edge cases and environment-specific gotchas
|
||||
- The specific tools or sequences to use (not the full range of options)
|
||||
- One default per decision point with one escape hatch
|
||||
|
||||
Do not include:
|
||||
- Concepts the agent already knows (what JSON is, how HTTP works, what a CSV is)
|
||||
- Exhaustive option lists — pick a default; the agent doesn't benefit from choosing
|
||||
- Steps the agent handles independently — over-specifying leads to unproductive paths
|
||||
- Restatements of the description — it's already in context
|
||||
|
||||
## Calibrating control
|
||||
|
||||
**Be prescriptive** when operations are fragile, consistency matters, or a specific sequence must be followed:
|
||||
```markdown
|
||||
Run exactly:
|
||||
\`\`\`bash
|
||||
python scripts/migrate.py --verify --backup
|
||||
\`\`\`
|
||||
Do not modify the command or add additional flags.
|
||||
```
|
||||
|
||||
**Give freedom** when multiple approaches are valid. Explaining *why* outperforms rigid directives — agents make better decisions when they understand the purpose.
|
||||
|
||||
## Defaults not menus
|
||||
|
||||
Never present a list of equivalent options — pick one and mention the alternative briefly:
|
||||
```markdown
|
||||
# Too many options
|
||||
Use pypdf, pdfplumber, PyMuPDF, or pdf2image...
|
||||
|
||||
# Default with escape hatch
|
||||
Use pdfplumber for text extraction. For scanned PDFs requiring OCR, use pdf2image instead.
|
||||
```
|
||||
|
||||
## Gotchas sections
|
||||
|
||||
Highest value content — environment-specific facts that defy reasonable assumptions. Place near the top of the body so the agent reads them before encountering the situation.
|
||||
|
||||
```markdown
|
||||
## Gotchas
|
||||
- The `users` table uses soft deletes. Always include `WHERE deleted_at IS NULL`.
|
||||
- User ID is `user_id` in the database, `uid` in auth, `accountId` in billing. Same value.
|
||||
```
|
||||
|
||||
Each entry must be a specific, surprising fact — not a general tip or reminder.
|
||||
|
||||
## Progressive disclosure
|
||||
|
||||
Keep `SKILL.md` under 500 lines. When more content is needed, move it to `references/` and load conditionally:
|
||||
|
||||
```markdown
|
||||
If the API returns a non-200 status, read `references/api-errors.md`.
|
||||
```
|
||||
|
||||
"If X, read Y" is more useful than "see references/ for details." The agent loads on demand rather than up front.
|
||||
|
||||
## Auditing guidance
|
||||
|
||||
Flag as FAIL if:
|
||||
- A sentence answers "no" to the core test (would agent get this wrong without it?) — it is padding
|
||||
- Decision points present a menu of options with no default
|
||||
- Instructions repeat content already in the description
|
||||
- Prescriptive sequences are used where flexibility is fine, or vice versa
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
- A rationale is missing from an include/exclude rule (present but unexplained)
|
||||
- Gotchas are correct but placed late in the body rather than near the top
|
||||
- A conditional reference trigger is vague ("see references/") rather than specific ("If X, read Y")
|
||||
@@ -0,0 +1,48 @@
|
||||
# Description Quality Reference
|
||||
|
||||
Source: agentskills.io — optimizing-descriptions
|
||||
|
||||
## How triggering works
|
||||
|
||||
At startup, agents load only the `name` and `description` of each skill. When a user's task matches a description, the agent reads the full `SKILL.md` into context. **The description carries the entire triggering burden** — the body is never seen until after triggering.
|
||||
|
||||
Agents typically consult skills only for tasks requiring knowledge beyond their defaults. Specialized knowledge — unfamiliar APIs, domain-specific workflows, uncommon formats — is where description wording makes the difference.
|
||||
|
||||
## What a good description does
|
||||
|
||||
- **Imperative phrasing** — "Use when..." not "This skill does...". The agent is deciding whether to act.
|
||||
- **User intent, not mechanics** — describe what the user is trying to achieve, not how the skill works internally.
|
||||
- **Err toward being pushy** — explicitly name contexts where the skill applies, including cases where the user doesn't name the domain: "even if they don't mention X explicitly."
|
||||
- **Specificity over vagueness** — "parses and validates OpenAPI specs" beats "helps with APIs."
|
||||
- **Near-miss exclusions** — add "Do not use when..." only if a near-miss skill exists that could steal activations. Use strong near-misses (queries that share keywords but need something different), not weak ones ("write a fibonacci function").
|
||||
- **Hard limit: 1024 characters** — descriptions grow during revision; check length before finalising.
|
||||
|
||||
## Before / after
|
||||
|
||||
```yaml
|
||||
# Weak
|
||||
description: Process CSV files.
|
||||
|
||||
# Strong
|
||||
description: >
|
||||
Analyze CSV and tabular data files — compute summary statistics,
|
||||
add derived columns, generate charts, and clean messy data. Use when
|
||||
the user has a CSV, TSV, or Excel file and wants to explore, transform,
|
||||
or visualize the data, even if they don't explicitly mention "CSV" or
|
||||
"analysis."
|
||||
```
|
||||
|
||||
The strong version names capabilities precisely and broadens applicability beyond explicit keyword matches.
|
||||
|
||||
## Auditing guidance
|
||||
|
||||
Flag as FAIL if:
|
||||
- Phrasing is descriptive ("This skill...") not imperative ("Use when...")
|
||||
- Capabilities are vague ("helps with APIs") — require precise verbs and nouns
|
||||
- No indirect trigger coverage when indirect cases clearly exist
|
||||
- No near-miss exclusions when a sibling skill could plausibly steal activations
|
||||
- Length exceeds 1024 characters
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
- Indirect trigger coverage exists but could be more specific
|
||||
- Near-miss exclusions are present but target weak near-misses only
|
||||
Reference in New Issue
Block a user