Defame1297
fc305ba7d9
Five documents told authors that a prose-form dangling routing target blocks. The
gate reports it as a SUGGESTION and exits 0. Verified on fixtures: `-> name` and
`/name` are blocking ERRORs, the prose form is SUGGESTION-tier unless a second
resolving target in the same sentence corroborates it. ADR-0020 and gates.md were
right; contract.md, retrofit.md, description-quality.md, finding-criteria.md and
agent-author's contract.md were wrong — and they are what an author and an auditor
actually read. The whole 39-skill corpus was retrofitted against them.
skill-audit was also self-contradictory: it imports validate.sh's SUGGESTIONs into
the Structure dimension verbatim while its own rubric grades the same target a FAIL,
so one target got reported twice at two tiers. The script owns the grade; the rubric
now says so.
The YAML-fold trap that broke gitea-labels-milestones (#100) was warned about only
in retrofit.md, reachable only from the improve flow when a budget is exceeded. It
is now in both contract.md files, which SKILL.md mandates on the create flow too.
agent-audit loaded both rubrics unconditionally on every run — 3,323 words for a
clean audit against skill-audit's 1,636. dac9cad fixed exactly this in skill-audit
and edited agent-audit in the same commit without applying it. Same treatment: the
criteria move to a new finding-criteria.md and load per dimension. Clean run now
2,083 words, a 37% cut.
Routing: apm-workflow's description shed dependency installation while still owning
the flow, and apm-install's boundary did not exclude it, so "install my apm
dependencies" matched the CLI-binary skill with no route back. Fixed on both sides.
forge regains two of the three phrasings the retrofit deleted.
forge Step 1 called grill-with-docs unconditionally — a skill in plugins/bin, which
kyberforge does not declare as a dependency. It resolves here only because the
walk-up sweeps sibling plugins; a standalone install dead-ends. Step 1 now names
the cross-plugin dependency and gives an inline fallback. Declaring it properly in
apm.yml remains the better fix.
Also: both audit SKILL.md files now grade exit 2 as "did not run, dimension
unverified" rather than as findings; skill-audit's README row described content that
moved, which its own finding-criteria.md grades a FAIL; and body-discipline.md's
`git show <sha>:plugins/...` command is fenced, since an installed plugin cache has
no repo and file-structure.md makes a bare repo path a FAIL.
Refs: #100, #101, #125
ADR: 0020
54 lines
6.1 KiB
Markdown
54 lines
6.1 KiB
Markdown
# skill-audit
|
|
|
|
Audit a skill directory against the agentskills.io specification and the house context-budget contract (ADR-0020). Runs structural validation then a qualitative review across description quality, body discipline, patterns, formatting, file structure, scripts, and internal consistency, plus a provenance chain check.
|
|
|
|
## What it does
|
|
|
|
1. Runs `scripts/validate.sh` and `scripts/validate-provenance.sh` for structural and provenance checks, plus `scripts/vale-wrap.sh` — a Vale prefilter that deterministically flags non-imperative description openers, composition and architecture notes, vague wording, padding phrases, and "There is/are" sentence openers
|
|
2. Reads all files in the skill directory
|
|
3. Applies qualitative checks across five dimension groups — always loading `references/finding-criteria.md`, then one rubric from `references/` per group the criteria put in play
|
|
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to `skill-author`
|
|
|
|
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words).
|
|
|
|
Alongside those it runs four shape checks that are not length measurements at all. Two are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent, and every `references/<file>.md` the body names must exist on disk. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
|
|
|
|
## Usage
|
|
|
|
```
|
|
/skill-audit
|
|
```
|
|
|
|
Provide the path to the skill directory to audit when invoking.
|
|
|
|
## Files
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `SKILL.md` | Skill instructions for agents |
|
|
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
|
|
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
|
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
|
|
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
|
|
| `assets/vale/styles/Kyberforge/CompositionNote.yml` | Vale rule — flags composition and architecture notes in a description (e.g. "cross-cutting", "entry point", "rather than duplicating") |
|
|
| `assets/vale/styles/Kyberforge/DescriptionOpener.yml` | Vale rule — flags non-imperative "This..." description openers |
|
|
| `assets/vale/styles/Kyberforge/PaddingPhrase.yml` | Vale rule — flags generic "see references/" padding phrasing in conditional references |
|
|
| `assets/vale/styles/Kyberforge/SentenceOpenerThereIs.yml` | Vale rule — flags body sentences starting with "There is"/"There are" |
|
|
| `assets/vale/styles/Kyberforge/VagueWording.yml` | Vale rule — flags known filler wording (e.g. "helps with", "utilize") |
|
|
| `references/finding-criteria.md` | Every dimension's FAIL and SUGGESTION criteria — the one Step 3 file loaded on every run; it decides which rubrics below are worth loading |
|
|
| `references/description-quality.md` | Rubric for the description dimension — why the description is the expensive part, the hand-invoked (`disable-model-invocation`) contract, the three-part shape, when an indirect trigger is warranted, near-miss exclusions, and a before/after pair |
|
|
| `references/body-discipline.md` | Rubric for the body-discipline dimension — the core test, the 600/900 body-only budget against the 2,770-word whole-file backstop, the mandatory-dispatch rule, and the Gotchas constraints |
|
|
| `references/patterns.md` | Rubric for the patterns dimension — which instruction construct fits which job, and how each is correctly formed |
|
|
| `references/file-structure.md` | Rubric for the file-structure and internal-consistency dimensions — permitted directories, cross-plugin path rules and their two structural exemptions, README drift |
|
|
| `references/formatting-and-scripts.md` | Rubric for the formatting and scripts dimensions — heading and fencing conventions, and the agentic-use criteria for bundled scripts |
|
|
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded only on a script failure) |
|
|
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
|
| `tests/validate.bats` | (source-only) Bats test suite for validate.sh |
|
|
| `tests/validate-provenance.bats` | (source-only) Bats test suite for validate-provenance.sh |
|
|
| `tests/README.md` | (source-only) Setup instructions for bats-support and bats-assert test dependencies |
|
|
|
|
Rows marked **(source-only)** exist in the authoring source (`.apm/skills/skill-audit/`) but are
|
|
not present in an installed plugin: `scripts/sync-plugin-content.sh` strips
|
|
`<category>/<name>/tests` when it generates the flat mirror, because these are dev-time fixtures no
|
|
plugin host needs to discover (ADR-0017). Run them from a repo checkout, not from an install.
|