Defame1297
fc305ba7d9
Five documents told authors that a prose-form dangling routing target blocks. The
gate reports it as a SUGGESTION and exits 0. Verified on fixtures: `-> name` and
`/name` are blocking ERRORs, the prose form is SUGGESTION-tier unless a second
resolving target in the same sentence corroborates it. ADR-0020 and gates.md were
right; contract.md, retrofit.md, description-quality.md, finding-criteria.md and
agent-author's contract.md were wrong — and they are what an author and an auditor
actually read. The whole 39-skill corpus was retrofitted against them.
skill-audit was also self-contradictory: it imports validate.sh's SUGGESTIONs into
the Structure dimension verbatim while its own rubric grades the same target a FAIL,
so one target got reported twice at two tiers. The script owns the grade; the rubric
now says so.
The YAML-fold trap that broke gitea-labels-milestones (#100) was warned about only
in retrofit.md, reachable only from the improve flow when a budget is exceeded. It
is now in both contract.md files, which SKILL.md mandates on the create flow too.
agent-audit loaded both rubrics unconditionally on every run — 3,323 words for a
clean audit against skill-audit's 1,636. dac9cad fixed exactly this in skill-audit
and edited agent-audit in the same commit without applying it. Same treatment: the
criteria move to a new finding-criteria.md and load per dimension. Clean run now
2,083 words, a 37% cut.
Routing: apm-workflow's description shed dependency installation while still owning
the flow, and apm-install's boundary did not exclude it, so "install my apm
dependencies" matched the CLI-binary skill with no route back. Fixed on both sides.
forge regains two of the three phrasings the retrofit deleted.
forge Step 1 called grill-with-docs unconditionally — a skill in plugins/bin, which
kyberforge does not declare as a dependency. It resolves here only because the
walk-up sweeps sibling plugins; a standalone install dead-ends. Step 1 now names
the cross-plugin dependency and gives an inline fallback. Declaring it properly in
apm.yml remains the better fix.
Also: both audit SKILL.md files now grade exit 2 as "did not run, dimension
unverified" rather than as findings; skill-audit's README row described content that
moved, which its own finding-criteria.md grades a FAIL; and body-discipline.md's
`git show <sha>:plugins/...` command is fenced, since an installed plugin cache has
no repo and file-structure.md makes a bare repo path a FAIL.
Refs: #100, #101, #125
ADR: 0020
87 lines
5.3 KiB
Markdown
87 lines
5.3 KiB
Markdown
---
|
|
name: skill-audit
|
|
description: >
|
|
Use when the user wants a skill directory audited against the agentskills.io
|
|
spec — "audit this skill", "review my SKILL.md", "is this ready to ship" — or
|
|
after hand-editing a skill outside skill-author. Not applying fixes ->
|
|
skill-author.
|
|
allowed-tools: Bash Read
|
|
metadata:
|
|
category: factory
|
|
source_keys:
|
|
- agentskills-home
|
|
- agentskills-spec
|
|
- agentskills-best-practices
|
|
- agentskills-optimizing-descriptions
|
|
- agentskills-using-scripts
|
|
---
|
|
|
|
## Gotchas
|
|
|
|
- Do not narrate PASS/FAIL per check while auditing. Gather findings internally and surface them only in the Step 4 report. Narrating each check as you go is the default failure mode here.
|
|
- A skill carrying `disable-model-invocation: true` is hand-invoked — its description is never routed against, so the trigger, capability and boundary rules do not apply. Audit it as one plain human-facing sentence instead.
|
|
- `validate.sh` reports two independent length families: the 500-line / 2,770-word pair counts the whole file for spec conformance, while the 250/400-character and 600/900-word pair is the house context budget and its word half counts the **body only**. A skill can sit inside one and fail the other — report them separately.
|
|
- Vale reporting `0 files` scanned means NOT RUN, not clean. Fall back to full Step 3 judgment for every dimension it would have covered.
|
|
|
|
## Step 1 — Deterministic checks
|
|
|
|
Resolve all three paths against this skill's own directory so they work from a repo checkout and an installed plugin cache alike. Run exactly:
|
|
|
|
```bash
|
|
bash scripts/validate.sh <skill-dir>
|
|
bash scripts/validate-provenance.sh <skill-dir>
|
|
bash scripts/vale-wrap.sh <skill-dir>/SKILL.md
|
|
```
|
|
|
|
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both, at the tier the script assigned. Report each once; never re-grade one under another dimension. Unresolved boundary targets are where this bites, because their tier turns on notation.
|
|
|
|
If any of the three cannot run, or exits non-zero for a reason other than findings, read `references/validation-scripts.md` — it carries the manual fallback and the misleading exit codes. Ordinary content FAILs are the expected outcome here and need no fallback.
|
|
|
|
`validate-provenance.sh` prints nothing on success, so read its exit code before you read its silence. **0** is a genuine pass. **1** means real findings: its FAILs and INFOs become a separate `### Provenance` dimension, and it emits Why and Fix itself — surface those verbatim. **2** means the check never ran — a usage or environment error, reason on stderr, no findings and often no stdout at all. On a 2, report `### Provenance` as unverified and quote the stderr reason. Never grade an exit 2 as a clean pass: empty stdout there means nothing was checked, not that nothing was wrong.
|
|
|
|
`vale-wrap.sh` applies the bundled `Kyberforge` style as a prefilter. Pass no `--config`; the wrapper locates its own. Every rule is graded `error`, so every alert is a FAIL. Report each one citing its rule ID, filed under the dimension it belongs to, and do not re-derive it by judgment:
|
|
|
|
| Rule | Dimension |
|
|
|---|---|
|
|
| `Kyberforge.DescriptionOpener`, `Kyberforge.CompositionNote`, `Kyberforge.VagueWording` | description |
|
|
| `Kyberforge.SentenceOpenerThereIs` | body-discipline |
|
|
| `Kyberforge.PaddingPhrase` | patterns |
|
|
|
|
## Step 2 — Read the whole skill
|
|
|
|
Read `SKILL.md`, `README.md`, and every text file under `scripts/`, `references/`, `assets/` and `tests/`. Skip binaries only — internal-consistency findings need the full picture.
|
|
|
|
## Step 3 — Qualitative audit
|
|
|
|
Read `references/finding-criteria.md` first — every dimension's FAIL and SUGGESTION criteria. Load the rubric below only for a dimension the criteria put in play: one carrying a candidate finding, or one where the criterion alone does not settle the call.
|
|
|
|
| Dimension | Rubric |
|
|
|---|---|
|
|
| description | `references/description-quality.md` |
|
|
| body-discipline | `references/body-discipline.md` |
|
|
| patterns | `references/patterns.md` |
|
|
| file-structure, internal-consistency | `references/file-structure.md` |
|
|
| formatting, scripts | `references/formatting-and-scripts.md` |
|
|
|
|
Each rubric is self-contained and grounded in the agentskills.io specification plus the house context budget (ADR-0020). Cite file and line number for every finding.
|
|
|
|
## Step 4 — Report
|
|
|
|
Open with a coverage line naming every dimension checked:
|
|
|
|
```text
|
|
Checked: structure · description · body-discipline · patterns · file-structure · formatting · scripts · internal-consistency · provenance
|
|
```
|
|
|
|
Then output only the dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each. Omit clean dimensions — their absence is what confirms they passed.
|
|
|
|
Each finding:
|
|
|
|
```text
|
|
FAIL/SUGGESTION <finding> — file:line
|
|
Why: <why this is a problem>
|
|
Fix: <exact change — quote before/after where applicable>
|
|
```
|
|
|
|
Close with a `## Result` block holding one line: `PASS`, `PASS (N suggestions)`, or `FAIL (N fails · M suggestions)`, each optionally followed by ` · P info`. INFO findings are observational and never change PASS/FAIL; omit `· P info` when there are none. Add a second line, `Run skill-author to address findings.`, whenever there is at least one finding. Do not apply fixes — report and propose only.
|