fix(kyberforge): restore skill-audit's script-failure fallback and E100 diagnostic
The ADR-0020 body trim took `skill-audit` from 2,623 body words to a dispatch shape, and two things went out with it that were not padding. The manual structural fallback was one. Its replacement was a single sentence telling the auditor to report an INFO when `validate.sh` cannot run — so with no `python3` or no PyYAML, `skill-audit` reported the gap honestly and then audited nothing structural at all. Every ADR-0020 measurement, the whole-file ceilings, the name-to-directory match, the `references/` pointer check and the script hygiene checks silently left the audit. A skill's whole Structure dimension hanging on one optional interpreter is the same vacuous-pass shape the gate scripts were just fixed for, one layer up. The `E100 Runtime error ... does not exist` diagnostic was the other. That exit code means an explicit relative `--config` was passed to `vale-wrap.sh` while vale itself was installed and working; without the note, Step 1's fallback reads exit 2 as "vale unavailable" and downgrades the description, body-discipline and patterns dimensions to full LLM judgment for a config error it could have fixed. That misreading is already recorded in CONTEXT.md as the reason both audit skills stopped passing `--config` at all. Both are restored in `references/validation-scripts.md`, loaded only when a Step 1 script fails — so the body pays nothing for them on a clean run, which is what the dispatch pattern is for. The file also carries the by-hand boundary-target procedure and the three ways to misread the result, including that `INFO ... DID NOT RUN` is not a pass. `references/file-structure.md` gains the one sanctioned spelling for a cross-skill reference. The possessive form (``skill-audit's references/validation-scripts.md``) is the only spelling both rules accept: a full repo path is what that section already forbids, and a bare `references/<file>.md` is now a hard ERROR from the ADR-0020 pointer check, which requires the file to exist in the skill's *own* directory. Without the rule the two constraints look mutually exclusive. Refs: ADR-0020
This commit is contained in:
@@ -6,10 +6,12 @@ Audit a skill directory against the agentskills.io specification and the house c
|
||||
|
||||
1. Runs `scripts/validate.sh` and `scripts/validate-provenance.sh` for structural and provenance checks, plus `scripts/vale-wrap.sh` — a Vale prefilter that deterministically flags non-imperative description openers, composition and architecture notes, vague wording, padding phrases, and "There is/are" sentence openers
|
||||
2. Reads all files in the skill directory
|
||||
3. Applies qualitative checks across six dimension groups, loading one rubric from `references/` per group
|
||||
3. Applies qualitative checks across five dimension groups, loading one rubric from `references/` per group
|
||||
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to `skill-author`
|
||||
|
||||
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words, plus resolvable boundary targets).
|
||||
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words).
|
||||
|
||||
Alongside those it runs four shape checks that are not length measurements at all. Two are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent, and every `references/<file>.md` the body names must exist on disk. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -24,7 +26,7 @@ Provide the path to the skill directory to audit when invoking.
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `SKILL.md` | Skill instructions for agents |
|
||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
||||
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
|
||||
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
|
||||
@@ -38,6 +40,7 @@ Provide the path to the skill directory to audit when invoking.
|
||||
| `references/patterns.md` | Rubric for the patterns dimension — which instruction construct fits which job, and how each is correctly formed |
|
||||
| `references/file-structure.md` | Rubric for the file-structure and internal-consistency dimensions — permitted directories, cross-plugin path rules and their two structural exemptions, README drift |
|
||||
| `references/formatting-and-scripts.md` | Rubric for the formatting and scripts dimensions — heading and fencing conventions, and the agentic-use criteria for bundled scripts |
|
||||
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded only on a script failure) |
|
||||
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
||||
| `tests/validate.bats` | (source-only) Bats test suite for validate.sh |
|
||||
| `tests/validate-provenance.bats` | (source-only) Bats test suite for validate-provenance.sh |
|
||||
|
||||
Reference in New Issue
Block a user