fix(kyberforge): restore skill-audit's script-failure fallback and E100 diagnostic
The ADR-0020 body trim took `skill-audit` from 2,623 body words to a dispatch shape, and two things went out with it that were not padding. The manual structural fallback was one. Its replacement was a single sentence telling the auditor to report an INFO when `validate.sh` cannot run — so with no `python3` or no PyYAML, `skill-audit` reported the gap honestly and then audited nothing structural at all. Every ADR-0020 measurement, the whole-file ceilings, the name-to-directory match, the `references/` pointer check and the script hygiene checks silently left the audit. A skill's whole Structure dimension hanging on one optional interpreter is the same vacuous-pass shape the gate scripts were just fixed for, one layer up. The `E100 Runtime error ... does not exist` diagnostic was the other. That exit code means an explicit relative `--config` was passed to `vale-wrap.sh` while vale itself was installed and working; without the note, Step 1's fallback reads exit 2 as "vale unavailable" and downgrades the description, body-discipline and patterns dimensions to full LLM judgment for a config error it could have fixed. That misreading is already recorded in CONTEXT.md as the reason both audit skills stopped passing `--config` at all. Both are restored in `references/validation-scripts.md`, loaded only when a Step 1 script fails — so the body pays nothing for them on a clean run, which is what the dispatch pattern is for. The file also carries the by-hand boundary-target procedure and the three ways to misread the result, including that `INFO ... DID NOT RUN` is not a pass. `references/file-structure.md` gains the one sanctioned spelling for a cross-skill reference. The possessive form (``skill-audit's references/validation-scripts.md``) is the only spelling both rules accept: a full repo path is what that section already forbids, and a bare `references/<file>.md` is now a hard ERROR from the ADR-0020 pointer check, which requires the file to exist in the skill's *own* directory. Without the rule the two constraints look mutually exclusive. Refs: ADR-0020
This commit is contained in:
@@ -6,10 +6,12 @@ Audit a skill directory against the agentskills.io specification and the house c
|
||||
|
||||
1. Runs `scripts/validate.sh` and `scripts/validate-provenance.sh` for structural and provenance checks, plus `scripts/vale-wrap.sh` — a Vale prefilter that deterministically flags non-imperative description openers, composition and architecture notes, vague wording, padding phrases, and "There is/are" sentence openers
|
||||
2. Reads all files in the skill directory
|
||||
3. Applies qualitative checks across six dimension groups, loading one rubric from `references/` per group
|
||||
3. Applies qualitative checks across five dimension groups, loading one rubric from `references/` per group
|
||||
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to `skill-author`
|
||||
|
||||
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words, plus resolvable boundary targets).
|
||||
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words).
|
||||
|
||||
Alongside those it runs four shape checks that are not length measurements at all. Two are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent, and every `references/<file>.md` the body names must exist on disk. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -24,7 +26,7 @@ Provide the path to the skill directory to audit when invoking.
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `SKILL.md` | Skill instructions for agents |
|
||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
||||
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
|
||||
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
|
||||
@@ -38,6 +40,7 @@ Provide the path to the skill directory to audit when invoking.
|
||||
| `references/patterns.md` | Rubric for the patterns dimension — which instruction construct fits which job, and how each is correctly formed |
|
||||
| `references/file-structure.md` | Rubric for the file-structure and internal-consistency dimensions — permitted directories, cross-plugin path rules and their two structural exemptions, README drift |
|
||||
| `references/formatting-and-scripts.md` | Rubric for the formatting and scripts dimensions — heading and fencing conventions, and the agentic-use criteria for bundled scripts |
|
||||
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded only on a script failure) |
|
||||
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
||||
| `tests/validate.bats` | (source-only) Bats test suite for validate.sh |
|
||||
| `tests/validate-provenance.bats` | (source-only) Bats test suite for validate-provenance.sh |
|
||||
|
||||
@@ -33,7 +33,9 @@ bash scripts/validate-provenance.sh <skill-dir>
|
||||
scripts/vale-wrap.sh <skill-dir>/SKILL.md
|
||||
```
|
||||
|
||||
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both. If it cannot run at all (no `python3`, Bash denied), report that as an INFO finding rather than guessing; what it measures is not reproducible by reading.
|
||||
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both.
|
||||
|
||||
If any of the three fails, cannot run, or reports something needing interpretation, read `references/validation-scripts.md` — it carries the manual fallback and the misleading exit codes.
|
||||
|
||||
`validate-provenance.sh` prints nothing on success. Its FAIL and INFO findings become a separate `### Provenance` dimension, and it emits Why and Fix itself — surface those verbatim.
|
||||
|
||||
|
||||
@@ -28,6 +28,14 @@ resolving there. Flag any `../`, `../../`, or absolute repo path (`plugins/<plug
|
||||
and its APM-native equivalent `.apm/skills/<other>/`) appearing in `SKILL.md`, `scripts/`,
|
||||
`references/` or `assets/`.
|
||||
|
||||
**Referring to another skill's file.** There is one sanctioned spelling, and it is possessive:
|
||||
`skill-audit's references/validation-scripts.md`. Write the skill by name and let the reader
|
||||
resolve it — do not spell the repo path. The full path is the thing this section forbids, and
|
||||
`references/validation-scripts.md` on its own is a hard ERROR from the ADR-0020 gate, which
|
||||
requires an unqualified `references/` pointer to exist in the skill's OWN directory. The
|
||||
possessive form is the only spelling both rules accept; the gate recognises it and skips the
|
||||
on-disk check. Flag any other spelling of a cross-skill reference.
|
||||
|
||||
Two directories are exempt, and the exemptions are structural rather than discretionary:
|
||||
|
||||
- **`references/sources.md`.** Its `Research doc:` fields are development-time provenance pointers,
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
- **URL:** https://agentskills.io/specification.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** Complete SKILL.md format specification — frontmatter fields, constraints, body content, optional directories, progressive disclosure levels, file references, validation
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md, references/patterns.md, references/file-structure.md, references/formatting-and-scripts.md
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md, references/patterns.md, references/file-structure.md, references/formatting-and-scripts.md, references/validation-scripts.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-best-practices
|
||||
@@ -47,7 +47,7 @@
|
||||
- **URL:** https://agentskills.io/skill-creation/using-scripts.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** Using scripts in skills — one-off commands, self-contained scripts with inline dependencies, designing scripts for agentic use (no interactive prompts, --help, structured output, idempotency)
|
||||
- **Contributing files:** SKILL.md, references/formatting-and-scripts.md
|
||||
- **Contributing files:** SKILL.md, references/formatting-and-scripts.md, references/validation-scripts.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-quickstart
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
source_keys:
|
||||
- agentskills-spec
|
||||
- agentskills-using-scripts
|
||||
---
|
||||
|
||||
# Validation Scripts Reference
|
||||
|
||||
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting.
|
||||
Nothing here is needed on a clean run.
|
||||
|
||||
## Report the gap, do not guess
|
||||
|
||||
If a script cannot run at all — Bash denied, `python3` unavailable, PyYAML not importable, `vale`
|
||||
not installed — say so as an **INFO** finding naming the script and the missing dependency, then
|
||||
fall back to the manual checks below. An INFO never changes PASS/FAIL. Silently omitting the
|
||||
dimension a script would have covered reports a clean audit that checked less than it claims to
|
||||
have checked, and the Step 4 coverage line then names a dimension nothing actually examined.
|
||||
|
||||
## Manual structural fallback
|
||||
|
||||
`validate.sh` needs `python3` **and** PyYAML, and refuses to start without either — the description
|
||||
value has to be measured after YAML folding is resolved, so skipping the ADR-0020 gates would be a
|
||||
vacuous pass rather than a partial one. The two are checked separately, so the message already names
|
||||
the right one — report it verbatim rather than diagnosing further:
|
||||
|
||||
```text
|
||||
Error: python3 is required but was not found on PATH.
|
||||
Error: PyYAML is required but is not importable by python3.
|
||||
```
|
||||
|
||||
Without them — or with Bash denied, or on a permission error — work this list
|
||||
by hand and file the results under `### Structure` exactly as the script's output would have been:
|
||||
|
||||
- **`name`** present, 1–64 characters, kebab-case (lowercase letters, digits and hyphens; no
|
||||
leading, trailing or doubled hyphen), and **matching the skill's directory name** exactly.
|
||||
- **`description`** present and non-empty; no unfilled `FILL IN:` placeholder in it. An absent or
|
||||
empty description is a **FAIL**, never a silent skip — it is the one field preloaded into every
|
||||
session, so a skill without one can never be routed to.
|
||||
- **Description length**, measured on the folded YAML value with newlines collapsed to single
|
||||
spaces — not on the raw block scalar, which counts indentation. 250 characters SUGGESTION, 400
|
||||
FAIL (ADR-0020), 1,024 FAIL (agentskills.io spec).
|
||||
- **Body length**, counting everything after the frontmatter's closing `---`. 600 words
|
||||
SUGGESTION, 900 FAIL (ADR-0020).
|
||||
- **Whole-file ceilings**, counting the file including frontmatter: 500 lines FAIL, 2,770 words
|
||||
FAIL (agentskills.io spec). These are a different measurement from the two above — report them
|
||||
as separate findings, never merged.
|
||||
- **A boundary clause is present** — either the prose form (`do not` / `instead` / `rather than` /
|
||||
`not for`) or ADR-0020's compressed `Not <thing> -> <name>` arrow. **SUGGESTION**, not FAIL:
|
||||
the absence is deterministic, but whether this skill warrants one is the auditor's call.
|
||||
- **Boundary targets resolve** — **FAIL** on a name that resolves to nothing. See the section
|
||||
below; resolving these by hand is the one item on this list with a procedure of its own.
|
||||
- **Every `references/<file>.md` named in the body exists on disk** — **FAIL**, not a suggestion.
|
||||
A dispatch table or "read X" trigger naming a missing file sends the agent nowhere. Ignore
|
||||
mentions inside fenced code blocks, and ignore a mention whose own line says the file is gone
|
||||
(`removed`, `deleted`, `renamed`, `superseded`, `replaced`, `obsolete`, `deprecated`, `former`,
|
||||
`gone`, `no longer`, `used to`) — that is a historical note, not a dispatch entry.
|
||||
- **Gotchas discipline**, both **SUGGESTION**. Locate the section by a heading that *is* Gotchas
|
||||
(`## Common Gotchas` counts; `## Gotcha handling` and `## Why gotchas matter` do not), running to
|
||||
the next heading at the same level or shallower. More than five top-level entries is one
|
||||
suggestion; a section over 25% of the body word count is a second, independent one. Count
|
||||
entries at column 0 only — an indented child bullet is not an entry — and ignore fenced code
|
||||
blocks for both.
|
||||
- **No unfilled `FILL IN:` placeholder** anywhere in the body.
|
||||
- **Every file in `scripts/`** carries the executable bit and contains no interactive prompt —
|
||||
no bare `read`, no `select`, nothing that blocks on a TTY.
|
||||
|
||||
## Resolving boundary targets by hand
|
||||
|
||||
Targets are read from **both** boundary forms. The compressed `Not <thing> -> <name>` arrow and the
|
||||
prose form are each parsed *and* target-checked, so a typo in prose phrasing fails exactly as an
|
||||
arrow typo does — do not check only the names after an arrow.
|
||||
|
||||
Build the universe by walking up **from the `SKILL.md` under audit**, never from the validator's own
|
||||
location. The nearest ancestor holding `plugins/*/.apm/skills/` or `plugins/*/.apm/agents/` is the
|
||||
authoring root, falling back to the nearest ancestor holding `.git`. When one is found the universe
|
||||
is every skill and agent under `<root>/plugins/*/`, plus the skill's own apm package, plus the
|
||||
packages that package declares in its `apm.yml` under `dependencies.apm`. Deployed `.claude/` and
|
||||
`.agents/` trees are consulted **only** when no authoring root exists — they are gitignored
|
||||
`apm install` output, and reading them would make a fresh clone and a developer machine disagree.
|
||||
|
||||
Three ways to read the result wrong:
|
||||
|
||||
- **A hyphenated name used attributively is not a dangling target.** "Use pre-commit hooks instead
|
||||
of ad-hoc scripts" reads as a route to `pre-commit` on wording alone. What separates a route from
|
||||
prose is grammar: a route target is terminal — followed by punctuation, a conjunction, or a
|
||||
boundary word — whereas a compound modifier is followed by the noun it modifies. A name followed
|
||||
by an ordinary noun still *confirms* a route when it exists, but never raises a FAIL on its own.
|
||||
- **A SUGGESTION-tier unresolved target is not a FAIL you may promote.** Terminal position alone is
|
||||
not evidence of a route: "run `pre-commit` instead", "see `commit-msg`" and "use the clean-up
|
||||
instead" are all terminal and all prose. A prose-form target earns a FAIL only when its own
|
||||
sentence names another target that *does* resolve; otherwise the script reports it and moves on,
|
||||
and so should you. Route notation — `/name` and `-> name` — is exempt and always FAILs, and it is
|
||||
the fix to recommend when the author did mean a route.
|
||||
- **`INFO boundary-target resolution DID NOT RUN` is not a pass.** The script prints it, and exits
|
||||
0, when no universe could be determined for that path — the usual cause being a skill copy
|
||||
audited outside its package. Report it as an INFO naming the unchecked targets and re-run against
|
||||
the real directory; filing it as clean signs off targets nothing verified.
|
||||
|
||||
## Script-specific failures
|
||||
|
||||
- **`validate-provenance.sh` printed nothing.** That is a pass, not a skip. It also exits 0
|
||||
silently when the skill has no `source_keys` and no `references/sources.md` — nothing to
|
||||
validate is not a finding.
|
||||
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
||||
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
|
||||
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
|
||||
- **`E100 Runtime error ... does not exist` (exit 2) from `vale-wrap.sh`.** An explicit relative
|
||||
`--config` was passed. Pass none: the wrapper locates its own `assets/vale/.vale.ini` from its
|
||||
own path, so a resolved script path plus an unresolved config path produces exactly this. Do not
|
||||
read this exit code as vale being unavailable — that misreading sends the audit down the
|
||||
fallback path while vale was installed and working the whole time.
|
||||
- **The `vale` binary is genuinely absent** (`command not found`). Report one INFO naming it, then
|
||||
fall back to full Step 3 judgment for the description, body-discipline and patterns dimensions —
|
||||
the prefilter's whole coverage. Judge those by rubric rather than dropping them.
|
||||
- **A path argument that does not exist is a hard error** in `vale-wrap.sh`, deliberately: bare
|
||||
`vale` would fall back to reading stdin and print a clean-looking `0 errors ... in stdin`, which
|
||||
the `0 files` guard above does not catch.
|
||||
@@ -6,10 +6,12 @@ Audit a skill directory against the agentskills.io specification and the house c
|
||||
|
||||
1. Runs `scripts/validate.sh` and `scripts/validate-provenance.sh` for structural and provenance checks, plus `scripts/vale-wrap.sh` — a Vale prefilter that deterministically flags non-imperative description openers, composition and architecture notes, vague wording, padding phrases, and "There is/are" sentence openers
|
||||
2. Reads all files in the skill directory
|
||||
3. Applies qualitative checks across six dimension groups, loading one rubric from `references/` per group
|
||||
3. Applies qualitative checks across five dimension groups, loading one rubric from `references/` per group
|
||||
4. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix — and a result block with handoff to `skill-author`
|
||||
|
||||
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words, plus resolvable boundary targets).
|
||||
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words).
|
||||
|
||||
Alongside those it runs four shape checks that are not length measurements at all. Two are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent, and every `references/<file>.md` the body names must exist on disk. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -24,7 +26,7 @@ Provide the path to the skill directory to audit when invoking.
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `SKILL.md` | Skill instructions for agents |
|
||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description length, line count, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
|
||||
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
|
||||
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
|
||||
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
|
||||
@@ -38,6 +40,7 @@ Provide the path to the skill directory to audit when invoking.
|
||||
| `references/patterns.md` | Rubric for the patterns dimension — which instruction construct fits which job, and how each is correctly formed |
|
||||
| `references/file-structure.md` | Rubric for the file-structure and internal-consistency dimensions — permitted directories, cross-plugin path rules and their two structural exemptions, README drift |
|
||||
| `references/formatting-and-scripts.md` | Rubric for the formatting and scripts dimensions — heading and fencing conventions, and the agentic-use criteria for bundled scripts |
|
||||
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded only on a script failure) |
|
||||
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
|
||||
| `tests/validate.bats` | (source-only) Bats test suite for validate.sh |
|
||||
| `tests/validate-provenance.bats` | (source-only) Bats test suite for validate-provenance.sh |
|
||||
|
||||
@@ -33,7 +33,9 @@ bash scripts/validate-provenance.sh <skill-dir>
|
||||
scripts/vale-wrap.sh <skill-dir>/SKILL.md
|
||||
```
|
||||
|
||||
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both. If it cannot run at all (no `python3`, Bash denied), report that as an INFO finding rather than guessing; what it measures is not reproducible by reading.
|
||||
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both.
|
||||
|
||||
If any of the three fails, cannot run, or reports something needing interpretation, read `references/validation-scripts.md` — it carries the manual fallback and the misleading exit codes.
|
||||
|
||||
`validate-provenance.sh` prints nothing on success. Its FAIL and INFO findings become a separate `### Provenance` dimension, and it emits Why and Fix itself — surface those verbatim.
|
||||
|
||||
|
||||
@@ -28,6 +28,14 @@ resolving there. Flag any `../`, `../../`, or absolute repo path (`plugins/<plug
|
||||
and its APM-native equivalent `.apm/skills/<other>/`) appearing in `SKILL.md`, `scripts/`,
|
||||
`references/` or `assets/`.
|
||||
|
||||
**Referring to another skill's file.** There is one sanctioned spelling, and it is possessive:
|
||||
`skill-audit's references/validation-scripts.md`. Write the skill by name and let the reader
|
||||
resolve it — do not spell the repo path. The full path is the thing this section forbids, and
|
||||
`references/validation-scripts.md` on its own is a hard ERROR from the ADR-0020 gate, which
|
||||
requires an unqualified `references/` pointer to exist in the skill's OWN directory. The
|
||||
possessive form is the only spelling both rules accept; the gate recognises it and skips the
|
||||
on-disk check. Flag any other spelling of a cross-skill reference.
|
||||
|
||||
Two directories are exempt, and the exemptions are structural rather than discretionary:
|
||||
|
||||
- **`references/sources.md`.** Its `Research doc:` fields are development-time provenance pointers,
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
- **URL:** https://agentskills.io/specification.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** Complete SKILL.md format specification — frontmatter fields, constraints, body content, optional directories, progressive disclosure levels, file references, validation
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md, references/patterns.md, references/file-structure.md, references/formatting-and-scripts.md
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md, references/patterns.md, references/file-structure.md, references/formatting-and-scripts.md, references/validation-scripts.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-best-practices
|
||||
@@ -47,7 +47,7 @@
|
||||
- **URL:** https://agentskills.io/skill-creation/using-scripts.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** Using scripts in skills — one-off commands, self-contained scripts with inline dependencies, designing scripts for agentic use (no interactive prompts, --help, structured output, idempotency)
|
||||
- **Contributing files:** SKILL.md, references/formatting-and-scripts.md
|
||||
- **Contributing files:** SKILL.md, references/formatting-and-scripts.md, references/validation-scripts.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-quickstart
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
---
|
||||
source_keys:
|
||||
- agentskills-spec
|
||||
- agentskills-using-scripts
|
||||
---
|
||||
|
||||
# Validation Scripts Reference
|
||||
|
||||
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting.
|
||||
Nothing here is needed on a clean run.
|
||||
|
||||
## Report the gap, do not guess
|
||||
|
||||
If a script cannot run at all — Bash denied, `python3` unavailable, PyYAML not importable, `vale`
|
||||
not installed — say so as an **INFO** finding naming the script and the missing dependency, then
|
||||
fall back to the manual checks below. An INFO never changes PASS/FAIL. Silently omitting the
|
||||
dimension a script would have covered reports a clean audit that checked less than it claims to
|
||||
have checked, and the Step 4 coverage line then names a dimension nothing actually examined.
|
||||
|
||||
## Manual structural fallback
|
||||
|
||||
`validate.sh` needs `python3` **and** PyYAML, and refuses to start without either — the description
|
||||
value has to be measured after YAML folding is resolved, so skipping the ADR-0020 gates would be a
|
||||
vacuous pass rather than a partial one. The two are checked separately, so the message already names
|
||||
the right one — report it verbatim rather than diagnosing further:
|
||||
|
||||
```text
|
||||
Error: python3 is required but was not found on PATH.
|
||||
Error: PyYAML is required but is not importable by python3.
|
||||
```
|
||||
|
||||
Without them — or with Bash denied, or on a permission error — work this list
|
||||
by hand and file the results under `### Structure` exactly as the script's output would have been:
|
||||
|
||||
- **`name`** present, 1–64 characters, kebab-case (lowercase letters, digits and hyphens; no
|
||||
leading, trailing or doubled hyphen), and **matching the skill's directory name** exactly.
|
||||
- **`description`** present and non-empty; no unfilled `FILL IN:` placeholder in it. An absent or
|
||||
empty description is a **FAIL**, never a silent skip — it is the one field preloaded into every
|
||||
session, so a skill without one can never be routed to.
|
||||
- **Description length**, measured on the folded YAML value with newlines collapsed to single
|
||||
spaces — not on the raw block scalar, which counts indentation. 250 characters SUGGESTION, 400
|
||||
FAIL (ADR-0020), 1,024 FAIL (agentskills.io spec).
|
||||
- **Body length**, counting everything after the frontmatter's closing `---`. 600 words
|
||||
SUGGESTION, 900 FAIL (ADR-0020).
|
||||
- **Whole-file ceilings**, counting the file including frontmatter: 500 lines FAIL, 2,770 words
|
||||
FAIL (agentskills.io spec). These are a different measurement from the two above — report them
|
||||
as separate findings, never merged.
|
||||
- **A boundary clause is present** — either the prose form (`do not` / `instead` / `rather than` /
|
||||
`not for`) or ADR-0020's compressed `Not <thing> -> <name>` arrow. **SUGGESTION**, not FAIL:
|
||||
the absence is deterministic, but whether this skill warrants one is the auditor's call.
|
||||
- **Boundary targets resolve** — **FAIL** on a name that resolves to nothing. See the section
|
||||
below; resolving these by hand is the one item on this list with a procedure of its own.
|
||||
- **Every `references/<file>.md` named in the body exists on disk** — **FAIL**, not a suggestion.
|
||||
A dispatch table or "read X" trigger naming a missing file sends the agent nowhere. Ignore
|
||||
mentions inside fenced code blocks, and ignore a mention whose own line says the file is gone
|
||||
(`removed`, `deleted`, `renamed`, `superseded`, `replaced`, `obsolete`, `deprecated`, `former`,
|
||||
`gone`, `no longer`, `used to`) — that is a historical note, not a dispatch entry.
|
||||
- **Gotchas discipline**, both **SUGGESTION**. Locate the section by a heading that *is* Gotchas
|
||||
(`## Common Gotchas` counts; `## Gotcha handling` and `## Why gotchas matter` do not), running to
|
||||
the next heading at the same level or shallower. More than five top-level entries is one
|
||||
suggestion; a section over 25% of the body word count is a second, independent one. Count
|
||||
entries at column 0 only — an indented child bullet is not an entry — and ignore fenced code
|
||||
blocks for both.
|
||||
- **No unfilled `FILL IN:` placeholder** anywhere in the body.
|
||||
- **Every file in `scripts/`** carries the executable bit and contains no interactive prompt —
|
||||
no bare `read`, no `select`, nothing that blocks on a TTY.
|
||||
|
||||
## Resolving boundary targets by hand
|
||||
|
||||
Targets are read from **both** boundary forms. The compressed `Not <thing> -> <name>` arrow and the
|
||||
prose form are each parsed *and* target-checked, so a typo in prose phrasing fails exactly as an
|
||||
arrow typo does — do not check only the names after an arrow.
|
||||
|
||||
Build the universe by walking up **from the `SKILL.md` under audit**, never from the validator's own
|
||||
location. The nearest ancestor holding `plugins/*/.apm/skills/` or `plugins/*/.apm/agents/` is the
|
||||
authoring root, falling back to the nearest ancestor holding `.git`. When one is found the universe
|
||||
is every skill and agent under `<root>/plugins/*/`, plus the skill's own apm package, plus the
|
||||
packages that package declares in its `apm.yml` under `dependencies.apm`. Deployed `.claude/` and
|
||||
`.agents/` trees are consulted **only** when no authoring root exists — they are gitignored
|
||||
`apm install` output, and reading them would make a fresh clone and a developer machine disagree.
|
||||
|
||||
Three ways to read the result wrong:
|
||||
|
||||
- **A hyphenated name used attributively is not a dangling target.** "Use pre-commit hooks instead
|
||||
of ad-hoc scripts" reads as a route to `pre-commit` on wording alone. What separates a route from
|
||||
prose is grammar: a route target is terminal — followed by punctuation, a conjunction, or a
|
||||
boundary word — whereas a compound modifier is followed by the noun it modifies. A name followed
|
||||
by an ordinary noun still *confirms* a route when it exists, but never raises a FAIL on its own.
|
||||
- **A SUGGESTION-tier unresolved target is not a FAIL you may promote.** Terminal position alone is
|
||||
not evidence of a route: "run `pre-commit` instead", "see `commit-msg`" and "use the clean-up
|
||||
instead" are all terminal and all prose. A prose-form target earns a FAIL only when its own
|
||||
sentence names another target that *does* resolve; otherwise the script reports it and moves on,
|
||||
and so should you. Route notation — `/name` and `-> name` — is exempt and always FAILs, and it is
|
||||
the fix to recommend when the author did mean a route.
|
||||
- **`INFO boundary-target resolution DID NOT RUN` is not a pass.** The script prints it, and exits
|
||||
0, when no universe could be determined for that path — the usual cause being a skill copy
|
||||
audited outside its package. Report it as an INFO naming the unchecked targets and re-run against
|
||||
the real directory; filing it as clean signs off targets nothing verified.
|
||||
|
||||
## Script-specific failures
|
||||
|
||||
- **`validate-provenance.sh` printed nothing.** That is a pass, not a skip. It also exits 0
|
||||
silently when the skill has no `source_keys` and no `references/sources.md` — nothing to
|
||||
validate is not a finding.
|
||||
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
||||
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
|
||||
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
|
||||
- **`E100 Runtime error ... does not exist` (exit 2) from `vale-wrap.sh`.** An explicit relative
|
||||
`--config` was passed. Pass none: the wrapper locates its own `assets/vale/.vale.ini` from its
|
||||
own path, so a resolved script path plus an unresolved config path produces exactly this. Do not
|
||||
read this exit code as vale being unavailable — that misreading sends the audit down the
|
||||
fallback path while vale was installed and working the whole time.
|
||||
- **The `vale` binary is genuinely absent** (`command not found`). Report one INFO naming it, then
|
||||
fall back to full Step 3 judgment for the description, body-discipline and patterns dimensions —
|
||||
the prefilter's whole coverage. Judge those by rubric rather than dropping them.
|
||||
- **A path argument that does not exist is a hard error** in `vale-wrap.sh`, deliberately: bare
|
||||
`vale` would fall back to reading stdin and print a clean-looking `0 errors ... in stdin`, which
|
||||
the `0 files` guard above does not catch.
|
||||
Reference in New Issue
Block a user