refactor(skills): retrofit the corpus to the ADR-0020 context contract (#129)
Retrofits all 39 skills to ADR-0020's description/body context contract, then fixes what six rounds of independent review found in that retrofit — including four ways the hot gate itself failed open. Closes #99, #107, #108, #110, #111, #114, #115, #120. ## The retrofit (waves 1-5) | | Start | Now | |---|---|---| | Description FAILs (>400 chars) | 26 | **0** | | Body FAILs (>900 words, body-only) | 9 | **0** | | Dangling routing targets | 2 | **0** | | `Kyberforge.CompositionNote` | 10 | **0** | | Preload tax | 21,005 chars | **~10,500** | Under the 12,000-char success criterion. Per-wave detail is on #99. ## The review fixes **The gate failed open four ways, three of them found after the retrofit shipped.** An unrecognised follower token made a dangling target vanish. A skill directory with no `SKILL.md` resolved as a valid target, so a commit could be green locally and red in a fresh clone — three existing fixtures were relying on that, one of which made the install-leak A/B pass vacuously. Then the free-standing `/name` sweep turned out to be gated on the sentence carrying a boundary marker, so route notation in any other sentence was invisible — not an ERROR, not a SUGGESTION, not an INFO — which left the documented "`/name` always blocks" promise false from a second direction. All four fixed and pinned. **Two checks were silently not running.** `validate-provenance.sh` checks 7-8 were dead across nine skills. Waking them exposed a deeper problem: they assume `Research doc:` names a source index, but 30 of 121 entries point at topic content documents, so every new check-7 INFO was a false positive and check 8 was saved from a false-FAIL flood only by an *unannounced* skip. Checks 7/8 are now scoped to source indexes and every skip announces itself (#121). **The retrofit's own anti-goal, four times.** ADR-0020 warns that a blunt gate gets satisfied by deleting content rather than relocating it. `diagnose` and `skill-audit` relocated prose and then read it unconditionally; `prototype` and `vale-config` deleted rules outright that survived nowhere. All four addressed. ## Verification - `bash tests/run-tests.sh --strict` — 24 suites, 0 skipped, 0 failed - `bash tests/run-bats.sh` — 325 tests, 0 failures - `pre-commit run --all-files` — 17/17 - `pre-commit run --hook-stage pre-push --all-files` — 16/16, with `apm marketplace check` and `apm pack --check-clean` run against the remote, not skipped - `scripts/skill-size-check.sh` over all 39 skills — rc 0, 0 ERROR/FAIL, SUGGESTION-only - Preload tax measured at **10,498 chars**, max description 390 — both inside budget - Every new test proven non-vacuous by a deliberate mutation of the behaviour it covers **Per-commit sync, stated accurately:** the ten commits from the latest review round each pass `check-plugin-content-sync` in isolation, verified by checking each out in a detached worktree with a clean between. The earlier gitea window (`dfacf05..bedbd1d`, nine commits) does **not** — its mirror was regenerated in one batch at `bbc7300`. An earlier revision of this description claimed the property held for every commit; it does not, and a bisect through that window lands on a red commit. **Squash-merge** to collapse it, or accept that this range is not bisectable. ## Version bump Six plugins and the catalog take a **patch**, not a minor. The branch is **89 commits — 40 `fix` / 30 `refactor` / 12 `docs` / 5 `chore` / 2 `test` — zero `feat`, zero `!`, zero `BREAKING CHANGE`** — and adds no skill, agent, command or hook. (Two earlier revisions of this section cited a stale histogram, most recently 78 commits; the figures above are measured at HEAD.) Both rules this repo ships (`forge/references/version-bump.md`, landing in this PR, and `git-commits/references/conventional-commits-spec.md`) make that a patch, and the catalog set is unchanged at 7 entries. Not settled by that: four published files were removed from the installed tree, three moved, and `caveman` gained `disable-model-invocation`, retiring its old triggers. Under a strict reading those are major-class and currently ship under `refactor:` with no marker. Whether the deployed skill surface is a public contract is written down nowhere — worth deciding, but it outlives this PR. ## Deliberately not in scope #112 (cherry-pick ownership, now resolved in favour of `git-commits`), #113 (`rtk git` normalisation), #116 (research fan-out), #101 (audit-skill merge), #122 (non-spec skill-root files), #123 (no PRD producer) stay open. #117 is the one worth reading: the contract's remedy is to move prose into `references/`, which is exactly where neither the size gate nor Vale looks — and the blind spot is wider than #117 currently records, since there is no root `.vale.ini` at all, so every ADR, `CONTEXT.md` and `README.md` is unlinted too. That blind spot let this branch carry two `level: error` `Kyberforge.SentenceOpenerThereIs` violations into `references/` files it created — `provider-adapter-author/references/provider-matrix.md:31` and `agent-audit/references/finding-criteria.md:95`. Both are reworded in `afadaae`, confirmed by routing each file through the audit's own `vale-wrap.sh` (1 error each before, 0 after). Five further occurrences sit in `references/` files already on `main`; those are the pre-existing corpus and stay with #117, which is the real fix. Also unfixed and not this PR's: `apm install` appends a duplicate `SessionStart` entry to `.claude/settings.json`, so a fresh clone cannot get pre-push green without an edit AGENTS.md warns against. Reproduces identically on `main`. Co-authored-by: Defame1297 <gitea@rkdr.net> Reviewed-on: https://git.dev.rkdr.net/Defame1297/holocron/pulls/129 Co-authored-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net> Co-committed-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net>
This commit was merged in pull request #129.
This commit is contained in:
@@ -31,7 +31,16 @@ Include content the agent lacks:
|
||||
Move to `references/`, behind an explicit "If X, read `references/<file>.md`" trigger — the literal
|
||||
conditional form, never a generic pointer. Write the real filename in the skill under audit; the
|
||||
angle brackets are a placeholder here, and a literal `references/file.md` in a body is an ERROR
|
||||
from the ADR-0020 gate because no such file exists on disk. Move:
|
||||
from the ADR-0020 gate because no such file exists on disk.
|
||||
|
||||
**A dispatch table satisfies this requirement on its own.** A table row already pairs a condition
|
||||
with a target, which is exactly what the literal form encodes; restating each row underneath as a
|
||||
prose conditional duplicates the routing in the one body whose whole purpose is to be short. Where a
|
||||
body dispatches, audit the table for condition/target completeness and stop there — do not require
|
||||
the conditional form as well. The literal form is what a body needs when it loads a reference
|
||||
*without* a dispatch table: a single mid-procedure deepening, an escape hatch, an error path.
|
||||
|
||||
Move:
|
||||
|
||||
- Lookup tables and spec restatements
|
||||
- Output schemas, templates and example blocks
|
||||
@@ -71,10 +80,28 @@ table** plus the gates common to every branch, and each flow lives in its own se
|
||||
`references/` file. Inlining all of them is a FAIL regardless of word count, because every
|
||||
invocation then pays for every branch it did not take.
|
||||
|
||||
The reference shape in this repo is `apm-workflow`: a **421-word body** dispatching to roughly
|
||||
3,000 words of references across five mutually exclusive invocations. Its whole-file count is 554
|
||||
words — cite 421 when calibrating a body, or the conflation this section warns against reappears
|
||||
in the finding itself.
|
||||
The reference shape in this repo is `apm-workflow`: a **294-word body** dispatching to 3,154 words
|
||||
of references across five mutually exclusive flows. Its whole-file count is 348 words — cite 294
|
||||
when calibrating a body, or the conflation this section warns against reappears in the finding
|
||||
itself. The 3,154 counts the five flow files only; `references/sources.md` is a provenance record
|
||||
and is never loaded at runtime, so counting it inflates the dispatched total.
|
||||
|
||||
### What earns the wiring exemption
|
||||
|
||||
A dispatch table earns the exemption above on its properties, not on which skill it appears in.
|
||||
Audit any dispatching body against these four:
|
||||
|
||||
- Every flow the skill handles has a row, and every row names a target file that exists on disk.
|
||||
- Each row pairs a condition the agent can evaluate from the request with exactly one target. A row
|
||||
keyed on a literal slash invocation fails this: a model-invoked activation never produces that
|
||||
string, so the routing silently falls to whatever else the row carries.
|
||||
- One line after the table tells the agent to read the file its row matched, and only that one.
|
||||
- The gates every branch needs sit in the body, not inside one flow's file — see the reachability
|
||||
precondition below.
|
||||
|
||||
A table missing any of the four is not exempt, and the literal-conditional requirement applies to it
|
||||
as written. The exemption covers the wiring form only: every other rule in this file applies to a
|
||||
dispatching skill exactly as it applies to any other.
|
||||
|
||||
## Gotchas sections
|
||||
|
||||
@@ -96,19 +123,32 @@ Constraints:
|
||||
`validate.sh` emits it through `suggest()` and the run still exits 0.
|
||||
- **A Gotcha that paraphrases a step in the body below it is a FAIL.** It has no independent
|
||||
content, and it teaches the agent that Gotchas can be skimmed because the real instruction is
|
||||
coming. This one is the auditor's call — no script detects it.
|
||||
coming. This one is the auditor's call — no script detects it. The Fix is conditional: delete the
|
||||
Gotcha only if the surviving copy is reachable from every branch that needs it — see the
|
||||
reachability precondition below.
|
||||
- **A Gotchas section exceeding 25% of the body is a SUGGESTION** — the body has been inverted into
|
||||
a preamble. Same tier and same reasoning as the entry count, and independent of it: either can
|
||||
fire without the other.
|
||||
- Place the section near the top. A gotcha read after the mistake is worthless, which is also why
|
||||
Gotchas is the one construct exempt from moving to `references/`.
|
||||
|
||||
Worked negative example — `git-commits` carries twelve entries, of which four restate content
|
||||
that already appears below or in the description:
|
||||
Worked negative example — **`git-commits` v0.1.2 at commit `5e23250`, a fixed pre-retrofit
|
||||
snapshot, not the current file.** The live skill is v0.1.3 and matches none of the citations below;
|
||||
they are quoted as they stood before the ADR-0020 retrofit, and are not to be refreshed against
|
||||
`HEAD`. The snapshot is reachable only from a checkout of the authoring repo — an installed plugin
|
||||
cache holds no git history and no such path — so read the citations below as quoted rather than
|
||||
going to look for the file. From a checkout:
|
||||
|
||||
```text
|
||||
git show 5e23250:<the git plugin>/.apm/skills/git-commits/SKILL.md
|
||||
```
|
||||
|
||||
That body carried twelve Gotchas, four of which restated content already below them or already in
|
||||
the description:
|
||||
|
||||
| Gotcha | Restates |
|
||||
|---|---|
|
||||
| `:31` "Communicates SemVer impact" | the description |
|
||||
| `:31` "SemVer mapping is not optional" | the description |
|
||||
| `:32` "Confirmation gates are mandatory for destructive operations" | step 9 at `:52` |
|
||||
| `:33` "Never skip hooks with `--no-verify`" | step 9 at `:52` |
|
||||
| `:36` "Never commit secrets" | step 2 at `:45` |
|
||||
@@ -118,6 +158,25 @@ All four are FAILs under the paraphrase rule. The entry count and the section's
|
||||
fails the run on its own. What makes this worth auditing directly is that the four paraphrase FAILs
|
||||
pass every word gate there is; only reading the construct finds them.
|
||||
|
||||
### The paraphrase rule has a reachability precondition
|
||||
|
||||
**A Gotcha that restates a step may be deleted only when the surviving copy is reachable from every
|
||||
branch that needs it.** In a dispatch body it usually is not: each flow file is loaded alone, so a
|
||||
step in one is invisible to an invocation that took another branch. When the restated rule is a
|
||||
safety gate more than one flow needs, the Fix is to **move it into the body's common-gates section**,
|
||||
never to drop it in favour of the per-flow copy.
|
||||
|
||||
Row four is the case that proves it. Following the rule literally, the retrofit deleted the
|
||||
always-loaded secrets Gotcha and kept step 2 of `references/create-commit.md` — but `git-commits`
|
||||
dispatches to exactly one flow file, and `references/rewrite-history.md` stages changes and runs
|
||||
`--amend`, which commits newly staged content exactly as a fresh commit does. A grep for `secret`
|
||||
across the skill in that state returned one hit, on a path two of three branches never reach: that
|
||||
branch could commit a credential with no check anywhere in its loaded context, against this repo's
|
||||
governance hard prohibition. v0.1.3 carries the rule as gate 2 of "Gates on every flow" instead.
|
||||
|
||||
So check reachability before writing the Fix. Rows one to three are unaffected — the description is
|
||||
loaded on every invocation, and confirmation is likewise a common gate rather than a per-flow step.
|
||||
|
||||
## Calibrating control
|
||||
|
||||
**Be prescriptive** when operations are fragile, consistency matters, or a specific sequence must be
|
||||
@@ -146,23 +205,5 @@ Use pypdf, pdfplumber, PyMuPDF, or pdf2image...
|
||||
Use pdfplumber for text extraction. For scanned PDFs requiring OCR, use pdf2image instead.
|
||||
```
|
||||
|
||||
## Auditing guidance
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A sentence answers "no" to the core test — it is padding
|
||||
- The body exceeds 900 words counted body-only (`validate.sh` reports it)
|
||||
- Two or more mutually exclusive flows are inlined instead of dispatched
|
||||
- A Gotcha paraphrases a step in the body below it
|
||||
- A decision point presents a menu of options with no default
|
||||
- An instruction repeats content already in the description
|
||||
- A prescriptive sequence is used where flexibility is fine, or the reverse
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- The body exceeds 600 words counted body-only but stays at or under 900
|
||||
- The Gotchas section carries more than five entries
|
||||
- The Gotchas section exceeds 25% of the body
|
||||
- A rationale is missing from an include/exclude rule — present but unexplained
|
||||
- Gotchas are correct but placed late in the body rather than near the top
|
||||
- Content that only one branch reaches is inlined where a `references/` file would serve
|
||||
The FAIL and SUGGESTION criteria for this dimension live in `references/finding-criteria.md`,
|
||||
which Step 3 loads on every run.
|
||||
|
||||
@@ -40,8 +40,11 @@ A model-invoked description carries exactly three things:
|
||||
the agent is deciding whether to act, not reading a catalogue entry.
|
||||
2. **At most one capability clause.** What it does, in one clause. Never an enumeration.
|
||||
3. **Boundary clause.** Compressed form: `Not <thing> -> <skill-name>.` The target must resolve to
|
||||
a real skill directory or agent file in the authoring source; `validate.sh` checks that
|
||||
deterministically and a dangling target already surfaces as a Structure FAIL.
|
||||
a real skill directory or agent file in the authoring source. `validate.sh` checks that
|
||||
deterministically and grades it by notation: an unresolved `/name` or arrow target is an ERROR
|
||||
and reaches the report as a Structure FAIL, while an unresolved prose-form target ("use `y`
|
||||
instead") is only a SUGGESTION unless a second target in the same sentence resolves. Take the
|
||||
script's tier as given and report it once, under Structure.
|
||||
|
||||
Everything else belongs in the body or in `README.md`.
|
||||
|
||||
@@ -80,38 +83,5 @@ description: >
|
||||
|
||||
(`data-model` is illustrative. In a real description the target has to resolve.)
|
||||
|
||||
## Auditing guidance
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- **Over 400 characters.** Measured on the folded YAML value, not the raw source lines.
|
||||
`validate.sh` reports the number; do not re-derive it, but do point the Fix at what to cut.
|
||||
- **Internal mechanics appear in the description.** Any of:
|
||||
- capability enumeration or a feature list;
|
||||
- output-format detail ("Produces a compact findings report with Why and Fix per finding");
|
||||
- composition or architecture notes ("composes X rather than duplicating Y", "a cross-cutting
|
||||
shared skill", "the human-facing entry point", "replaces the old flat invocation");
|
||||
- implementation detail ("self-validates via a bundled deterministic script").
|
||||
|
||||
None of it can change a routing decision and all of it is preloaded.
|
||||
`Kyberforge.CompositionNote` catches the common phrasings deterministically; the rest is
|
||||
judgment. This is the rule that deflates a description, so apply it before reaching for length.
|
||||
- **The same trigger stated twice in two registers** — a verb list, then the same verbs re-quoted
|
||||
as user phrasings, usually in the same order. One register, whichever routes better.
|
||||
- **Descriptive rather than imperative phrasing** (`This skill ...`, `This is the ...`).
|
||||
`Kyberforge.DescriptionOpener` catches any opener matching `^This`.
|
||||
- **Vague capabilities** ("helps with APIs" where "parses and validates OpenAPI specs" was
|
||||
available). `Kyberforge.VagueWording` catches the known filler; imprecision outside that list is
|
||||
judgment.
|
||||
- **A boundary clause naming a target that does not resolve** to a real skill directory or agent
|
||||
file in the authoring source. `validate.sh` reports the unresolved name.
|
||||
- **Trigger-list, boundary or indirect-trigger content on a hand-invoked skill** — see Step 0.
|
||||
- **Over 1024 characters** — the agentskills.io specification ceiling, unchanged and independent
|
||||
of the 400-character house ceiling above.
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- **Over 250 characters** but at or under 400. This tier is what moves the corpus average; the FAIL
|
||||
tier only stops outliers. Report it rather than treating a 399-character description as clean.
|
||||
- A near-miss exclusion is present but targets a weak near-miss.
|
||||
- An indirect trigger is present and warranted but could name the omitted phrasing more precisely.
|
||||
The FAIL and SUGGESTION criteria for this dimension live in `references/finding-criteria.md`,
|
||||
which Step 3 loads on every run.
|
||||
|
||||
@@ -24,9 +24,19 @@ knows to look at. Flag any other directory as a FAIL.
|
||||
## Cross-plugin path references
|
||||
|
||||
A plugin is copied to a cache on install, and a path that climbs out of the skill directory stops
|
||||
resolving there. Flag any `../`, `../../`, or absolute repo path (`plugins/<plugin>/skills/<other>/`
|
||||
and its APM-native equivalent `.apm/skills/<other>/`) appearing in `SKILL.md`, `scripts/`,
|
||||
`references/` or `assets/`.
|
||||
resolving there. Flag a path in `SKILL.md`, `scripts/`, `references/` or `assets/` when it
|
||||
**resolves outside the skill directory** — an absolute repo path
|
||||
(`plugins/<plugin>/skills/<other>/` and its APM-native equivalent `.apm/skills/<other>/`), a
|
||||
plugin-root path (`docs/`, `bin/`), or a `../` chain that leaves the skill root.
|
||||
|
||||
Resolve before flagging, twice over:
|
||||
|
||||
- **Resolve the path.** `$SKILL_DIR/../assets/templates` climbs one level from a `scripts/`
|
||||
directory and lands back inside the same skill, so it resolves in a cache install and is not a
|
||||
finding. A bare `../` is not the defect; leaving the skill is.
|
||||
- **Skip fenced code blocks.** A path inside a fenced block is an example, and rubrics quote outside
|
||||
paths deliberately as negative examples of what not to write. Flag a fenced path only when the
|
||||
surrounding prose presents it as the form to copy.
|
||||
|
||||
**Referring to another skill's file.** There is one sanctioned spelling, and it is possessive:
|
||||
`skill-audit's references/validation-scripts.md`. Write the skill by name and let the reader
|
||||
@@ -39,9 +49,10 @@ on-disk check. Flag any other spelling of a cross-skill reference.
|
||||
Two directories are exempt, and the exemptions are structural rather than discretionary:
|
||||
|
||||
- **`references/sources.md`.** Its `Research doc:` fields are development-time provenance pointers,
|
||||
not runtime references. They are expected to be unresolvable after install, and
|
||||
`validate-provenance.sh` handles that by skipping upstream checks silently when the path is
|
||||
absent. Flagging them would make every correctly-provenanced skill fail.
|
||||
not runtime references. They are expected to be unresolvable after install, so
|
||||
`validate-provenance.sh` does not treat an absent path as a FAIL — it emits an INFO naming the
|
||||
slug and stating that checks 7 and 8 did not run for it. Flagging them as broken references
|
||||
would make every correctly-provenanced skill fail.
|
||||
- **`tests/`.** Test files are dev-only and may reference repo-level infrastructure such as a shared
|
||||
`tests/test_helper/`. The exemption is conditional on the dependency being declared: if `tests/`
|
||||
exists and `tests/README.md` is absent or does not document it, that is a FAIL.
|
||||
@@ -60,19 +71,5 @@ The skill has to agree with itself. Three checks:
|
||||
A stale README row is the most common finding here and the easiest to miss from inside an
|
||||
authoring pass, because the author knows what was intended and reads it into the gap.
|
||||
|
||||
## Auditing guidance
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A directory outside the four permitted ones exists
|
||||
- Test files sit in `scripts/`
|
||||
- A non-spec file sits at the skill root
|
||||
- A cross-plugin or parent-relative path appears outside the two exempt locations
|
||||
- `tests/` exists but `tests/README.md` is missing or does not document its repo-level dependency
|
||||
- `README.md` is absent, or its file table has a missing or stale row
|
||||
- `SKILL.md` describes a script invocation the script does not accept
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- An optional directory exists but holds only a placeholder README
|
||||
- `README.md` is accurate but describes a file's purpose more thinly than `SKILL.md` does
|
||||
The FAIL and SUGGESTION criteria for this dimension live in `references/finding-criteria.md`,
|
||||
which Step 3 loads on every run.
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
source_keys:
|
||||
- agentskills-spec
|
||||
- agentskills-best-practices
|
||||
- agentskills-optimizing-descriptions
|
||||
- agentskills-using-scripts
|
||||
---
|
||||
|
||||
# Finding Criteria
|
||||
|
||||
Every FAIL and SUGGESTION criterion, for every qualitative dimension, and nothing else. The
|
||||
reasoning each criterion stands on, its worked examples and its house rules stay in that
|
||||
dimension's rubric, which Step 3 loads only for a dimension this file puts in play.
|
||||
|
||||
Two rules on using it:
|
||||
|
||||
- A criterion that plainly applies is a finding. Write it up citing file and line.
|
||||
- A criterion that might apply, or whose call the wording here does not settle, is a reason to load
|
||||
that dimension's rubric — never a reason to drop the candidate. This file decides which rubrics
|
||||
to read; it does not settle a close call on its own.
|
||||
|
||||
## description — `references/description-quality.md`
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- **Over 400 characters.** Measured on the folded YAML value, not the raw source lines.
|
||||
`validate.sh` reports the number; do not re-derive it, but do point the Fix at what to cut.
|
||||
- **Internal mechanics appear in the description.** Any of:
|
||||
- capability enumeration or a feature list;
|
||||
- output-format detail ("Produces a compact findings report with Why and Fix per finding");
|
||||
- composition or architecture notes ("composes X rather than duplicating Y", "a cross-cutting
|
||||
shared skill", "the human-facing entry point", "replaces the old flat invocation");
|
||||
- implementation detail ("self-validates via a bundled deterministic script").
|
||||
|
||||
None of it can change a routing decision and all of it is preloaded.
|
||||
`Kyberforge.CompositionNote` catches the common phrasings deterministically; the rest is
|
||||
judgment. This is the rule that deflates a description, so apply it before reaching for length.
|
||||
- **The same trigger stated twice in two registers** — a verb list, then the same verbs re-quoted
|
||||
as user phrasings, usually in the same order. One register, whichever routes better.
|
||||
- **Descriptive rather than imperative phrasing** (`This skill ...`, `This is the ...`).
|
||||
`Kyberforge.DescriptionOpener` catches any opener matching `^This`.
|
||||
- **Vague capabilities** ("helps with APIs" where "parses and validates OpenAPI specs" was
|
||||
available). `Kyberforge.VagueWording` catches the known filler; imprecision outside that list is
|
||||
judgment.
|
||||
- **Trigger-list, boundary or indirect-trigger content on a hand-invoked skill** — see Step 0 of
|
||||
`references/description-quality.md`.
|
||||
- **Over 1024 characters** — the agentskills.io specification ceiling, unchanged and independent
|
||||
of the 400-character house ceiling above.
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- **Over 250 characters** but at or under 400. This tier is what moves the corpus average; the FAIL
|
||||
tier only stops outliers. Report it rather than treating a 399-character description as clean.
|
||||
- A near-miss exclusion is present but targets a weak near-miss.
|
||||
- An indirect trigger is present and warranted but could name the omitted phrasing more precisely.
|
||||
|
||||
**An unresolved boundary target is not graded here.** `validate.sh` owns that call and tiers it by
|
||||
notation — `/name` or an arrow form is an ERROR, the bare prose form a SUGGESTION unless a second
|
||||
target in the same sentence resolves — and Step 1 has already filed it under `### Structure` at that
|
||||
tier. Re-grading it as a description FAIL puts one target in the report twice at two tiers. What is
|
||||
left to judgment here is semantic and the script cannot reach it: whether a target that *does*
|
||||
resolve is the right sibling to exclude, and whether a clause naming no target at all ("examine the
|
||||
files manually") should have named one.
|
||||
|
||||
## body-discipline — `references/body-discipline.md`
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A sentence answers "no" to the core test — it is padding
|
||||
- The body exceeds 900 words counted body-only (`validate.sh` reports it)
|
||||
- Two or more mutually exclusive flows are inlined instead of dispatched
|
||||
- A Gotcha paraphrases a step in the body below it that every branch reaching the Gotcha also
|
||||
reaches
|
||||
- A decision point presents a menu of options with no default
|
||||
- An instruction repeats content already in the description
|
||||
- A prescriptive sequence is used where flexibility is fine, or the reverse
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- The body exceeds 600 words counted body-only but stays at or under 900
|
||||
- The Gotchas section carries more than five entries
|
||||
- The Gotchas section exceeds 25% of the body
|
||||
- A rationale is missing from an include/exclude rule — present but unexplained
|
||||
- Gotchas are correct but placed late in the body rather than near the top
|
||||
- Content that only one branch reaches is inlined where a `references/` file would serve
|
||||
|
||||
## patterns — `references/patterns.md`
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A Gotcha entry is a general tip or a reminder rather than a fact that defies a reasonable
|
||||
assumption
|
||||
- An inner code fence is unescaped inside a markdown block, breaking the render
|
||||
- A checklist wraps a single step
|
||||
- A conditional reference gives no trigger — `Kyberforge.PaddingPhrase` reports the common form
|
||||
- The agent must produce a specific format and no output template is given
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- Gotchas are correctly formed but placed late in the body
|
||||
- An output template is present but permissive where the consumer needs it exact
|
||||
- A conditional reference names a trigger that is real but broader than the branch it guards
|
||||
|
||||
## file-structure and internal-consistency — `references/file-structure.md`
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A directory outside the four permitted ones exists
|
||||
- Test files sit in `scripts/`
|
||||
- A non-spec file sits at the skill root
|
||||
- A path that resolves outside the skill directory appears outside the two exempt locations, in
|
||||
prose rather than in a fenced example
|
||||
- `tests/` exists but `tests/README.md` is missing or does not document its repo-level dependency
|
||||
- `README.md` is absent, or its file table has a missing or stale row
|
||||
- `SKILL.md` describes a script invocation the script does not accept
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- An optional directory exists but holds only a placeholder README
|
||||
- `README.md` is accurate but describes a file's purpose more thinly than `SKILL.md` does
|
||||
|
||||
## formatting and scripts — `references/formatting-and-scripts.md`
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A script prompts interactively, in any form
|
||||
- A script exposes no `--help`
|
||||
- A destructive script has no `--dry-run`
|
||||
- Data and diagnostics share a stream, so the output cannot be piped
|
||||
- A relative path named in the body does not resolve
|
||||
- Heading levels are inconsistent enough to break the document's structure
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- Exit codes are meaningful but undocumented in `--help`
|
||||
- A code block is untagged where a language applies
|
||||
- A script is idempotent in practice but does not say so, leaving a re-run's safety unclear
|
||||
- List indentation or section spacing is inconsistent without breaking the render
|
||||
@@ -44,20 +44,5 @@ follow from that:
|
||||
undocumented one is a coin flip.
|
||||
- **`--dry-run` present for destructive operations.**
|
||||
|
||||
## Auditing guidance
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A script prompts interactively, in any form
|
||||
- A script exposes no `--help`
|
||||
- A destructive script has no `--dry-run`
|
||||
- Data and diagnostics share a stream, so the output cannot be piped
|
||||
- A relative path named in the body does not resolve
|
||||
- Heading levels are inconsistent enough to break the document's structure
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- Exit codes are meaningful but undocumented in `--help`
|
||||
- A code block is untagged where a language applies
|
||||
- A script is idempotent in practice but does not say so, leaving a re-run's safety unclear
|
||||
- List indentation or section spacing is inconsistent without breaking the render
|
||||
The FAIL and SUGGESTION criteria for this dimension live in `references/finding-criteria.md`,
|
||||
which Step 3 loads on every run.
|
||||
|
||||
@@ -50,19 +50,5 @@ forms are judgment.
|
||||
`references/` when only one dispatch branch produces that output. A template inlined for a branch
|
||||
most invocations never take is body-discipline padding.
|
||||
|
||||
## Auditing guidance
|
||||
|
||||
Flag as FAIL if:
|
||||
|
||||
- A Gotcha entry is a general tip or a reminder rather than a fact that defies a reasonable
|
||||
assumption
|
||||
- An inner code fence is unescaped inside a markdown block, breaking the render
|
||||
- A checklist wraps a single step
|
||||
- A conditional reference gives no trigger — `Kyberforge.PaddingPhrase` reports the common form
|
||||
- The agent must produce a specific format and no output template is given
|
||||
|
||||
Flag as SUGGESTION if:
|
||||
|
||||
- Gotchas are correctly formed but placed late in the body
|
||||
- An output template is present but permissive where the consumer needs it exact
|
||||
- A conditional reference names a trigger that is real but broader than the branch it guards
|
||||
The FAIL and SUGGESTION criteria for this dimension live in `references/finding-criteria.md`,
|
||||
which Step 3 loads on every run.
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
- **URL:** https://agentskills.io/specification.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** Complete SKILL.md format specification — frontmatter fields, constraints, body content, optional directories, progressive disclosure levels, file references, validation
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md, references/patterns.md, references/file-structure.md, references/formatting-and-scripts.md, references/validation-scripts.md
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/description-quality.md, references/patterns.md, references/file-structure.md, references/formatting-and-scripts.md, references/finding-criteria.md, references/validation-scripts.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-best-practices
|
||||
@@ -23,7 +23,7 @@
|
||||
- **URL:** https://agentskills.io/skill-creation/best-practices.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** Best practices for skill creators — starting from real expertise, spending context wisely, calibrating control, instruction patterns (gotchas, templates, checklists, validation loops)
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/patterns.md
|
||||
- **Contributing files:** SKILL.md, references/body-discipline.md, references/patterns.md, references/finding-criteria.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-optimizing-descriptions
|
||||
@@ -31,7 +31,7 @@
|
||||
- **URL:** https://agentskills.io/skill-creation/optimizing-descriptions.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** How to systematically test and improve skill descriptions for triggering accuracy — eval queries, trigger rate testing, train/validation splits, optimization loop
|
||||
- **Contributing files:** SKILL.md, references/description-quality.md
|
||||
- **Contributing files:** SKILL.md, references/description-quality.md, references/finding-criteria.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-evaluating-skills
|
||||
@@ -47,7 +47,7 @@
|
||||
- **URL:** https://agentskills.io/skill-creation/using-scripts.md
|
||||
- **Research doc:** plugins/kyberforge/docs/research/docs/agentskillsio/sources.md
|
||||
- **Description:** Using scripts in skills — one-off commands, self-contained scripts with inline dependencies, designing scripts for agentic use (no interactive prompts, --help, structured output, idempotency)
|
||||
- **Contributing files:** SKILL.md, references/formatting-and-scripts.md, references/validation-scripts.md
|
||||
- **Contributing files:** SKILL.md, references/formatting-and-scripts.md, references/finding-criteria.md, references/validation-scripts.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## agentskills-quickstart
|
||||
|
||||
@@ -99,9 +99,12 @@ Three ways to read the result wrong:
|
||||
|
||||
## Script-specific failures
|
||||
|
||||
- **`validate-provenance.sh` printed nothing.** That is a pass, not a skip. It also exits 0
|
||||
silently when the skill has no `source_keys` and no `references/sources.md` — nothing to
|
||||
validate is not a finding.
|
||||
- **`validate-provenance.sh` printed nothing *and exited 0*.** That is a pass, not a skip — it
|
||||
exits 0 silently when the skill has no `source_keys` and no `references/sources.md`, and nothing
|
||||
to validate is not a finding. Check the exit code before you believe the silence: a target that
|
||||
is not a directory, a directory holding no `SKILL.md`, a missing or extra argument, and an absent
|
||||
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
|
||||
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
|
||||
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
||||
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
|
||||
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
|
||||
|
||||
Reference in New Issue
Block a user