Files

Defame1297 88866b81a3 fix(kyberforge): announce every provenance skip and scope checks 7-8 to source indexes

The clean provenance bill was an artifact. Checks 7 and 8 assume `Research doc:`
names a source index whose H2s are slugs, but 30 of 121 corpus entries point at
topic content documents whose H2s are topics. Those 30 produced every new check-7
INFO — all false positives. Check 8 aimed at the same documents, which carry no
`Status:` line at all, would have emitted a large false-FAIL flood; the only thing
preventing it was an unannounced `rd_status != extracted` skip. So "0 new FAILs"
rested on exactly the fail-open class this branch exists to remove, and naively
fixing the skip would have turned the branch red.

Checks 7/8 now run only when the research doc's basename is `sources.md`, and every
other case emits a visible INFO naming the slug. The dangling-path INFO stays ahead
of the basename gate, because a path that does not resolve is rot whatever it is
named. `parse_status` accepts the bullet form and a trailing note after the
backticked value, so a status it cannot read no longer reads as "nothing to check".

Corpus: 36 INFOs of which 30 were false, to 56 of which none are. FAIL stays 0, and
no Status line flipped to `extracted` under the new parser, so no FAIL was
suppressed by luck.

Also closed, each a silent pass: a nonexistent directory, a directory with no
SKILL.md, and extra arguments now exit 2; a UTF-8 BOM no longer defeats frontmatter
parsing; the bare `except Exception: return False` that turned an unreadable file
into a clean pass is gone, with all reads pinned to UTF-8; check 3 walks nested
`references/` subdirectories; `FILL IN:` at end of line no longer escapes checks 1
and 6; duplicate `## slug` blocks and repeated `Research doc:` lines are announced
rather than half-read.

The agent-audit copy carried all of the above unfixed and is now ported, minus the
four fixes that are genuinely N/A at agent scope — it reads a plugin-root
`sources.md` and has no checks 7/8 and no `references/` tree. Its silent exit 0 for
a file outside plugin scope is preserved deliberately: that is a verdict about a
valid file, not a skip, and `check-scope-walkup-sync.sh` pins it. Every exit-2 gate
therefore decides from the argument alone, before the walk-up runs.

`validation-scripts.md` said flatly that silence from the validator is a pass, not
a skip. That sentence is what made a typo'd path dangerous, and both copies are
corrected here. The matching SKILL.md exit-code guidance lands with the audit
rubric change, which touches the same files.

Tests: skill-audit 45 to 65, agent-audit 24 to 43, every new case proven by mutation.

Refs: #111, #118, #121

2026-09-01 12:37:50 +00:00

8.3 KiB

Raw Blame History

source_keys

agentskills-spec

agentskills-using-scripts

Validation Scripts Reference

Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting. Nothing here is needed on a clean run.

Report the gap, do not guess

If a script cannot run at all — Bash denied, python3 unavailable, PyYAML not importable, vale not installed — say so as an INFO finding naming the script and the missing dependency, then fall back to the manual checks below. An INFO never changes PASS/FAIL. Silently omitting the dimension a script would have covered reports a clean audit that checked less than it claims to have checked, and the Step 4 coverage line then names a dimension nothing actually examined.

Manual structural fallback

validate.sh needs python3 and PyYAML, and refuses to start without either — the description value has to be measured after YAML folding is resolved, so skipping the ADR-0020 gates would be a vacuous pass rather than a partial one. The two are checked separately, so the message already names the right one — report it verbatim rather than diagnosing further:

Error: python3 is required but was not found on PATH.
Error: PyYAML is required but is not importable by python3.

Without them — or with Bash denied, or on a permission error — work this list by hand and file the results under ### Structure exactly as the script's output would have been:

name present, 1–64 characters, kebab-case (lowercase letters, digits and hyphens; no leading, trailing or doubled hyphen), and matching the skill's directory name exactly.
description present and non-empty; no unfilled FILL IN: placeholder in it. An absent or empty description is a FAIL, never a silent skip — it is the one field preloaded into every session, so a skill without one can never be routed to.
Description length, measured on the folded YAML value with newlines collapsed to single spaces — not on the raw block scalar, which counts indentation. 250 characters SUGGESTION, 400 FAIL (ADR-0020), 1,024 FAIL (agentskills.io spec).
Body length, counting everything after the frontmatter's closing ---. 600 words SUGGESTION, 900 FAIL (ADR-0020).
Whole-file ceilings, counting the file including frontmatter: 500 lines FAIL, 2,770 words FAIL (agentskills.io spec). These are a different measurement from the two above — report them as separate findings, never merged.
A boundary clause is present — either the prose form (do not / instead / rather than / not for) or ADR-0020's compressed Not <thing> -> <name> arrow. SUGGESTION, not FAIL: the absence is deterministic, but whether this skill warrants one is the auditor's call.
Boundary targets resolve — FAIL on a name that resolves to nothing. See the section below; resolving these by hand is the one item on this list with a procedure of its own.
Every references/<file>.md named in the body exists on disk — FAIL, not a suggestion. A dispatch table or "read X" trigger naming a missing file sends the agent nowhere. Ignore mentions inside fenced code blocks, and ignore a mention whose own line says the file is gone (removed, deleted, renamed, superseded, replaced, obsolete, deprecated, former, gone, no longer, used to) — that is a historical note, not a dispatch entry.
Gotchas discipline, both SUGGESTION. Locate the section by a heading that is Gotchas (## Common Gotchas counts; ## Gotcha handling and ## Why gotchas matter do not), running to the next heading at the same level or shallower. More than five top-level entries is one suggestion; a section over 25% of the body word count is a second, independent one. Count entries at column 0 only — an indented child bullet is not an entry — and ignore fenced code blocks for both.
No unfilled FILL IN: placeholder anywhere in the body.
Every file in scripts/ carries the executable bit and contains no interactive prompt — no bare read, no select, nothing that blocks on a TTY.

Resolving boundary targets by hand

Targets are read from both boundary forms. The compressed Not <thing> -> <name> arrow and the prose form are each parsed and target-checked, so a typo in prose phrasing fails exactly as an arrow typo does — do not check only the names after an arrow.

Build the universe by walking up from the SKILL.md under audit, never from the validator's own location. The nearest ancestor holding plugins/*/.apm/skills/ or plugins/*/.apm/agents/ is the authoring root, falling back to the nearest ancestor holding .git. When one is found the universe is every skill and agent under <root>/plugins/*/, plus the skill's own apm package, plus the packages that package declares in its apm.yml under dependencies.apm. Deployed .claude/ and .agents/ trees are consulted only when no authoring root exists — they are gitignored apm install output, and reading them would make a fresh clone and a developer machine disagree.

Three ways to read the result wrong:

A hyphenated name used attributively is not a dangling target. "Use pre-commit hooks instead of ad-hoc scripts" reads as a route to pre-commit on wording alone. What separates a route from prose is grammar: a route target is terminal — followed by punctuation, a conjunction, or a boundary word — whereas a compound modifier is followed by the noun it modifies. A name followed by an ordinary noun still confirms a route when it exists, but never raises a FAIL on its own.
A SUGGESTION-tier unresolved target is not a FAIL you may promote. Terminal position alone is not evidence of a route: "run pre-commit instead", "see commit-msg" and "use the clean-up instead" are all terminal and all prose. A prose-form target earns a FAIL only when its own sentence names another target that does resolve; otherwise the script reports it and moves on, and so should you. Route notation — /name and -> name — is exempt and always FAILs, and it is the fix to recommend when the author did mean a route.
INFO boundary-target resolution DID NOT RUN is not a pass. The script prints it, and exits 0, when no universe could be determined for that path — the usual cause being a skill copy audited outside its package. Report it as an INFO naming the unchecked targets and re-run against the real directory; filing it as clean signs off targets nothing verified.

Script-specific failures

validate-provenance.sh printed nothing and exited 0. That is a pass, not a skip — it exits 0 silently when the skill has no source_keys and no references/sources.md, and nothing to validate is not a finding. Check the exit code before you believe the silence: a target that is not a directory, a directory holding no SKILL.md, a missing or extra argument, and an absent python3 all exit 2 with a message on stderr. Exit 2 means the script never ran — report it as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
vale reports 0 files. Treat the pass as NOT RUN, not as clean, and fall back to full Step 3 judgment for the dimensions it would have covered. The bundled Kyberforge style is scoped by glob in assets/vale/.vale.ini; a file outside those globs is silently not linted.
E100 Runtime error ... does not exist (exit 2) from vale-wrap.sh. An explicit relative --config was passed. Pass none: the wrapper locates its own assets/vale/.vale.ini from its own path, so a resolved script path plus an unresolved config path produces exactly this. Do not read this exit code as vale being unavailable — that misreading sends the audit down the fallback path while vale was installed and working the whole time.
The vale binary is genuinely absent (command not found). Report one INFO naming it, then fall back to full Step 3 judgment for the description, body-discipline and patterns dimensions — the prefilter's whole coverage. Judge those by rubric rather than dropping them.
A path argument that does not exist is a hard error in vale-wrap.sh, deliberately: bare vale would fall back to reading stdin and print a clean-looking 0 errors ... in stdin, which the 0 files guard above does not catch.

8.3 KiB Raw Blame History Unescape Escape