holocron/plugins/kyberforge/skills/skill-audit/references/validation-scripts.md

source_keys

source_keys
agentskills-spec agentskills-using-scripts

Validation Scripts Reference

Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting — including validate-provenance.sh exiting 0 having printed something, which is INFO findings, not a clean run. Its silent exit 0 is the only outcome that needs nothing here.

Report the gap, do not guess

If a script cannot run at all — Bash denied, python3 unavailable, PyYAML not importable, vale not installed — say so as an INFO finding naming the script and the missing dependency, then fall back to the manual checks below. An INFO never changes PASS/FAIL. Silently omitting the dimension a script would have covered reports a clean audit that checked less than it claims to have checked, and the Step 4 coverage line then names a dimension nothing actually examined.

Manual structural fallback

validate.sh needs python3 and PyYAML, and refuses to start without either — the description value has to be measured after YAML folding is resolved, so skipping the ADR-0020 gates would be a vacuous pass rather than a partial one. The two are checked separately, so the message already names the right one — report it verbatim rather than diagnosing further:

Error: python3 is required but was not found on PATH.
Error: PyYAML is required but is not importable by python3.

Without them — or with Bash denied, or on a permission error — work this list by hand and file the results under ### Structure exactly as the script's output would have been:

• name present, 1–64 characters, kebab-case (lowercase letters, digits and hyphens; no leading, trailing or doubled hyphen), and matching the skill's directory name exactly.
• description present and non-empty; no unfilled FILL IN: placeholder in it. An absent or empty description is a FAIL, never a silent skip — it is the one field preloaded into every session, so a skill without one can never be routed to.
• Description length, measured on the folded YAML value with newlines collapsed to single spaces — not on the raw block scalar, which counts indentation. 250 characters SUGGESTION, 400 FAIL (ADR-0020), 1,024 FAIL (agentskills.io spec).
• Body length, counting everything after the frontmatter's closing ---. 600 words SUGGESTION, 900 FAIL (ADR-0020).
• Whole-file ceilings, counting the file including frontmatter: 500 lines FAIL, 2,770 words FAIL (agentskills.io spec). These are a different measurement from the two above — report them as separate findings, never merged.
• A boundary clause is present — either the prose form (do not / instead / rather than / not for) or ADR-0020's compressed Not <thing> -> <name> arrow. SUGGESTION, not FAIL: the absence is deterministic, but whether this skill warrants one is the auditor's call.
• Boundary targets resolve — FAIL on a name that resolves to nothing. See the section below; resolving these by hand is the one item on this list with a procedure of its own.
• Every references/<file>.md named in the body exists on disk — FAIL, not a suggestion. A dispatch table or "read X" trigger naming a missing file sends the agent nowhere. Ignore mentions inside fenced code blocks, and ignore a mention whose own line says the file is gone (removed, deleted, renamed, superseded, replaced, obsolete, deprecated, former, gone, no longer, used to) — that is a historical note, not a dispatch entry.
• Gotchas discipline, both SUGGESTION. Locate the section by a heading that is Gotchas (## Common Gotchas counts; ## Gotcha handling and ## Why gotchas matter do not), running to the next heading at the same level or shallower. More than five top-level entries is one suggestion; a section over 25% of the body word count is a second, independent one. Count entries at column 0 only — an indented child bullet is not an entry — and ignore fenced code blocks for both.
• No unfilled FILL IN: placeholder anywhere in the body.
• Every file in scripts/ carries the executable bit and contains no interactive prompt — no bare read, no select, nothing that blocks on a TTY.

Resolving boundary targets by hand

Targets are read from both boundary forms. The compressed Not <thing> -> <name> arrow and the prose form are each parsed and target-checked, so a typo in prose phrasing fails exactly as an arrow typo does — do not check only the names after an arrow.

Build the universe by walking up from the SKILL.md under audit, never from the validator's own location. The nearest ancestor holding plugins/*/.apm/skills/ or plugins/*/.apm/agents/ is the authoring root, falling back to the nearest ancestor holding .git. When one is found the universe is every skill and agent under <root>/plugins/*/, plus the skill's own apm package, plus the packages that package declares in its apm.yml under dependencies.apm. Deployed .claude/ and .agents/ trees are consulted only when no authoring root exists — they are gitignored apm install output, and reading them would make a fresh clone and a developer machine disagree.

Three ways to read the result wrong:

• A hyphenated name used attributively is not a dangling target. "Use pre-commit hooks instead of ad-hoc scripts" reads as a route to pre-commit on wording alone. What separates a route from prose is grammar: a route target is terminal — followed by punctuation, a conjunction, or a boundary word — whereas a compound modifier is followed by the noun it modifies. A name followed by an ordinary noun still confirms a route when it exists, but never raises a FAIL on its own.
• A SUGGESTION-tier unresolved target is not a FAIL you may promote. Terminal position alone is not evidence of a route: "run pre-commit instead", "see commit-msg" and "use the clean-up instead" are all terminal and all prose. A prose-form target earns a FAIL only when its own sentence names another target that does resolve; otherwise the script reports it and moves on, and so should you. Route notation — /name and -> name — is exempt and always FAILs, and it is the fix to recommend when the author did mean a route.
• INFO boundary-target resolution DID NOT RUN is not a pass. The script prints it, and exits 0, when no universe could be determined for that path — the usual cause being a skill copy audited outside its package. Report it as an INFO naming the unchecked targets and re-run against the real directory; filing it as clean signs off targets nothing verified.

Script-specific failures

• validate-provenance.sh printed nothing and exited 0. That is a pass, not a skip — it exits 0 silently when the skill has no source_keys and no references/sources.md, and nothing to validate is not a finding. Check the exit code before you believe the silence: a target that is not a directory, a directory holding no SKILL.md, a missing or extra argument, and an absent python3 all exit 2 with a message on stderr. Exit 2 means the script never ran — report it as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
• A check-9 INFO — '<field>' changed for '<slug>' since <ref> — means go read, not just relay. Check 9 diffs the current references/sources.md against a base ref and flags a slug whose Description or Contributing files text differs. It is structurally incapable of telling you whether the new wording is still true — it only detects that the text changed — so when this INFO fires, open that slug's own entry: the document named in its Research doc: field, and the files its Contributing files list names. Read whichever the changed field is a claim about — a Description-only change often leaves the file list untouched, so "open the Contributing files" is where to look, not proof that they are what moved. Confirm by reading whether the (possibly strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify for you: every other check here is a structural fact you can relay as-is, but check 9's job is only to tell you where to spend that reading effort, not to replace it. Acknowledging the INFO without opening those files is not auditing it. Its companion — '<field>' removed for '<slug>' since <ref> — is the same obligation in the other direction: a claim withdrawn rather than rewritten. No other check here requires the field, so confirm the removal was deliberate.
• The check-9 base ref defaults to git merge-base HEAD origin/main, and there are two ways to override it. --base-ref=<ref> on the command line, or the VALIDATE_PROVENANCE_BASE_REF environment variable; the flag wins when both are given, including when it is given empty (--base-ref=), which selects the default resolution and ignores the environment. Reach for one on a fork, a long-lived branch, or a mirror whose remote is not called origin — and when a review asks what changed since a specific commit rather than since the branch point.
• A single check-9 INFO naming a whole-check skip is an unaudited dimension, not a finding about the skill. There are three: "no repo root above the skill directory", "no base ref could be resolved", and "<path> is not tracked at <ref>". The third is the one to read carefully — it fires when the base ref resolved but git show <ref>:<path> did not, which covers both a genuinely new sources.md (nothing to flag) and a path git does not know under that name: a renamed skill directory, or an installed, gitignored copy such as a deployed .claude/skills/ tree. Auditing the deployed copy silently checks nothing; re-run against the authoring path under plugins/*/.apm/skills/.
• vale reports 0 files. Treat the pass as NOT RUN, not as clean, and fall back to full Step 3 judgment for the dimensions it would have covered. The bundled Kyberforge style is scoped by glob in assets/vale/.vale.ini; a file outside those globs is silently not linted.
• E100 Runtime error ... does not exist (exit 2) from vale-wrap.sh. An explicit relative --config was passed. Pass none: the wrapper locates its own assets/vale/.vale.ini from its own path, so a resolved script path plus an unresolved config path produces exactly this. Do not read this exit code as vale being unavailable — that misreading sends the audit down the fallback path while vale was installed and working the whole time.
• The vale binary is genuinely absent (command not found). Report one INFO naming it, then fall back to full Step 3 judgment for the description, body-discipline and patterns dimensions — the prefilter's whole coverage. Judge those by rubric rather than dropping them.
• A path argument that does not exist is a hard error in vale-wrap.sh, deliberately: bare vale would fall back to reading stdin and print a clean-looking 0 errors ... in stdin, which the 0 files guard above does not catch.