holocron/plugins/kyberforge/skills/skill-audit/references/validation-scripts.md

source_keys

source_keys
agentskills-spec agentskills-using-scripts

Validation Scripts Reference

Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting. Nothing here is needed on a clean run.

Report the gap, do not guess

If a script cannot run at all — Bash denied, python3 unavailable, PyYAML not importable, vale not installed — say so as an INFO finding naming the script and the missing dependency, then fall back to the manual checks below. An INFO never changes PASS/FAIL. Silently omitting the dimension a script would have covered reports a clean audit that checked less than it claims to have checked, and the Step 4 coverage line then names a dimension nothing actually examined.

Manual structural fallback

validate.sh needs python3 and PyYAML, and refuses to start without either — the description value has to be measured after YAML folding is resolved, so skipping the ADR-0020 gates would be a vacuous pass rather than a partial one. The two are checked separately, so the message already names the right one — report it verbatim rather than diagnosing further:

Error: python3 is required but was not found on PATH.
Error: PyYAML is required but is not importable by python3.

Without them — or with Bash denied, or on a permission error — work this list by hand and file the results under ### Structure exactly as the script's output would have been:

• name present, 1–64 characters, kebab-case (lowercase letters, digits and hyphens; no leading, trailing or doubled hyphen), and matching the skill's directory name exactly.
• description present and non-empty; no unfilled FILL IN: placeholder in it. An absent or empty description is a FAIL, never a silent skip — it is the one field preloaded into every session, so a skill without one can never be routed to.
• Description length, measured on the folded YAML value with newlines collapsed to single spaces — not on the raw block scalar, which counts indentation. 250 characters SUGGESTION, 400 FAIL (ADR-0020), 1,024 FAIL (agentskills.io spec).
• Body length, counting everything after the frontmatter's closing ---. 600 words SUGGESTION, 900 FAIL (ADR-0020).
• Whole-file ceilings, counting the file including frontmatter: 500 lines FAIL, 2,770 words FAIL (agentskills.io spec). These are a different measurement from the two above — report them as separate findings, never merged.
• A boundary clause is present — either the prose form (do not / instead / rather than / not for) or ADR-0020's compressed Not <thing> -> <name> arrow. SUGGESTION, not FAIL: the absence is deterministic, but whether this skill warrants one is the auditor's call.
• Boundary targets resolve — FAIL on a name that resolves to nothing. See the section below; resolving these by hand is the one item on this list with a procedure of its own.
• Every references/<file>.md named in the body exists on disk — FAIL, not a suggestion. A dispatch table or "read X" trigger naming a missing file sends the agent nowhere. Ignore mentions inside fenced code blocks, and ignore a mention whose own line says the file is gone (removed, deleted, renamed, superseded, replaced, obsolete, deprecated, former, gone, no longer, used to) — that is a historical note, not a dispatch entry.
• Gotchas discipline, both SUGGESTION. Locate the section by a heading that is Gotchas (## Common Gotchas counts; ## Gotcha handling and ## Why gotchas matter do not), running to the next heading at the same level or shallower. More than five top-level entries is one suggestion; a section over 25% of the body word count is a second, independent one. Count entries at column 0 only — an indented child bullet is not an entry — and ignore fenced code blocks for both.
• No unfilled FILL IN: placeholder anywhere in the body.
• Every file in scripts/ carries the executable bit and contains no interactive prompt — no bare read, no select, nothing that blocks on a TTY.

Resolving boundary targets by hand

Targets are read from both boundary forms. The compressed Not <thing> -> <name> arrow and the prose form are each parsed and target-checked, so a typo in prose phrasing fails exactly as an arrow typo does — do not check only the names after an arrow.

Build the universe by walking up from the SKILL.md under audit, never from the validator's own location. The nearest ancestor holding plugins/*/.apm/skills/ or plugins/*/.apm/agents/ is the authoring root, falling back to the nearest ancestor holding .git. When one is found the universe is every skill and agent under <root>/plugins/*/, plus the skill's own apm package, plus the packages that package declares in its apm.yml under dependencies.apm. Deployed .claude/ and .agents/ trees are consulted only when no authoring root exists — they are gitignored apm install output, and reading them would make a fresh clone and a developer machine disagree.

Three ways to read the result wrong:

• A hyphenated name used attributively is not a dangling target. "Use pre-commit hooks instead of ad-hoc scripts" reads as a route to pre-commit on wording alone. What separates a route from prose is grammar: a route target is terminal — followed by punctuation, a conjunction, or a boundary word — whereas a compound modifier is followed by the noun it modifies. A name followed by an ordinary noun still confirms a route when it exists, but never raises a FAIL on its own.
• A SUGGESTION-tier unresolved target is not a FAIL you may promote. Terminal position alone is not evidence of a route: "run pre-commit instead", "see commit-msg" and "use the clean-up instead" are all terminal and all prose. A prose-form target earns a FAIL only when its own sentence names another target that does resolve; otherwise the script reports it and moves on, and so should you. Route notation — /name and -> name — is exempt and always FAILs, and it is the fix to recommend when the author did mean a route.
• INFO boundary-target resolution DID NOT RUN is not a pass. The script prints it, and exits 0, when no universe could be determined for that path — the usual cause being a skill copy audited outside its package. Report it as an INFO naming the unchecked targets and re-run against the real directory; filing it as clean signs off targets nothing verified.

Script-specific failures

• validate-provenance.sh printed nothing and exited 0. That is a pass, not a skip — it exits 0 silently when the skill has no source_keys and no references/sources.md, and nothing to validate is not a finding. Check the exit code before you believe the silence: a target that is not a directory, a directory holding no SKILL.md, a missing or extra argument, and an absent python3 all exit 2 with a message on stderr. Exit 2 means the script never ran — report it as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
• A check-9 INFO — '<field>' changed for '<slug>' since <ref> — means go read, not just relay. Check 9 diffs the current references/sources.md against a base ref (default: the merge base with origin/main) and flags a slug whose Description or Contributing files text differs. It is structurally incapable of telling you whether the new wording is still true — it only detects that the text changed — so when this INFO fires, open the Contributing files it names and the document named in that slug's Research doc: field, and confirm by reading whether the (possibly strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify for you: every other check here is a structural fact you can relay as-is, but check 9's job is only to tell you where to spend that reading effort, not to replace it. Acknowledging the INFO without opening those files is not auditing it. A single INFO naming "no base ref could be resolved" or "no repo root above the skill directory" is the same graceful-skip pattern as every other check here that cannot run — treat it as an unaudited dimension for that reason, not as a finding about the skill.
• vale reports 0 files. Treat the pass as NOT RUN, not as clean, and fall back to full Step 3 judgment for the dimensions it would have covered. The bundled Kyberforge style is scoped by glob in assets/vale/.vale.ini; a file outside those globs is silently not linted.
• E100 Runtime error ... does not exist (exit 2) from vale-wrap.sh. An explicit relative --config was passed. Pass none: the wrapper locates its own assets/vale/.vale.ini from its own path, so a resolved script path plus an unresolved config path produces exactly this. Do not read this exit code as vale being unavailable — that misreading sends the audit down the fallback path while vale was installed and working the whole time.
• The vale binary is genuinely absent (command not found). Report one INFO naming it, then fall back to full Step 3 judgment for the description, body-discipline and patterns dimensions — the prefilter's whole coverage. Judge those by rubric rather than dropping them.
• A path argument that does not exist is a hard error in vale-wrap.sh, deliberately: bare vale would fall back to reading stdin and print a clean-looking 0 errors ... in stdin, which the 0 files guard above does not catch.