holocron/plugins/kyberforge/skills/skill-audit/references/validation-scripts.md

---
source_keys:
  - agentskills-spec
  - agentskills-using-scripts
---

# Validation Scripts Reference

Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting —
including `validate-provenance.sh` exiting **0 having printed something**, which is INFO findings,
not a clean run. Its silent exit 0 is the only outcome that needs nothing here.

## Report the gap, do not guess

If a script cannot run at all — Bash denied, `python3` unavailable, PyYAML not importable, `vale`
not installed — say so as an **INFO** finding naming the script and the missing dependency, then
fall back to the manual checks below. An INFO never changes PASS/FAIL. Silently omitting the
dimension a script would have covered reports a clean audit that checked less than it claims to
have checked, and the Step 4 coverage line then names a dimension nothing actually examined.

## Manual structural fallback

`validate.sh` needs `python3` **and** PyYAML, and refuses to start without either — the description
value has to be measured after YAML folding is resolved, so skipping these gates would be a
vacuous pass rather than a partial one. The two are checked separately, so the message already names
the right one — report it verbatim rather than diagnosing further:

```text
Error: python3 is required but was not found on PATH.
Error: PyYAML is required but is not importable by python3.
```

Without them — or with Bash denied, or on a permission error — work this list
by hand and file the results under `### Structure` exactly as the script's output would have been:

- **`name`** present, 1–64 characters, kebab-case (lowercase letters, digits and hyphens; no
  leading, trailing or doubled hyphen), and **matching the skill's directory name** exactly.
- **`description`** present and non-empty; no unfilled `FILL IN:` placeholder in it. An absent or
  empty description is a **FAIL**, never a silent skip — it is the one field preloaded into every
  session, so a skill without one can never be routed to.
- **Description length**, measured on the folded YAML value with newlines collapsed to single
  spaces — not on the raw block scalar, which counts indentation. 250 characters SUGGESTION, 400
  FAIL (house), 1,024 FAIL (agentskills.io spec).
- **Body length**, counting everything after the frontmatter's closing `---`. 600 words
  SUGGESTION, 900 FAIL (house).
- **Whole-file ceilings**, counting the file including frontmatter: 500 lines FAIL, 2,770 words
  FAIL (agentskills.io spec). These are a different measurement from the two above — report them
  as separate findings, never merged.
- **A boundary clause is present** — either the prose form (`do not` / `instead` / `rather than` /
  `not for`) or the compressed `Not <thing> -> <name>` arrow. **SUGGESTION**, not FAIL:
  the absence is deterministic, but whether this skill warrants one is the auditor's call.
- **Boundary targets resolve** — **FAIL** on a name that resolves to nothing. See the section
  below; resolving these by hand is the one item on this list with a procedure of its own.
- **Every `references/<file>.md` named in the body exists on disk** — **FAIL**, not a suggestion.
  A dispatch table or "read X" trigger naming a missing file sends the agent nowhere. Ignore
  mentions inside fenced code blocks, and ignore a mention whose own line says the file is gone
  (`removed`, `deleted`, `renamed`, `superseded`, `replaced`, `obsolete`, `deprecated`, `former`,
  `gone`, `no longer`, `used to`) — that is a historical note, not a dispatch entry.
- **Gotchas discipline**, both **SUGGESTION**. Locate the section by a heading that *is* Gotchas
  (`## Common Gotchas` counts; `## Gotcha handling` and `## Why gotchas matter` do not), running to
  the next heading at the same level or shallower. More than five top-level entries is one
  suggestion; a section over 25% of the body word count is a second, independent one. Count
  entries at column 0 only — an indented child bullet is not an entry — and ignore fenced code
  blocks for both.
- **No unfilled `FILL IN:` placeholder** anywhere in the body.
- **Every file in `scripts/`** carries the executable bit and contains no interactive prompt —
  no bare `read`, no `select`, nothing that blocks on a TTY.

## Resolving boundary targets by hand

Targets are read from **both** boundary forms. The compressed `Not <thing> -> <name>` arrow and the
prose form are each parsed *and* target-checked, so a typo in prose phrasing fails exactly as an
arrow typo does — do not check only the names after an arrow.

Build the universe by walking up **from the `SKILL.md` under audit**, never from the validator's own
location. The nearest ancestor holding `plugins/*/.apm/skills/` or `plugins/*/.apm/agents/` is the
authoring root, falling back to the nearest ancestor holding `.git`. When one is found the universe
is every skill and agent under `<root>/plugins/*/`, plus the skill's own apm package, plus the
packages that package declares in its `apm.yml` under `dependencies.apm`. Deployed `.claude/` and
`.agents/` trees are consulted **only** when no authoring root exists — they are gitignored
`apm install` output, and reading them would make a fresh clone and a developer machine disagree.

Three ways to read the result wrong:

- **A hyphenated name used attributively is not a dangling target.** "Use pre-commit hooks instead
  of ad-hoc scripts" reads as a route to `pre-commit` on wording alone. What separates a route from
  prose is grammar: a route target is terminal — followed by punctuation, a conjunction, or a
  boundary word — whereas a compound modifier is followed by the noun it modifies. A name followed
  by an ordinary noun still *confirms* a route when it exists, but never raises a FAIL on its own.
- **A SUGGESTION-tier unresolved target is not a FAIL you may promote.** Terminal position alone is
  not evidence of a route: "run `pre-commit` instead", "see `commit-msg`" and "use the clean-up
  instead" are all terminal and all prose. A prose-form target earns a FAIL only when its own
  sentence names another target that *does* resolve; otherwise the script reports it and moves on,
  and so should you. Route notation — `/name` and `-> name` — is exempt and always FAILs, and it is
  the fix to recommend when the author did mean a route.
- **`INFO boundary-target resolution DID NOT RUN` is not a pass.** The script prints it, and exits
  0, when no universe could be determined for that path — the usual cause being a skill copy
  audited outside its package. Report it as an INFO naming the unchecked targets and re-run against
  the real directory; filing it as clean signs off targets nothing verified.

## Script-specific failures

- **`validate-provenance.sh` printed nothing *and exited 0*.** That is a pass, not a skip — it
  exits 0 silently when the skill has no `source_keys` and no `references/sources.md`, and nothing
  to validate is not a finding. Check the exit code before you believe the silence: a target that
  is not a directory, a directory holding no `SKILL.md`, a missing or extra argument, and an absent
  `python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
  as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
  Check 9 diffs the current `references/sources.md` against a base ref and flags a slug whose
  `Description` or `Contributing files` text differs. It is structurally incapable of telling you
  whether the new wording is still *true* — it only detects that the text changed — so when this
  INFO fires, open that slug's own entry: the document named in its `Research doc:` field, and the
  files its `Contributing files` list names. Read whichever the changed field is a claim *about* —
  a Description-only change often leaves the file list untouched, so "open the Contributing files"
  is where to look, not proof that they are what moved. Confirm by reading whether the (possibly
  strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
  for you: every other check here is a structural fact you can relay as-is, but check 9's job is
  only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
  without opening those files is not auditing it. Its companion — `'<field>' removed for '<slug>'
  since <ref>` — is the same obligation in the other direction: a claim withdrawn rather than
  rewritten. No other check here requires the field, so confirm the removal was deliberate.
- **The check-9 base ref defaults to `git merge-base HEAD origin/main`, and there are two ways to
  override it.** `--base-ref=<ref>` on the command line, or the `VALIDATE_PROVENANCE_BASE_REF`
  environment variable; the flag wins when both are given, including when it is given empty
  (`--base-ref=`), which selects the default resolution and ignores the environment. Reach for one
  on a fork, a long-lived branch, or a mirror whose remote is not called `origin` — and when a
  review asks what changed since a specific commit rather than since the branch point.
- **A single check-9 INFO naming a whole-check skip is an unaudited dimension, not a finding about
  the skill.** There are three: "no repo root above the skill directory", "no base ref could be
  resolved", and "`<path>` is not tracked at `<ref>`". The third is the one to read carefully — it
  fires when the base ref resolved but `git show <ref>:<path>` did not, which covers both a
  genuinely new `sources.md` (nothing to flag) and a path git does not know under that name: a
  renamed skill directory, or an installed, gitignored copy such as a deployed `.claude/skills/`
  tree. Auditing the deployed copy silently checks nothing; re-run against the authoring path under
  `plugins/*/.apm/skills/`.
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
  Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
  scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
- **`E100 Runtime error ... does not exist` (exit 2) from `vale-wrap.sh`.** An explicit relative
  `--config` was passed. Pass none: the wrapper locates its own `assets/vale/.vale.ini` from its
  own path, so a resolved script path plus an unresolved config path produces exactly this. Do not
  read this exit code as vale being unavailable — that misreading sends the audit down the
  fallback path while vale was installed and working the whole time.
- **The `vale` binary is genuinely absent** (`command not found`). Report one INFO naming it, then
  fall back to full Step 3 judgment for the description, body-discipline and patterns dimensions —
  the prefilter's whole coverage. Judge those by rubric rather than dropping them.
- **A path argument that does not exist is a hard error** in `vale-wrap.sh`, deliberately: bare
  `vale` would fall back to reading stdin and print a clean-looking `0 errors ... in stdin`, which
  the `0 files` guard above does not catch.