fix(skill-audit): make check 9 reachable, wrap-safe and never silently skipped

Check 9 shipped in #130 to close #118, but three defects meant it could not do the job it was
added for.

Why:
- It is INFO-only, so it always exits 0 — and SKILL.md graded exit 0 "a genuine pass" and said the
  script "prints nothing on success". Every check-9 INFO was discarded before it reached a report,
  behind three further doors that only opened on a non-zero exit.
- `parse_field_raw()` matched `(.+)`, which does not span newlines, so only the first physical line
  of a wrapped value was compared. Rewriting only the continuation line of a wrapped Description
  from a hedge to a confident claim produced no finding at all — verbatim the regression #118 was
  filed about. The bullet branch had the same shape: a wrapped bullet broke the loop and dropped
  every later entry.
- A `git show` failure at the base ref was treated as "creation, nothing to flag" and skipped the
  whole skill with no output, collapsing "absent at that ref" with "not tracked under that name".
  A gitignored `.claude/skills/` copy reported clean while the authoring path reported four changed
  claims. The script's own usage text promises this is "never a silent skip".

Implementation notes:
- Exit-code guidance re-keyed on output as well as code: 0-and-silent passes, 0-with-output is
  INFO-only findings, 1 is FAILs, 2 never ran.
- `parse_field_raw()` is line-based and joins continuation lines; `normalize_field_text()`'s
  docstring is now true rather than aspirational. A reorder deliberately fires: the two fields share
  one parser, and order-insensitivity would mean splitting a prose Description on commas.
- The discarded `show_err` is now surfaced as one whole-check INFO naming both readings.
- `--base-ref=` given empty now beats the env var, as the usage text always claimed.

`validate.sh` gains an ADR-0022 `metadata.version` check at FAIL tier, because any lower tier lets
skill-author Step 4 report done on a file the commit gate then refuses. Its `read` heuristic now
skips here-doc bodies — reflowing the one offending line would have cleared the finding and left
the cause, since every usage() heredoc is one wrap from putting the English verb in column 0.

Impact: provenance tests 73 -> 82, validate tests 64 -> 72. Test 72 previously deleted origin/main
before asserting the override, so it proved the flag works with no default rather than that it beats
one; it now moves origin/main forward first.

Refs: #118
ADR: 0022
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeH8SCbcrCAQrtymkNuhKP
This commit is contained in:
2026-09-09 05:15:05 +00:00
parent ed8c99efbd
commit 175ea89c0a
8 changed files with 704 additions and 65 deletions

View File

@@ -6,8 +6,9 @@ source_keys:
# Validation Scripts Reference
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting.
Nothing here is needed on a clean run.
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting —
including `validate-provenance.sh` exiting **0 having printed something**, which is INFO findings,
not a clean run. Its silent exit 0 is the only outcome that needs nothing here.
## Report the gap, do not guess
@@ -106,18 +107,33 @@ Three ways to read the result wrong:
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
Check 9 diffs the current `references/sources.md` against a base ref (default: the merge base with
`origin/main`) and flags a slug whose `Description` or `Contributing files` text differs. It is
structurally incapable of telling you whether the new wording is still *true* — it only detects
that the text changed — so when this INFO fires, open the Contributing files it names and the
document named in that slug's `Research doc:` field, and confirm by reading whether the (possibly
Check 9 diffs the current `references/sources.md` against a base ref and flags a slug whose
`Description` or `Contributing files` text differs. It is structurally incapable of telling you
whether the new wording is still *true* — it only detects that the text changed — so when this
INFO fires, open that slug's own entry: the document named in its `Research doc:` field, and the
files its `Contributing files` list names. Read whichever the changed field is a claim *about* —
a Description-only change often leaves the file list untouched, so "open the Contributing files"
is where to look, not proof that they are what moved. Confirm by reading whether the (possibly
strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
for you: every other check here is a structural fact you can relay as-is, but check 9's job is
only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
without opening those files is not auditing it. A single INFO naming "no base ref could be
resolved" or "no repo root above the skill directory" is the same graceful-skip pattern as every
other check here that cannot run — treat it as an unaudited dimension for that reason, not as a
finding about the skill.
without opening those files is not auditing it. Its companion — `'<field>' removed for '<slug>'
since <ref>` — is the same obligation in the other direction: a claim withdrawn rather than
rewritten. No other check here requires the field, so confirm the removal was deliberate.
- **The check-9 base ref defaults to `git merge-base HEAD origin/main`, and there are two ways to
override it.** `--base-ref=<ref>` on the command line, or the `VALIDATE_PROVENANCE_BASE_REF`
environment variable; the flag wins when both are given, including when it is given empty
(`--base-ref=`), which selects the default resolution and ignores the environment. Reach for one
on a fork, a long-lived branch, or a mirror whose remote is not called `origin` — and when a
review asks what changed since a specific commit rather than since the branch point.
- **A single check-9 INFO naming a whole-check skip is an unaudited dimension, not a finding about
the skill.** There are three: "no repo root above the skill directory", "no base ref could be
resolved", and "`<path>` is not tracked at `<ref>`". The third is the one to read carefully — it
fires when the base ref resolved but `git show <ref>:<path>` did not, which covers both a
genuinely new `sources.md` (nothing to flag) and a path git does not know under that name: a
renamed skill directory, or an installed, gitignored copy such as a deployed `.claude/skills/`
tree. Auditing the deployed copy silently checks nothing; re-run against the authoring path under
`plugins/*/.apm/skills/`.
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.