fix(skill-audit): make check 9 reachable, wrap-safe and never silently skipped
Check 9 shipped in #130 to close #118, but three defects meant it could not do the job it was added for. Why: - It is INFO-only, so it always exits 0 — and SKILL.md graded exit 0 "a genuine pass" and said the script "prints nothing on success". Every check-9 INFO was discarded before it reached a report, behind three further doors that only opened on a non-zero exit. - `parse_field_raw()` matched `(.+)`, which does not span newlines, so only the first physical line of a wrapped value was compared. Rewriting only the continuation line of a wrapped Description from a hedge to a confident claim produced no finding at all — verbatim the regression #118 was filed about. The bullet branch had the same shape: a wrapped bullet broke the loop and dropped every later entry. - A `git show` failure at the base ref was treated as "creation, nothing to flag" and skipped the whole skill with no output, collapsing "absent at that ref" with "not tracked under that name". A gitignored `.claude/skills/` copy reported clean while the authoring path reported four changed claims. The script's own usage text promises this is "never a silent skip". Implementation notes: - Exit-code guidance re-keyed on output as well as code: 0-and-silent passes, 0-with-output is INFO-only findings, 1 is FAILs, 2 never ran. - `parse_field_raw()` is line-based and joins continuation lines; `normalize_field_text()`'s docstring is now true rather than aspirational. A reorder deliberately fires: the two fields share one parser, and order-insensitivity would mean splitting a prose Description on commas. - The discarded `show_err` is now surfaced as one whole-check INFO naming both readings. - `--base-ref=` given empty now beats the env var, as the usage text always claimed. `validate.sh` gains an ADR-0022 `metadata.version` check at FAIL tier, because any lower tier lets skill-author Step 4 report done on a file the commit gate then refuses. Its `read` heuristic now skips here-doc bodies — reflowing the one offending line would have cleared the finding and left the cause, since every usage() heredoc is one wrap from putting the English verb in column 0. Impact: provenance tests 73 -> 82, validate tests 64 -> 72. Test 72 previously deleted origin/main before asserting the override, so it proved the flag works with no default rather than that it beats one; it now moves origin/main forward first. Refs: #118 ADR: 0022 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeH8SCbcrCAQrtymkNuhKP
This commit is contained in:
@@ -6,8 +6,9 @@ source_keys:
|
||||
|
||||
# Validation Scripts Reference
|
||||
|
||||
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting.
|
||||
Nothing here is needed on a clean run.
|
||||
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting —
|
||||
including `validate-provenance.sh` exiting **0 having printed something**, which is INFO findings,
|
||||
not a clean run. Its silent exit 0 is the only outcome that needs nothing here.
|
||||
|
||||
## Report the gap, do not guess
|
||||
|
||||
@@ -106,18 +107,33 @@ Three ways to read the result wrong:
|
||||
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
|
||||
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
|
||||
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
|
||||
Check 9 diffs the current `references/sources.md` against a base ref (default: the merge base with
|
||||
`origin/main`) and flags a slug whose `Description` or `Contributing files` text differs. It is
|
||||
structurally incapable of telling you whether the new wording is still *true* — it only detects
|
||||
that the text changed — so when this INFO fires, open the Contributing files it names and the
|
||||
document named in that slug's `Research doc:` field, and confirm by reading whether the (possibly
|
||||
Check 9 diffs the current `references/sources.md` against a base ref and flags a slug whose
|
||||
`Description` or `Contributing files` text differs. It is structurally incapable of telling you
|
||||
whether the new wording is still *true* — it only detects that the text changed — so when this
|
||||
INFO fires, open that slug's own entry: the document named in its `Research doc:` field, and the
|
||||
files its `Contributing files` list names. Read whichever the changed field is a claim *about* —
|
||||
a Description-only change often leaves the file list untouched, so "open the Contributing files"
|
||||
is where to look, not proof that they are what moved. Confirm by reading whether the (possibly
|
||||
strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
|
||||
for you: every other check here is a structural fact you can relay as-is, but check 9's job is
|
||||
only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
|
||||
without opening those files is not auditing it. A single INFO naming "no base ref could be
|
||||
resolved" or "no repo root above the skill directory" is the same graceful-skip pattern as every
|
||||
other check here that cannot run — treat it as an unaudited dimension for that reason, not as a
|
||||
finding about the skill.
|
||||
without opening those files is not auditing it. Its companion — `'<field>' removed for '<slug>'
|
||||
since <ref>` — is the same obligation in the other direction: a claim withdrawn rather than
|
||||
rewritten. No other check here requires the field, so confirm the removal was deliberate.
|
||||
- **The check-9 base ref defaults to `git merge-base HEAD origin/main`, and there are two ways to
|
||||
override it.** `--base-ref=<ref>` on the command line, or the `VALIDATE_PROVENANCE_BASE_REF`
|
||||
environment variable; the flag wins when both are given, including when it is given empty
|
||||
(`--base-ref=`), which selects the default resolution and ignores the environment. Reach for one
|
||||
on a fork, a long-lived branch, or a mirror whose remote is not called `origin` — and when a
|
||||
review asks what changed since a specific commit rather than since the branch point.
|
||||
- **A single check-9 INFO naming a whole-check skip is an unaudited dimension, not a finding about
|
||||
the skill.** There are three: "no repo root above the skill directory", "no base ref could be
|
||||
resolved", and "`<path>` is not tracked at `<ref>`". The third is the one to read carefully — it
|
||||
fires when the base ref resolved but `git show <ref>:<path>` did not, which covers both a
|
||||
genuinely new `sources.md` (nothing to flag) and a path git does not know under that name: a
|
||||
renamed skill directory, or an installed, gitignored copy such as a deployed `.claude/skills/`
|
||||
tree. Auditing the deployed copy silently checks nothing; re-run against the authoring path under
|
||||
`plugins/*/.apm/skills/`.
|
||||
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
||||
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
|
||||
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
|
||||
|
||||
Reference in New Issue
Block a user