fix(skill-audit): flag a changed provenance claim, not just its shape

validate-provenance.sh checked that a sources.md entry was internally
consistent -- slugs resolve, Contributing files exist, back-references
match -- but never whether the asserted contribution was true. A
retrofit once turned an honest hedge into a false confident claim and
every existing check passed it silently.

A literal-filename cross-check (flag a description naming a .md file
absent from Contributing files) was tried and rejected: 3/95 flagged
against the real corpus, all three false positives, and it would not
have caught the actual bug -- the bad description never named a
literal filename. No bash script can verify semantic truth, so the fix
uses what git can reliably detect -- a changed field -- purely as a
trigger for what can verify semantics: the auditor reading the files.

New check 9 flags (INFO only, never FAIL) any Description or
Contributing-files text change against a base ref (default: merge-base
with origin/main, overridable via --base-ref). A slug absent at the
base ref is a creation, not a change, and is not flagged. skill-audit's
rubric now tells the auditor a check-9 INFO means open the named files
and verify by reading, not just relay it. skill-author's retrofit
checklist gained a matching authoring-time guardrail: don't upgrade a
hedge into a confident claim without re-reading the source first.

8 new bats tests (73 total, 0 failures).

Fixes: #118
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeH8SCbcrCAQrtymkNuhKP
This commit is contained in:
2026-09-07 20:37:03 +00:00
parent 2c6ce438b6
commit 09eea5e7ab
4 changed files with 404 additions and 16 deletions

View File

@@ -105,6 +105,19 @@ Three ways to read the result wrong:
is not a directory, a directory holding no `SKILL.md`, a missing or extra argument, and an absent
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
Check 9 diffs the current `references/sources.md` against a base ref (default: the merge base with
`origin/main`) and flags a slug whose `Description` or `Contributing files` text differs. It is
structurally incapable of telling you whether the new wording is still *true* — it only detects
that the text changed — so when this INFO fires, open the Contributing files it names and the
document named in that slug's `Research doc:` field, and confirm by reading whether the (possibly
strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
for you: every other check here is a structural fact you can relay as-is, but check 9's job is
only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
without opening those files is not auditing it. A single INFO naming "no base ref could be
resolved" or "no repo root above the skill directory" is the same graceful-skip pattern as every
other check here that cannot run — treat it as an unaudited dimension for that reason, not as a
finding about the skill.
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.