fix(skill-audit): flag a changed provenance claim, not just its shape

validate-provenance.sh checked that a sources.md entry was internally
consistent -- slugs resolve, Contributing files exist, back-references
match -- but never whether the asserted contribution was true. A
retrofit once turned an honest hedge into a false confident claim and
every existing check passed it silently.

A literal-filename cross-check (flag a description naming a .md file
absent from Contributing files) was tried and rejected: 3/95 flagged
against the real corpus, all three false positives, and it would not
have caught the actual bug -- the bad description never named a
literal filename. No bash script can verify semantic truth, so the fix
uses what git can reliably detect -- a changed field -- purely as a
trigger for what can verify semantics: the auditor reading the files.

New check 9 flags (INFO only, never FAIL) any Description or
Contributing-files text change against a base ref (default: merge-base
with origin/main, overridable via --base-ref). A slug absent at the
base ref is a creation, not a change, and is not flagged. skill-audit's
rubric now tells the auditor a check-9 INFO means open the named files
and verify by reading, not just relay it. skill-author's retrofit
checklist gained a matching authoring-time guardrail: don't upgrade a
hedge into a confident claim without re-reading the source first.

8 new bats tests (73 total, 0 failures).

Fixes: #118
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeH8SCbcrCAQrtymkNuhKP
This commit is contained in:
2026-09-07 20:37:03 +00:00
parent 2c6ce438b6
commit 09eea5e7ab
4 changed files with 404 additions and 16 deletions

View File

@@ -105,6 +105,16 @@ them for you. After every retrofit that adds, removes or renames a file:
zero.
- [ ] Re-run `/skill-audit` and confirm its `### Provenance` dimension does not report the new
file as missing `source_keys`.
- [ ] **Compression must not add authority the source text didn't have.** The bullet above is
about a `sources.md` entry going *stale* — Contributing files left uncited after content
moves. This is a distinct failure: a compression or rewrite pass that upgrades an honest
hedge in a Description into an unsupported confident claim, without the underlying source
having changed at all — "no forge-specific content drawn directly from it beyond that"
quietly becoming "Grounds Step 2's dispatch table." Nothing in `/skill-audit`'s structural
checks catches this; a bash script can verify an entry is internally consistent, never
whether the claim is *true*. If a retrofit strengthens or otherwise changes the wording of a
provenance claim, re-read the upstream research doc first and confirm the stronger wording
is actually still true before committing it.
## Worked example — a description retrofit