fix(skill-audit): flag a changed provenance claim, not just its shape
validate-provenance.sh checked that a sources.md entry was internally consistent -- slugs resolve, Contributing files exist, back-references match -- but never whether the asserted contribution was true. A retrofit once turned an honest hedge into a false confident claim and every existing check passed it silently. A literal-filename cross-check (flag a description naming a .md file absent from Contributing files) was tried and rejected: 3/95 flagged against the real corpus, all three false positives, and it would not have caught the actual bug -- the bad description never named a literal filename. No bash script can verify semantic truth, so the fix uses what git can reliably detect -- a changed field -- purely as a trigger for what can verify semantics: the auditor reading the files. New check 9 flags (INFO only, never FAIL) any Description or Contributing-files text change against a base ref (default: merge-base with origin/main, overridable via --base-ref). A slug absent at the base ref is a creation, not a change, and is not flagged. skill-audit's rubric now tells the auditor a check-9 INFO means open the named files and verify by reading, not just relay it. skill-author's retrofit checklist gained a matching authoring-time guardrail: don't upgrade a hedge into a confident claim without re-reading the source first. 8 new bats tests (73 total, 0 failures). Fixes: #118 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeH8SCbcrCAQrtymkNuhKP
This commit is contained in:
@@ -105,6 +105,16 @@ them for you. After every retrofit that adds, removes or renames a file:
|
||||
zero.
|
||||
- [ ] Re-run `/skill-audit` and confirm its `### Provenance` dimension does not report the new
|
||||
file as missing `source_keys`.
|
||||
- [ ] **Compression must not add authority the source text didn't have.** The bullet above is
|
||||
about a `sources.md` entry going *stale* — Contributing files left uncited after content
|
||||
moves. This is a distinct failure: a compression or rewrite pass that upgrades an honest
|
||||
hedge in a Description into an unsupported confident claim, without the underlying source
|
||||
having changed at all — "no forge-specific content drawn directly from it beyond that"
|
||||
quietly becoming "Grounds Step 2's dispatch table." Nothing in `/skill-audit`'s structural
|
||||
checks catches this; a bash script can verify an entry is internally consistent, never
|
||||
whether the claim is *true*. If a retrofit strengthens or otherwise changes the wording of a
|
||||
provenance claim, re-read the upstream research doc first and confirm the stronger wording
|
||||
is actually still true before committing it.
|
||||
|
||||
## Worked example — a description retrofit
|
||||
|
||||
|
||||
Reference in New Issue
Block a user