fix(skill-audit): flag a changed provenance claim, not just its shape
validate-provenance.sh checked that a sources.md entry was internally consistent -- slugs resolve, Contributing files exist, back-references match -- but never whether the asserted contribution was true. A retrofit once turned an honest hedge into a false confident claim and every existing check passed it silently. A literal-filename cross-check (flag a description naming a .md file absent from Contributing files) was tried and rejected: 3/95 flagged against the real corpus, all three false positives, and it would not have caught the actual bug -- the bad description never named a literal filename. No bash script can verify semantic truth, so the fix uses what git can reliably detect -- a changed field -- purely as a trigger for what can verify semantics: the auditor reading the files. New check 9 flags (INFO only, never FAIL) any Description or Contributing-files text change against a base ref (default: merge-base with origin/main, overridable via --base-ref). A slug absent at the base ref is a creation, not a change, and is not flagged. skill-audit's rubric now tells the auditor a check-9 INFO means open the named files and verify by reading, not just relay it. skill-author's retrofit checklist gained a matching authoring-time guardrail: don't upgrade a hedge into a confident claim without re-reading the source first. 8 new bats tests (73 total, 0 failures). Fixes: #118 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeH8SCbcrCAQrtymkNuhKP
This commit is contained in:
@@ -105,6 +105,19 @@ Three ways to read the result wrong:
|
||||
is not a directory, a directory holding no `SKILL.md`, a missing or extra argument, and an absent
|
||||
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
|
||||
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
|
||||
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
|
||||
Check 9 diffs the current `references/sources.md` against a base ref (default: the merge base with
|
||||
`origin/main`) and flags a slug whose `Description` or `Contributing files` text differs. It is
|
||||
structurally incapable of telling you whether the new wording is still *true* — it only detects
|
||||
that the text changed — so when this INFO fires, open the Contributing files it names and the
|
||||
document named in that slug's `Research doc:` field, and confirm by reading whether the (possibly
|
||||
strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
|
||||
for you: every other check here is a structural fact you can relay as-is, but check 9's job is
|
||||
only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
|
||||
without opening those files is not auditing it. A single INFO naming "no base ref could be
|
||||
resolved" or "no repo root above the skill directory" is the same graceful-skip pattern as every
|
||||
other check here that cannot run — treat it as an unaudited dimension for that reason, not as a
|
||||
finding about the skill.
|
||||
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
||||
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
|
||||
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.
|
||||
|
||||
Reference in New Issue
Block a user