Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

validate-provenance.sh passes a source_keys entry that asserts a contribution which never happened #118

New Issue
Closed
opened 2026-08-30 16:50:39 +00:00 by Claude · 2 comments
Claude commented 2026-08-30 16:50:39 +00:00
Collaborator
Copy Link

Found during the #99 ADR-0020 retrofit, wave 4 (forge).

What happens

validate-provenance.sh checks that a slug exists, that Contributing files entries resolve on disk, and that each contributing file back-references its parent slug (check 5). It never checks whether the asserted contribution is true — that the named source actually contributed anything to the named file.

So an entry can be confidently wrong and the script exits 0, silently.

The concrete case

forge's retrofit rewrote references/sources.md's claude-code-plugins-docs entry:

  • Contributing files went (none) → references/apm-routes.md.
  • The description was upgraded from an accurate hedge — "Background context for Step 2's plugin/marketplace rows; no forge-specific content drawn directly from it beyond that" — to "Grounds the plugin and marketplace-entry rows of Step 2's dispatch table".

Neither is true. Nothing in apm-routes.md derives from https://code.claude.com/docs/en/plugins: its content is the ADR-0015 supersession (a repo decision), the always-inline reasoning (a repo decision), the no-recheck reasoning (a repo decision), and two apm CLI commands — which are Microsoft apm, provenanced under a different slug entirely and not in forge's chain at all.

The rewritten description is also internally inconsistent by the validator's own rule: it claims the source grounds a table that lives in SKILL.md, but SKILL.md is not listed as a contributing file, and the slug is absent from SKILL.md's metadata.source_keys. Adding SKILL.md to close that gap would then fail check 5. The two halves of the entry cannot both be right — and the script reported nothing either way.

Why it matters

Provenance exists so a later reader can trace a claim to its source. An entry that is merely incomplete degrades gracefully — you look elsewhere. An entry that asserts a wrong source sends the reader to a document that does not support the text, and it survives every automated check, so nothing ever surfaces it. It is worse than the (none) it replaced.

This is also a predictable failure mode of the retrofit work itself: a compression pass rewrites prose, and rewriting a hedge into a confident claim reads like an improvement. Wave 4 produced three separate instances of invented-during-compression claims across three skills; this is the one that lives in the provenance layer, where the gate looks like it is watching.

Suggested fix

Full semantic verification is not automatable, but two cheap things would have caught this:

  1. Cross-check the two directions. If a sources.md description names a file (e.g. "Step 2's dispatch table" → SKILL.md), that file should appear in the entry's own Contributing files. A mismatch between the prose and the structured field is mechanically detectable and was present here.
  2. Flag a Contributing files transition away from (none) as needing review in the audit rubric, rather than treating it as ordinary. (none) is a deliberate honest declaration; changing it is a new claim.

The deeper point is that skill-audit's provenance dimension should treat a changed provenance assertion as something to verify by reading, since the script structurally cannot. Worth stating in the rubric.

Files

  • plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh — check 5 and the surrounding checks
  • plugins/kyberforge/.apm/skills/forge/references/sources.md — the case above, now corrected

Related

Found during #99 wave 4. See also #111 (no way to declare a reference file house-authored — the same file, the opposite gap: honest declaration is impossible while dishonest declaration is unchecked).

Found during the #99 ADR-0020 retrofit, wave 4 (`forge`). ## What happens `validate-provenance.sh` checks that a slug exists, that `Contributing files` entries resolve on disk, and that each contributing file back-references its parent slug (check 5). It never checks whether the asserted contribution is **true** — that the named source actually contributed anything to the named file. So an entry can be confidently wrong and the script exits 0, silently. ## The concrete case `forge`'s retrofit rewrote `references/sources.md`'s `claude-code-plugins-docs` entry: - `Contributing files` went `(none)` → `references/apm-routes.md`. - The description was upgraded from an accurate hedge — *"Background context for Step 2's plugin/marketplace rows; no forge-specific content drawn directly from it beyond that"* — to *"Grounds the plugin and marketplace-entry rows of Step 2's dispatch table"*. Neither is true. Nothing in `apm-routes.md` derives from `https://code.claude.com/docs/en/plugins`: its content is the ADR-0015 supersession (a repo decision), the always-inline reasoning (a repo decision), the no-recheck reasoning (a repo decision), and two `apm` CLI commands — which are Microsoft apm, provenanced under a different slug entirely and not in `forge`'s chain at all. The rewritten description is also internally inconsistent by the validator's own rule: it claims the source grounds a table that lives in `SKILL.md`, but `SKILL.md` is not listed as a contributing file, and the slug is absent from `SKILL.md`'s `metadata.source_keys`. Adding `SKILL.md` to close that gap would then **fail** check 5. The two halves of the entry cannot both be right — and the script reported nothing either way. ## Why it matters Provenance exists so a later reader can trace a claim to its source. An entry that is merely incomplete degrades gracefully — you look elsewhere. An entry that asserts a **wrong** source sends the reader to a document that does not support the text, and it survives every automated check, so nothing ever surfaces it. It is worse than the `(none)` it replaced. This is also a predictable failure mode of the retrofit work itself: a compression pass rewrites prose, and rewriting a hedge into a confident claim reads like an improvement. Wave 4 produced three separate instances of invented-during-compression claims across three skills; this is the one that lives in the provenance layer, where the gate looks like it is watching. ## Suggested fix Full semantic verification is not automatable, but two cheap things would have caught this: 1. **Cross-check the two directions.** If a `sources.md` description names a file (e.g. "Step 2's dispatch table" → `SKILL.md`), that file should appear in the entry's own `Contributing files`. A mismatch between the prose and the structured field is mechanically detectable and was present here. 2. **Flag a `Contributing files` transition away from `(none)`** as needing review in the audit rubric, rather than treating it as ordinary. `(none)` is a deliberate honest declaration; changing it is a new claim. The deeper point is that `skill-audit`'s provenance dimension should treat a *changed* provenance assertion as something to verify by reading, since the script structurally cannot. Worth stating in the rubric. ## Files - `plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh` — check 5 and the surrounding checks - `plugins/kyberforge/.apm/skills/forge/references/sources.md` — the case above, now corrected ## Related Found during #99 wave 4. See also #111 (no way to declare a reference file house-authored — the same file, the opposite gap: honest declaration is impossible while dishonest declaration is unchecked).
Claude added the Kind/Enhancement
Priority
Medium
3
Reviewed
Confirmed
1
 labels 2026-08-30 16:50:58 +00:00
Claude added this to the Skills & Agents milestone 2026-08-30 16:51:03 +00:00
Claude commented 2026-08-31 08:10:52 +00:00
Author
Collaborator
Copy Link

Still open — but a material change on refactor/adr0020-skill-retrofit strengthens the case

The defect this issue describes is unfixed: validate-provenance.sh still cannot tell a true contribution claim from a false one. What changed is that checks 7 and 8 were effectively dead across the entire git plugin and are now live, which means the checks this issue says are insufficient were, until now, not even running.

The mechanism

A sources.md entry's Research doc: value is a path, and very often a path plus an annotation:

plugins/git/docs/research/docs/git/remotes.md → `## Pushing (`git push`)`
.../pre-commit/hooks-reference.md § "pre-commit-hooks (official collection)"

The whole string was treated as the path. It did not resolve, so checks 7-8 skipped — silently. The script now strips the annotation at the first §, → or ( (with (none) recognised before the strip, since it is itself a parenthesis), and emits an INFO naming the slug whenever checks 7-8 still cannot run, instead of declining without saying so.

How dead it was

Counted across the nine skills in plugins/git/.apm/skills/:

skill Research doc: entries annotated
git-branches 4 4
git-commits 4 4
git-history 3 0
git-remotes 5 5
git-submodules 1 1
git-workflow 5 5
git-worktrees 1 1
pc-author 4 2
pc-run 4 2

Eight of nine skills carried at least one annotated value. git-history was the only skill in the plugin where checks 7-8 ever ran — which is precisely why its three INFOs were the only ones anyone had seen.

Result of turning them on

Corpus-wide, running skill-audit/scripts/validate-provenance.sh over all 39 skills:

  • main: 7 INFO, 0 FAIL
  • branch: 36 INFO, 0 FAIL

That is 32 findings that did not exist before and zero new FAILs. No skill turns out to be missing a sources.md entry it should have had; the checks were simply not looking. (Three of main's seven INFOs were the house-authored false positives resolved by #111 — the net movement is 7 → 36.)

Why this argues for the issue rather than against it

Checks 7-8 now run across the plugin and report 32 findings, and not one of them is the failure this issue describes. The forge/claude-code-plugins-docs entry would still pass every check on the branch. The gate got measurably more thorough without getting any closer to detecting a confidently wrong provenance claim — which is the point: the two cheap cross-checks suggested here (prose naming a file that is absent from Contributing files; a transition away from (none) flagged for review) remain unimplemented, and the rubric note about treating a changed provenance assertion as read-to-verify has not been written.

One caveat: the annotation-stripping change is not pinned by a test. skill-audit/tests/validate-provenance.bats is 17 cases and none covers it.

## Still open — but a material change on `refactor/adr0020-skill-retrofit` strengthens the case The defect this issue describes is unfixed: `validate-provenance.sh` still cannot tell a true contribution claim from a false one. What changed is that **checks 7 and 8 were effectively dead across the entire `git` plugin and are now live**, which means the checks this issue says are insufficient were, until now, not even running. ### The mechanism A `sources.md` entry's `Research doc:` value is a path, and very often a path **plus** an annotation: ``` plugins/git/docs/research/docs/git/remotes.md → `## Pushing (`git push`)` .../pre-commit/hooks-reference.md § "pre-commit-hooks (official collection)" ``` The whole string was treated as the path. It did not resolve, so checks 7-8 skipped — **silently**. The script now strips the annotation at the first `§`, `→` or `(` (with `(none)` recognised before the strip, since it is itself a parenthesis), and emits an INFO naming the slug whenever checks 7-8 still cannot run, instead of declining without saying so. ### How dead it was Counted across the nine skills in `plugins/git/.apm/skills/`: | skill | `Research doc:` entries | annotated | |---|---|---| | git-branches | 4 | 4 | | git-commits | 4 | 4 | | **git-history** | **3** | **0** | | git-remotes | 5 | 5 | | git-submodules | 1 | 1 | | git-workflow | 5 | 5 | | git-worktrees | 1 | 1 | | pc-author | 4 | 2 | | pc-run | 4 | 2 | Eight of nine skills carried at least one annotated value. `git-history` was the only skill in the plugin where checks 7-8 ever ran — which is precisely why its three INFOs were the only ones anyone had seen. ### Result of turning them on Corpus-wide, running `skill-audit/scripts/validate-provenance.sh` over all 39 skills: - `main`: **7 INFO, 0 FAIL** - branch: **36 INFO, 0 FAIL** That is 32 findings that did not exist before and **zero** new FAILs. No skill turns out to be missing a `sources.md` entry it should have had; the checks were simply not looking. (Three of `main`'s seven INFOs were the house-authored false positives resolved by #111 — the net movement is 7 → 36.) ### Why this argues for the issue rather than against it Checks 7-8 now run across the plugin and report 32 findings, and not one of them is the failure this issue describes. The `forge`/`claude-code-plugins-docs` entry would still pass every check on the branch. The gate got measurably more thorough without getting any closer to detecting a confidently wrong provenance claim — which is the point: the two cheap cross-checks suggested here (prose naming a file that is absent from `Contributing files`; a transition away from `(none)` flagged for review) remain unimplemented, and the rubric note about treating a *changed* provenance assertion as read-to-verify has not been written. One caveat: the annotation-stripping change is not pinned by a test. `skill-audit/tests/validate-provenance.bats` is 17 cases and none covers it.
Claude commented 2026-09-07 20:27:10 +00:00
Author
Collaborator
Copy Link

Implementation note

Two approaches from the suggested fix were tried and rejected before implementing, worth recording so they aren't re-proposed blind:

  • Literal filename cross-check ("does the description name a .md file absent from Contributing files?") — tested empirically against the real 95-entry corpus: 3/95 flagged, all three false positives (a research-doc path mentioned as provenance-only, a research doc's own filename named while explicitly disclaiming it informed anything, a self-reference to sources.md/CONTEXT.md). It also would not have caught the actual bug this issue describes — the bad description said "Grounds... Step 2's dispatch table," never naming SKILL.md literally. Not implemented.
  • Conclusion: no bash script can verify semantic truth (the issue says this explicitly). The fix uses what git can reliably detect — a changed field — purely as a trigger for what can verify semantics: the auditor reading the files, not more text-matching.

What shipped, defense in depth across two layers:

  1. Audit-time — validate-provenance.sh check 9 (new): diffs each sources.md entry's Description:/Contributing files: text against a base ref (git merge-base HEAD origin/main by default, overridable via --base-ref/VALIDATE_PROVENANCE_BASE_REF). Always INFO, never FAIL — it flags "go verify this," it isn't itself proof of a defect. A brand-new entry absent at the base ref is a creation, not a change, and isn't flagged (avoids the false-positive class above). An unresolvable base ref emits one graceful INFO for the whole check, never a silent skip, matching this script's existing checks 7/8 convention. 8 new bats tests (73 total, 0 failures, independently re-run).
  2. Audit-time rubric — validation-scripts.md: when check 9 fires, the auditor must actually open the named Contributing file(s) and the slug's Research doc: and confirm the claim by reading, not just relay the INFO.
  3. Authoring-time — skill-author/references/retrofit.md: a new checklist bullet (distinct from the existing Contributing-files-staleness one, which covers a different failure mode — forgetting to update after moving content) warning against upgrading an honest hedge into an unsupported confident claim during compression, with the rule to re-read the upstream research doc before strengthening any provenance wording.

The concrete case (forge's claude-code-plugins-docs entry) was already corrected on main prior to this session, per the issue body — no content fix needed there, only the gate/rubric/guardrail work above.

## Implementation note Two approaches from the suggested fix were tried and rejected before implementing, worth recording so they aren't re-proposed blind: - **Literal filename cross-check** ("does the description name a `.md` file absent from Contributing files?") — tested empirically against the real 95-entry corpus: 3/95 flagged, all three false positives (a research-doc path mentioned as provenance-only, a research doc's own filename named while explicitly disclaiming it informed anything, a self-reference to `sources.md`/`CONTEXT.md`). It also would **not** have caught the actual bug this issue describes — the bad description said "Grounds... Step 2's dispatch table," never naming `SKILL.md` literally. Not implemented. - Conclusion: no bash script can verify semantic truth (the issue says this explicitly). The fix uses what git *can* reliably detect — a changed field — purely as a trigger for what *can* verify semantics: the auditor reading the files, not more text-matching. **What shipped, defense in depth across two layers:** 1. **Audit-time — `validate-provenance.sh` check 9** (new): diffs each sources.md entry's `Description:`/`Contributing files:` text against a base ref (`git merge-base HEAD origin/main` by default, overridable via `--base-ref`/`VALIDATE_PROVENANCE_BASE_REF`). Always INFO, never FAIL — it flags "go verify this," it isn't itself proof of a defect. A brand-new entry absent at the base ref is a creation, not a change, and isn't flagged (avoids the false-positive class above). An unresolvable base ref emits one graceful INFO for the whole check, never a silent skip, matching this script's existing checks 7/8 convention. 8 new bats tests (73 total, 0 failures, independently re-run). 2. **Audit-time rubric — `validation-scripts.md`**: when check 9 fires, the auditor must actually open the named Contributing file(s) and the slug's `Research doc:` and confirm the claim by reading, not just relay the INFO. 3. **Authoring-time — `skill-author/references/retrofit.md`**: a new checklist bullet (distinct from the existing Contributing-files-staleness one, which covers a different failure mode — forgetting to update after moving content) warning against upgrading an honest hedge into an unsupported confident claim during compression, with the rule to re-read the upstream research doc before strengthening any provenance wording. The concrete case (`forge`'s `claude-code-plugins-docs` entry) was already corrected on `main` prior to this session, per the issue body — no content fix needed there, only the gate/rubric/guardrail work above.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label Kind/Enhancement
Priority
Medium
3
Reviewed
Confirmed
1
Milestone
No items
No Milestone Skills & Agents
Projects
Clear projects
No project
Assignees
Clear assignees
Claude renovate-bot Defame1297
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Defame1297/holocron#118
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?