Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

validate-provenance.sh: no way to declare a reference file as house-authored #111

New Issue
Closed
opened 2026-08-30 13:18:52 +00:00 by Claude · 2 comments
Claude commented 2026-08-30 13:18:52 +00:00
Collaborator
Copy Link

Found during the #99 ADR-0020 retrofit, wave 2 (git-branches).

What happens

validate-provenance.sh Check 3 emits an INFO for any references/ file with no source_keys frontmatter:

INFO  No source_keys frontmatter — references/orchestrator-contract.md
      Note: This references file has no source_keys — provenance cannot be verified.

There is no way to say "this file is house-authored and has no external source". The check tests if not ref_keys: against the parsed list, and parse_source_keys returns [] for both a missing key and an explicit source_keys: [] — so the two are indistinguishable and produce the identical INFO.

Verified by reading the script rather than inferred.

Why it matters

Some reference files legitimately have no research provenance. git-branches/references/orchestrator-contract.md is the concrete case: it holds the request/result JSON schemas, which are the skill's own interface design, not derived from any source. The plugin's sources.md never attributed the original in-body section either.

The only ways to silence the INFO today are both wrong:

  • Invent a slug — a false provenance claim, and it cascades into Checks 4–8 (research-doc resolution) and Check 5 (Contributing-files back-reference), so the lie has to be maintained in sources.md too.
  • Borrow an unrelated existing slug — same problem, quieter.

So the honest choice is to leave a permanent INFO on every house-authored reference file. That is tolerable at today's volume, but it trains readers to skim past provenance INFOs, which is exactly the signal that should stay sharp.

Suggested fix

Give the schema an explicit way to declare it — e.g. source_keys: [] treated as a deliberate declaration distinct from a missing key, or a dedicated provenance: house-authored field. Either way the check should pass silently on a file that has declared it has no external source, and keep the INFO only for files that never said.

Related

Sibling gate defects from the same retrofit: #107, #108, #109.

Found during the #99 ADR-0020 retrofit, wave 2 (`git-branches`). ## What happens `validate-provenance.sh` Check 3 emits an INFO for any `references/` file with no `source_keys` frontmatter: ``` INFO No source_keys frontmatter — references/orchestrator-contract.md Note: This references file has no source_keys — provenance cannot be verified. ``` There is no way to say "this file is house-authored and has no external source". The check tests `if not ref_keys:` against the **parsed list**, and `parse_source_keys` returns `[]` for both a missing key and an explicit `source_keys: []` — so the two are indistinguishable and produce the identical INFO. Verified by reading the script rather than inferred. ## Why it matters Some reference files legitimately have no research provenance. `git-branches/references/orchestrator-contract.md` is the concrete case: it holds the request/result JSON schemas, which are the skill's own interface design, not derived from any source. The plugin's `sources.md` never attributed the original in-body section either. The only ways to silence the INFO today are both wrong: - **Invent a slug** — a false provenance claim, and it cascades into Checks 4–8 (research-doc resolution) and Check 5 (Contributing-files back-reference), so the lie has to be maintained in `sources.md` too. - **Borrow an unrelated existing slug** — same problem, quieter. So the honest choice is to leave a permanent INFO on every house-authored reference file. That is tolerable at today's volume, but it trains readers to skim past provenance INFOs, which is exactly the signal that should stay sharp. ## Suggested fix Give the schema an explicit way to declare it — e.g. `source_keys: []` treated as a deliberate declaration distinct from a missing key, or a dedicated `provenance: house-authored` field. Either way the check should pass silently on a file that has *declared* it has no external source, and keep the INFO only for files that never said. ## Related Sibling gate defects from the same retrofit: #107, #108, #109.
Claude added the Kind/Enhancement
Reviewed
Confirmed
1
Priority
Low
4
 labels 2026-08-30 13:18:52 +00:00
Claude added this to the Skills & Agents milestone 2026-08-30 15:59:04 +00:00
Claude commented 2026-08-31 08:09:59 +00:00
Author
Collaborator
Copy Link

Fixed on refactor/adr0020-skill-retrofit (not yet pushed — the PR body will carry the close)

Implemented as the first of the two suggested shapes: source_keys: [] is now a declaration, not an absence.

validate-provenance.sh gained declares_empty_source_keys(fm), matching ^\s*source_keys:\s*\[\s*\]\s*$ against the raw frontmatter text rather than the parsed list — which is what made the two cases indistinguishable. Check 3 consults it before emitting:

if not ref_keys:
    if declares_empty_source_keys(ref_fm):
        continue
    emit_info(...)

A bare source_keys: with nothing after it is not accepted as the declaration and still emits the INFO — that shape reads as truncation or an abandoned edit, not a decision. The INFO's remedy text now names the escape hatch: "...or declare an explicit source_keys: [] if the file is house-authored and has no external source."

The named case is resolved. git-branches/references/orchestrator-contract.md carries source_keys: [] and the skill no longer emits that INFO — verified by running the script against plugins/git/.apm/skills/git-branches.

Worth recording: agent-audit/references/README.md already carried source_keys: [] on main and was emitting the INFO anyway. It needed no content change at all — only the script fix. That is the clearest evidence that the declaration was being written by authors and silently discarded.

Remaining, honestly: the change is not pinned by a test. skill-audit/tests/validate-provenance.bats is 17 cases and none of them covers the source_keys: [] declaration or the bare-source_keys: control. Worth adding before this behaviour is depended on.

Note the change lands only in skill-audit/scripts/validate-provenance.sh. The agent-audit copy is a different, shorter script with no references/ directory to check, so there is nothing to mirror there.

## Fixed on `refactor/adr0020-skill-retrofit` (not yet pushed — the PR body will carry the close) Implemented as the first of the two suggested shapes: `source_keys: []` is now a **declaration**, not an absence. `validate-provenance.sh` gained `declares_empty_source_keys(fm)`, matching `^\s*source_keys:\s*\[\s*\]\s*$` against the raw frontmatter text rather than the parsed list — which is what made the two cases indistinguishable. Check 3 consults it before emitting: ```python if not ref_keys: if declares_empty_source_keys(ref_fm): continue emit_info(...) ``` A bare `source_keys:` with nothing after it is **not** accepted as the declaration and still emits the INFO — that shape reads as truncation or an abandoned edit, not a decision. The INFO's remedy text now names the escape hatch: "...or declare an explicit `source_keys: []` if the file is house-authored and has no external source." **The named case is resolved.** `git-branches/references/orchestrator-contract.md` carries `source_keys: []` and the skill no longer emits that INFO — verified by running the script against `plugins/git/.apm/skills/git-branches`. **Worth recording:** `agent-audit/references/README.md` already carried `source_keys: []` on `main` and was emitting the INFO anyway. It needed no content change at all — only the script fix. That is the clearest evidence that the declaration was being written by authors and silently discarded. **Remaining, honestly:** the change is not pinned by a test. `skill-audit/tests/validate-provenance.bats` is 17 cases and none of them covers the `source_keys: []` declaration or the bare-`source_keys:` control. Worth adding before this behaviour is depended on. Note the change lands only in `skill-audit/scripts/validate-provenance.sh`. The `agent-audit` copy is a different, shorter script with no `references/` directory to check, so there is nothing to mirror there.
Claude commented 2026-09-01 13:59:44 +00:00
Author
Collaborator
Copy Link

Fixed by #129, squash-merged to main as 598a7c3.

validate-provenance.sh now treats an explicit source_keys: [] as a declaration that the file is house-authored with no external source — the honest thing this issue said there was no way to say. It passes silently, and the INFO now fires only on a file that declares nothing at all, so silence means "declared" rather than "could not tell".

The surrounding fail-open class went with it in the same PR: an unparsable Contributing files block no longer returns the same empty result as an explicit (none), and every skip announces itself. That mattered more than it looked — checks 7/8 had been dead across nine skills for exactly that reason, and waking them is what surfaced #121.

Stayed open after the merge because Gitea's closing-keyword parser consumed only the first reference in Closes #99, #107, …. Closing manually.

Fixed by #129, squash-merged to `main` as `598a7c3`. `validate-provenance.sh` now treats an explicit `source_keys: []` as a **declaration** that the file is house-authored with no external source — the honest thing this issue said there was no way to say. It passes silently, and the INFO now fires only on a file that declares nothing at all, so silence means "declared" rather than "could not tell". The surrounding fail-open class went with it in the same PR: an unparsable `Contributing files` block no longer returns the same empty result as an explicit `(none)`, and every skip announces itself. That mattered more than it looked — checks 7/8 had been dead across nine skills for exactly that reason, and waking them is what surfaced #121. Stayed open after the merge because Gitea's closing-keyword parser consumed only the first reference in `Closes #99, #107, …`. Closing manually.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
No Label Kind/Enhancement
Priority
Low
4
Reviewed
Confirmed
1
Milestone
No items
No Milestone Skills & Agents
Projects
Clear projects
No project
Assignees
Clear assignees
Claude renovate-bot Defame1297
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Defame1297/holocron#111
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?