feat: skill-audit — validate sources provenance chain #8

New Issue

Claude · 2026-06-26T20:34:19Z

Claude commented

2026-06-26 20:34:19 +00:00

Summary

Add a validate-provenance.sh script to skill-audit that validates the sources provenance chain is complete and internally consistent. Also add a new Provenance report dimension, update the sources.md template in skill-author, update skill-author Step 5 to write Research doc: fields, and backfill existing sources.md files.

Background

The provenance chain runs in three stages:

/research writes topic files + sources.md to a plugin-level output directory (e.g. plugins/<plugin>/docs/research/docs/<topic>/)
/skill-author reads those docs and builds the skill, recording provenance in references/sources.md (which skill files each source contributed to, and which research doc it came from) and source_keys frontmatter on each skill file (SKILL.md and references/*.md)

The chain is: research → plugin-level docs → skill. The plugin-level docs are not part of the skill — they live at the plugin level and are read as input during authoring.

Checks

New script: `scripts/validate-provenance.sh`

Always runs unconditionally (called from Step 1 of the audit alongside validate.sh). Handles the "nothing to validate" case internally.

Skill-level checks (FAIL):

Every source_keys entry in SKILL.md metadata has a matching slug in references/sources.md
Every source_keys entry in each references/*.md file (excluding sources.md itself) has a matching slug in references/sources.md
Every file listed under Contributing files in references/sources.md actually exists in the skill directory — entries starting with (none are skipped; multiple files are comma-separated
No unfilled FILL IN: placeholders remain in references/sources.md
Every non-(none) Contributing files entry in references/sources.md: the listed file's source_keys includes the parent slug (bidirectional check) — skip if file doesn't exist (already caught above) or if file is sources.md itself
Every entry in references/sources.md has a Research doc: field pointing to the upstream plugin-level research sources file

Upstream checks (assume marketplace context for now):

Forward: slug in skill's references/sources.md not found in the upstream research sources.md declared by Research doc: → INFO
Reverse: extracted slug in upstream research sources.md with Contributing files pointing at this skill, missing from skill's references/sources.md → FAIL

INFO (not FAIL):

references/*.md file (excluding sources.md) has no source_keys frontmatter when sources.md is present

Edge cases:

source_keys declared anywhere but references/sources.md absent → FAIL
Skip bidirectional Contributing files check for files that don't exist (caught by existence check) and for references/sources.md itself

New report dimension: `Provenance`

Always appears in the Checked: coverage line
Script embeds full FAIL/INFO format including Why and Fix per finding
INFO findings do not affect pass/fail verdict

Updated result block format

PASS
PASS (N suggestions)
PASS · P info
PASS (N suggestions) · P info
FAIL (N fails · M suggestions)
FAIL (N fails · M suggestions) · P info

INFO is counted separately and never changes the PASS/FAIL determination.

Changes required

scripts/validate-provenance.sh — new script with all checks above
tests/validate-provenance.bats — fixture-based tests covering: clean pass, missing slug, orphaned Contributing file, missing Research doc path, upstream mismatch both directions, (none) skip case, source_keys without sources.md
skill-audit/SKILL.md — add second script call to Step 1; add Provenance to coverage line; update result block format in Step 4
skill-author/assets/templates/references/sources.md — add Research doc: field; formalize comma-separated Contributing files; document (none) convention
skill-author/SKILL.md Step 5 — record Research doc: path when writing references/sources.md
Backfill skill-audit/references/sources.md — add Research doc: field to all entries
Backfill skill-author/references/sources.md — add Research doc: field to all entries

Non-goals

Requiring references/sources.md to be present — it is optional; only created when research input was provided
Auto-detecting marketplace vs. normal .agents context — deferred; for now assume marketplace

Prerequisite

Issue #4 must be merged first.

## Summary Add a `validate-provenance.sh` script to `skill-audit` that validates the sources provenance chain is complete and internally consistent. Also add a new `Provenance` report dimension, update the `sources.md` template in `skill-author`, update `skill-author` Step 5 to write `Research doc:` fields, and backfill existing sources.md files. ## Background The provenance chain runs in three stages: 1. `/research` writes topic files + `sources.md` to a plugin-level output directory (e.g. `plugins/<plugin>/docs/research/docs/<topic>/`) 2. `/skill-author` reads those docs and builds the skill, recording provenance in `references/sources.md` (which skill files each source contributed to, and which research doc it came from) and `source_keys` frontmatter on each skill file (`SKILL.md` and `references/*.md`) The chain is: research → plugin-level docs → skill. The plugin-level docs are not part of the skill — they live at the plugin level and are read as input during authoring. ## Checks ### New script: `scripts/validate-provenance.sh` Always runs unconditionally (called from Step 1 of the audit alongside `validate.sh`). Handles the "nothing to validate" case internally. **Skill-level checks (FAIL):** - [x] Every `source_keys` entry in `SKILL.md` metadata has a matching slug in `references/sources.md` - [x] Every `source_keys` entry in each `references/*.md` file (excluding `sources.md` itself) has a matching slug in `references/sources.md` - [x] Every file listed under `Contributing files` in `references/sources.md` actually exists in the skill directory — entries starting with `(none` are skipped; multiple files are comma-separated - [x] No unfilled `FILL IN:` placeholders remain in `references/sources.md` - [x] Every non-`(none)` `Contributing files` entry in `references/sources.md`: the listed file's `source_keys` includes the parent slug (bidirectional check) — skip if file doesn't exist (already caught above) or if file is `sources.md` itself - [x] Every entry in `references/sources.md` has a `Research doc:` field pointing to the upstream plugin-level research sources file **Upstream checks (assume marketplace context for now):** - Forward: slug in skill's `references/sources.md` not found in the upstream research sources.md declared by `Research doc:` → INFO - Reverse: extracted slug in upstream research sources.md with Contributing files pointing at this skill, missing from skill's `references/sources.md` → FAIL **INFO (not FAIL):** - `references/*.md` file (excluding `sources.md`) has no `source_keys` frontmatter when `sources.md` is present **Edge cases:** - `source_keys` declared anywhere but `references/sources.md` absent → FAIL - Skip bidirectional Contributing files check for files that don't exist (caught by existence check) and for `references/sources.md` itself ### New report dimension: `Provenance` - Always appears in the `Checked:` coverage line - Script embeds full FAIL/INFO format including Why and Fix per finding - INFO findings do not affect pass/fail verdict ### Updated result block format ``` PASS PASS (N suggestions) PASS · P info PASS (N suggestions) · P info FAIL (N fails · M suggestions) FAIL (N fails · M suggestions) · P info ``` INFO is counted separately and never changes the PASS/FAIL determination. ## Changes required - [x] `scripts/validate-provenance.sh` — new script with all checks above - [x] `tests/validate-provenance.bats` — fixture-based tests covering: clean pass, missing slug, orphaned Contributing file, missing Research doc path, upstream mismatch both directions, `(none)` skip case, source_keys without sources.md - [x] `skill-audit/SKILL.md` — add second script call to Step 1; add Provenance to coverage line; update result block format in Step 4 - [x] `skill-author/assets/templates/references/sources.md` — add `Research doc:` field; formalize comma-separated Contributing files; document `(none)` convention - [x] `skill-author/SKILL.md` Step 5 — record `Research doc:` path when writing `references/sources.md` - [x] Backfill `skill-audit/references/sources.md` — add `Research doc:` field to all entries - [x] Backfill `skill-author/references/sources.md` — add `Research doc:` field to all entries ## Non-goals - Requiring `references/sources.md` to be present — it is optional; only created when research input was provided - Auto-detecting marketplace vs. normal `.agents` context — deferred; for now assume marketplace ## Prerequisite Issue #4 must be merged first.

Claude added this to the Kyberforge basics milestone 2026-06-26 20:34:19 +00:00

Claude added the Kind/Enhancement

labels 2026-06-26 20:34:19 +00:00

Claude referenced this issue

2026-06-26 20:37:52 +00:00

bug: skill-author loses source provenance — research output not recorded in skill references #4