provenance: Research doc: slug and path conventions do not match what validate-provenance expects #121

Closed
opened 2026-08-31 08:05:25 +00:00 by Claude · 3 comments
Collaborator

Found during the #99 ADR-0020 retrofit and the follow-up audit pass on refactor/adr0020-skill-retrofit. Deliberately not fixed there, because half of it is a schema decision that has to be made before the other half can be mass-corrected.

Context: the checks only just woke up

validate-provenance.sh checks 7 and 8 resolve each sources.md entry's Research doc: path and look for the entry's slug as an ## H2 inside it. Until commit 27a7669 (fix(kyberforge): stop the provenance checker skipping check 8 on unparsed input, addresses #111) both checks were dead across the whole git plugin — an unparsable Contributing files block returned the same empty result as an explicit (none), so the checker read "nothing to check" and skipped rather than reporting that it could not tell.

With them live, the corpus now reports 36 INFOs and 0 FAILs. Measured just now by running the script over all 39 skill directories:

for d in plugins/*/.apm/skills/*/; do
  bash plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh "$d"
done

Nothing is red. This issue is about deciding what the convention actually is, not about a broken gate.

Two distinct shapes

(a) 30 slug-vs-heading mismatches

The slug in references/sources.md does not appear as an ## H2 in the file Research doc: names.

skill count
core/agentsmd-audit 1
git/git-branches 4
git/git-commits 3
git/git-history 3
git/git-remotes 5
git/git-submodules 1
git/git-workflow 4
git/git-worktrees 1
git/pc-author 2
git/pc-run 2
gitea/gitea-releases 4

The git-history 3 pre-date the retrofit; the other 27 surfaced with the checks.

Probable root cause, and it is mechanical. skill-author/references/create.md:168 specifies the field as:

- **Research doc:** <path> where <path> is the relative path from the repo root to the plugin-level research sources file this entry was drawn from (e.g. plugins/myplugin/docs/research/docs/<topic>/sources.md)

That is exactly the file whose H2s are the slugs. But 29 of these 30 point at a research topic file instead, whose H2s are section headings. Concretely, git-branches names plugins/git/docs/research/docs/git/gitflow.md, whose H2s are ## Core Philosophy, ## The Five Branch Types, … — while plugins/git/docs/research/docs/git/sources.md sitting beside it has ## nvie-gitflow-post, ## atlassian-gitflow-tutorial, ## gitflow-cheatsheet and every other slug the git plugin uses. The same holds for plugins/gitea/docs/research/docs/gitea/sources.md (5 slugs) and plugins/git/docs/research/docs/pre-commit/sources.md (4 slugs). Repointing at the sibling sources.md would clear these mechanically.

The 30th is a different case: core/agentsmd-audit's governance-secrets-hard-prohibition points at core/instructions/governance.md, annotated in-place as "org convention file, not a plugin research corpus entry". The path resolves, so check 7 runs and finds no matching H2 — a deliberate non-corpus pointer being measured against a corpus rule.

(b) 6 values that are not a single path

These resolve to nothing, so checks 7 and 8 do not run at all and the script says so:

skill slug value
core/provider-adapter-author adr-0002-0003-two-tier-claude-md docs/adr/0002-….md, docs/adr/0003-���.md, providers/claude-code/CLAUDE.md
git/git-commits org-commit-conventions core/instructions/commits.md, core/instructions/git.md
git/pc-author context7-pre-commit-com plugins/git/docs/research/docs/pre-commit/{overview,configuration,cli-reference,hook-authoring}.md
git/pc-author pre-commit-com same brace expansion
git/pc-run context7-pre-commit-com plugins/git/docs/research/docs/pre-commit/{overview,cli-reference,troubleshooting}.md
git/pc-run pre-commit-com same brace expansion

Two sub-shapes: comma-separated lists, and shell-style brace expansion. Both are humans writing "several documents" into a field whose only specified grammar is one path.

The decision to make first

Does Research doc: accept a list? (b) cannot be fixed without answering that, and (a) should not be mass-corrected until it is — if the answer is "yes, a list", then the natural fix for (a) may be "point at the topic files and the sources.md", not "swap one path for another".

Sub-questions the answer has to cover:

  1. List syntax: comma-separated on one line, or repeated - **Research doc:** bullets? Brace expansion is the one form to reject outright — it is shell syntax in a markdown field and nothing expands it.
  2. Is a non-corpus pointer (an ADR, core/instructions/*.md) a legal value at all, or does it need its own field / an explicit (none)-style declaration? Today it is legal-by-accident and produces an INFO that can never be cleared. Same shape as #111's problem: no honest way to declare the truthful thing.
  3. Once the grammar is settled, should the tier stay INFO or become a FAIL? It was kept at INFO on this branch deliberately rather than turning 36 entries red on a corpus that is otherwise clean.

Files

  • plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh — checks 6, 7, 8
  • plugins/kyberforge/.apm/skills/skill-author/references/create.md:168 — the only place the field's grammar is stated
  • plugins/kyberforge/.apm/skills/skill-audit/references/file-structure.md:51 — the references/sources.md exemption, which explains why an unresolvable path is an INFO rather than a broken-reference FAIL
  • the 11 references/sources.md files listed above
  • #111 — the same script, the same class of gap: no way to declare the honest thing
  • #118 — the other provenance gap found in this pass (an entry can assert a contribution that never happened, and exit 0)
  • #99 — the retrofit during which this surfaced
Found during the #99 ADR-0020 retrofit and the follow-up audit pass on `refactor/adr0020-skill-retrofit`. Deliberately **not** fixed there, because half of it is a schema decision that has to be made before the other half can be mass-corrected. ## Context: the checks only just woke up `validate-provenance.sh` checks 7 and 8 resolve each `sources.md` entry's `Research doc:` path and look for the entry's slug as an `## H2` inside it. Until commit `27a7669` (`fix(kyberforge): stop the provenance checker skipping check 8 on unparsed input`, addresses #111) both checks were **dead across the whole git plugin** — an unparsable `Contributing files` block returned the same empty result as an explicit `(none)`, so the checker read "nothing to check" and skipped rather than reporting that it could not tell. With them live, the corpus now reports **36 INFOs and 0 FAILs**. Measured just now by running the script over all 39 skill directories: ``` for d in plugins/*/.apm/skills/*/; do bash plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh "$d" done ``` Nothing is red. This issue is about deciding what the convention actually is, not about a broken gate. ## Two distinct shapes ### (a) 30 slug-vs-heading mismatches The slug in `references/sources.md` does not appear as an `## H2` in the file `Research doc:` names. | skill | count | |---|---| | `core/agentsmd-audit` | 1 | | `git/git-branches` | 4 | | `git/git-commits` | 3 | | `git/git-history` | 3 | | `git/git-remotes` | 5 | | `git/git-submodules` | 1 | | `git/git-workflow` | 4 | | `git/git-worktrees` | 1 | | `git/pc-author` | 2 | | `git/pc-run` | 2 | | `gitea/gitea-releases` | 4 | The `git-history` 3 pre-date the retrofit; the other 27 surfaced with the checks. **Probable root cause, and it is mechanical.** `skill-author/references/create.md:168` specifies the field as: > `- **Research doc:** <path>` where `<path>` is the relative path from the repo root to the **plugin-level research sources file** this entry was drawn from (e.g. `plugins/myplugin/docs/research/docs/<topic>/sources.md`) That is exactly the file whose H2s *are* the slugs. But 29 of these 30 point at a research **topic** file instead, whose H2s are section headings. Concretely, `git-branches` names `plugins/git/docs/research/docs/git/gitflow.md`, whose H2s are `## Core Philosophy`, `## The Five Branch Types`, … — while `plugins/git/docs/research/docs/git/sources.md` sitting beside it has `## nvie-gitflow-post`, `## atlassian-gitflow-tutorial`, `## gitflow-cheatsheet` and every other slug the git plugin uses. The same holds for `plugins/gitea/docs/research/docs/gitea/sources.md` (5 slugs) and `plugins/git/docs/research/docs/pre-commit/sources.md` (4 slugs). Repointing at the sibling `sources.md` would clear these mechanically. The 30th is a different case: `core/agentsmd-audit`'s `governance-secrets-hard-prohibition` points at `core/instructions/governance.md`, annotated in-place as *"org convention file, not a plugin research corpus entry"*. The path resolves, so check 7 runs and finds no matching H2 — a deliberate non-corpus pointer being measured against a corpus rule. ### (b) 6 values that are not a single path These resolve to nothing, so checks 7 and 8 do not run at all and the script says so: | skill | slug | value | |---|---|---| | `core/provider-adapter-author` | `adr-0002-0003-two-tier-claude-md` | `docs/adr/0002-….md, docs/adr/0003-���.md, providers/claude-code/CLAUDE.md` | | `git/git-commits` | `org-commit-conventions` | `core/instructions/commits.md, core/instructions/git.md` | | `git/pc-author` | `context7-pre-commit-com` | `plugins/git/docs/research/docs/pre-commit/{overview,configuration,cli-reference,hook-authoring}.md` | | `git/pc-author` | `pre-commit-com` | same brace expansion | | `git/pc-run` | `context7-pre-commit-com` | `plugins/git/docs/research/docs/pre-commit/{overview,cli-reference,troubleshooting}.md` | | `git/pc-run` | `pre-commit-com` | same brace expansion | Two sub-shapes: comma-separated lists, and shell-style brace expansion. Both are humans writing "several documents" into a field whose only specified grammar is one path. ## The decision to make first **Does `Research doc:` accept a list?** (b) cannot be fixed without answering that, and (a) should not be mass-corrected until it is — if the answer is "yes, a list", then the natural fix for (a) may be "point at the topic files *and* the sources.md", not "swap one path for another". Sub-questions the answer has to cover: 1. List syntax: comma-separated on one line, or repeated `- **Research doc:**` bullets? Brace expansion is the one form to reject outright — it is shell syntax in a markdown field and nothing expands it. 2. Is a non-corpus pointer (an ADR, `core/instructions/*.md`) a legal value at all, or does it need its own field / an explicit `(none)`-style declaration? Today it is legal-by-accident and produces an INFO that can never be cleared. Same shape as #111's problem: no honest way to declare the truthful thing. 3. Once the grammar is settled, should the tier stay INFO or become a FAIL? It was kept at INFO on this branch deliberately rather than turning 36 entries red on a corpus that is otherwise clean. ## Files - `plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh` — checks 6, 7, 8 - `plugins/kyberforge/.apm/skills/skill-author/references/create.md:168` — the only place the field's grammar is stated - `plugins/kyberforge/.apm/skills/skill-audit/references/file-structure.md:51` — the `references/sources.md` exemption, which explains why an unresolvable path is an INFO rather than a broken-reference FAIL - the 11 `references/sources.md` files listed above ## Related - #111 — the same script, the same class of gap: no way to declare the honest thing - #118 — the other provenance gap found in this pass (an entry can assert a contribution that never happened, and exit 0) - #99 — the retrofit during which this surfaced
Claude added this to the Skills & Agents milestone 2026-08-31 08:05:25 +00:00
Claude added the Kind/Enhancement
Reviewed
Confirmed
1
Priority
Medium
3
labels 2026-08-31 08:05:25 +00:00
Author
Collaborator

Two more items for this issue, found while adding the regression tests for 27a7669 (commit 5b80f30).

The same blind spot exists one function over

484357a / 27a7669 taught parse_contributing_files the bullet form (- **Contributing files:** on its own line, entries beneath). parse_status and parse_research_doc were not changed and still match only the inline spelling.

This is not hypothetical: plugins/gitea/.apm/skills/gitea-releases/references/sources.md already writes **Status:** with no leading hyphen. It is harmless today only by accident of scope — parse_status is applied to research docs, and no research doc currently uses the bullet form. The moment one does, the status test silently reads as absent.

That is the same failure shape this issue and #118 are about: a parser that returns "nothing found" and a caller that reads it as "nothing declared". Worth fixing in the same pass as shape (a), since both are edits to how sources.md and the research docs are read.

A Research doc: value can name two paths, and only the first is checked

gitea-releases' entry carries a semicolon-separated pair of annotated paths. strip_research_doc_annotation() keeps the first and drops the rest, so the second path is never resolved and checks 7-8 never run against it — silently, and with no INFO, because the first path resolves fine.

This is the same schema question as shape (b) in the issue body (comma-separated lists and {a,b,c}.md brace expansions), with a third separator. It strengthens the case for deciding the schema before mass-fixing shape (a): whatever Research doc: is allowed to contain — one path, a list, or a glob — needs to be settled once, and the parser and the four skills writing multi-path values then follow from it.

Both found while writing tests, not by any gate.

Two more items for this issue, found while adding the regression tests for `27a7669` (commit `5b80f30`). ## The same blind spot exists one function over `484357a` / `27a7669` taught `parse_contributing_files` the bullet form (`- **Contributing files:**` on its own line, entries beneath). `parse_status` and `parse_research_doc` were not changed and still match only the inline spelling. This is not hypothetical: `plugins/gitea/.apm/skills/gitea-releases/references/sources.md` already writes `**Status:**` with no leading hyphen. It is harmless *today* only by accident of scope — `parse_status` is applied to research docs, and no research doc currently uses the bullet form. The moment one does, the status test silently reads as absent. That is the same failure shape this issue and #118 are about: a parser that returns "nothing found" and a caller that reads it as "nothing declared". Worth fixing in the same pass as shape (a), since both are edits to how `sources.md` and the research docs are read. ## A `Research doc:` value can name two paths, and only the first is checked `gitea-releases`' entry carries a **semicolon-separated pair** of annotated paths. `strip_research_doc_annotation()` keeps the first and drops the rest, so the second path is never resolved and checks 7-8 never run against it — silently, and with no INFO, because the first path resolves fine. This is the same schema question as shape (b) in the issue body (comma-separated lists and `{a,b,c}.md` brace expansions), with a third separator. It strengthens the case for deciding the schema before mass-fixing shape (a): whatever `Research doc:` is allowed to contain — one path, a list, or a glob — needs to be settled once, and the parser and the four skills writing multi-path values then follow from it. Both found while writing tests, not by any gate.
Author
Collaborator

Automated note from the PR #135 review follow-up. This issue's own conclusion, state and labels are untouched — this comment only records paths that PR #135's branch renamed, so the issue stays actionable.

Paths this issue cites that have moved

  • skill-audit has been merged into factory-audit (ADR-0025, commit 620f20b). plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh is now plugins/kyberforge/.apm/skills/factory-audit/scripts/validate-provenance.sh.
  • skill-audit/references/file-structure.md is now factory-audit/references/skill-file-structure.md. The filename changed, not just the directory — a path rewrite that only swaps the directory will still dangle.
  • The citation skill-author/references/create.md:168 has drifted to skill-author/references/create.md:175.

Nothing else in this issue was re-validated.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01NwD8Egs5r4ndqeFLmhusX2

Automated note from the PR #135 review follow-up. This issue's own conclusion, state and labels are untouched — this comment only records paths that PR #135's branch renamed, so the issue stays actionable. **Paths this issue cites that have moved** - `skill-audit` has been merged into `factory-audit` (ADR-0025, commit `620f20b`). `plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh` is now `plugins/kyberforge/.apm/skills/factory-audit/scripts/validate-provenance.sh`. - `skill-audit/references/file-structure.md` is now `factory-audit/references/skill-file-structure.md`. The **filename** changed, not just the directory — a path rewrite that only swaps the directory will still dangle. - The citation `skill-author/references/create.md:168` has drifted to `skill-author/references/create.md:175`. Nothing else in this issue was re-validated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NwD8Egs5r4ndqeFLmhusX2
Author
Collaborator

Design decisions settled in a grill-with-docs session, implemented in PR #139 (branch feat/121-research-doc-grammar). The ADR is docs/adr/0028-research-doc-names-the-research-registry.md; the term is defined as Research registry in CONTEXT.md.

Decisions

  1. What Research doc: refers to. The plugin's Research registry: the research sources.md whose ## H2 headings are the source slugs. This is what skill-author/references/create.md already specified and what check 7 already assumes. The 29 entries pointing at topic docs get repointed. A topic-doc pointer survives only as parenthetical free-text annotation and is not checked.
  2. Lists and non-corpus pointers. Research doc: stays single-valued, so no list grammar is needed. Brace expansion, comma lists and semicolon pairs are gone. Shape (b) collapses: the pc-author and pc-run brace values are one registry, pre-commit/sources.md. An entry with no registry (ADRs, core/instructions/*.md) declares Research doc: none plus a new Basis: field, one - **Basis:** <path> bullet per path, existence-checked. git-workflow already writes none, so this codifies existing practice.
  3. Tier. A path that resolves but whose slug is missing (check 7), or a topic doc named in Research doc:, is a FAIL. A path that does not resolve stays INFO, so auditing a deployed copy elsewhere is not failed on something it cannot judge.
  4. Gate. Nothing in the repo ran validate-provenance.sh across the real corpus (only check-scope-walkup-sync.sh ran it, on fixtures), so a FAIL tier would be inert. A pre-push sweep, scripts/check-provenance-corpus.sh, lands in the same change.
  5. Parser parity (the earlier comment on this issue). The Research doc: and Basis: parsers accept the inline, bullet and header-plus-bullets spellings, with regression tests. parse_status is removed along with check 8.
  6. Check 8 is retired. Found during migration: check 8 requires every extracted registry slug to appear in the skill's sources.md. That is unsatisfiable when one registry serves many skills (git/sources.md backs 7), and it only ever looked satisfiable because topic-doc pointers kept it dormant. Check 7 already covers the direction that matters. An earlier draft of this comment listed check 8 as a FAIL; that was wrong.
  7. Historical Basis: paths. core/instructions/git.md and commits.md were deleted in 5deed07. A Basis: bullet annotated (removed in <sha>) skips the existence check. The sha is not verified, so the annotation can be written on any bullet to dodge the check; that weakness is accepted for a handful of entries.
  8. The lint entry is removed. house-vale-3-15-2-repro (in vale-config and vale-run) said none and claimed six behaviours were established "against purpose-built fixtures in this repo". No such fixture or test exists in the tree or in history (added in d1afdbe with no test files; the only vale test ever deleted, 4de5b6b, guards an unrelated E100). The entry and its source_keys citations are removed; the behavioural rules stay.

Result (PR #139)

  • Corpus sweep: 27 skills checked, passes with no allowlist.
  • factory-audit bats: 272/272. Full tests/run-tests.sh --strict: 0 failed, 0 skipped.
  • All pre-push hooks passed on push, including apm audit --ci, apm pack --check-clean, the new corpus gate and the ADR-0022 version-bump gate (16 skills bumped by one PATCH).
  • Vale 3.15.2 via vale-wrap.sh: 0 alerts across the 16 changed SKILL.md files, which is all the repo's Vale hook covers.
  • Migrated: 9 git skills, gitea-releases, agentsmd-audit, provider-adapter-author; the two lint entries removed.

Open

  • configuration-reference.md still says its rows were "reproduced against Vale 3.15.2" and "house-verified". That wording now has no provenance entry behind it. Whether to soften it, or commit real reproduction fixtures, is not decided here.
  • The ADR, references/ and sources.md files are outside any configured Vale scope, so they are unlinted rather than clean. Extending Vale to docs/adr/ would be a separate change.

Paths follow the earlier rename comment: factory-audit/scripts/validate-provenance.sh, create.md:175.

Design decisions settled in a grill-with-docs session, implemented in PR #139 (branch `feat/121-research-doc-grammar`). The ADR is `docs/adr/0028-research-doc-names-the-research-registry.md`; the term is defined as **Research registry** in `CONTEXT.md`. ## Decisions 1. **What `Research doc:` refers to.** The plugin's **Research registry**: the research `sources.md` whose `## H2` headings are the source slugs. This is what `skill-author/references/create.md` already specified and what check 7 already assumes. The 29 entries pointing at topic docs get repointed. A topic-doc pointer survives only as parenthetical free-text annotation and is not checked. 2. **Lists and non-corpus pointers.** `Research doc:` stays single-valued, so no list grammar is needed. Brace expansion, comma lists and semicolon pairs are gone. Shape (b) collapses: the `pc-author` and `pc-run` brace values are one registry, `pre-commit/sources.md`. An entry with no registry (ADRs, `core/instructions/*.md`) declares `Research doc: none` plus a new `Basis:` field, one `- **Basis:** <path>` bullet per path, existence-checked. `git-workflow` already writes `none`, so this codifies existing practice. 3. **Tier.** A path that resolves but whose slug is missing (check 7), or a topic doc named in `Research doc:`, is a **FAIL**. A path that does not resolve stays **INFO**, so auditing a deployed copy elsewhere is not failed on something it cannot judge. 4. **Gate.** Nothing in the repo ran `validate-provenance.sh` across the real corpus (only `check-scope-walkup-sync.sh` ran it, on fixtures), so a FAIL tier would be inert. A pre-push sweep, `scripts/check-provenance-corpus.sh`, lands in the same change. 5. **Parser parity** (the earlier comment on this issue). The `Research doc:` and `Basis:` parsers accept the inline, bullet and header-plus-bullets spellings, with regression tests. `parse_status` is removed along with check 8. 6. **Check 8 is retired.** Found during migration: check 8 requires every `extracted` registry slug to appear in the skill's `sources.md`. That is unsatisfiable when one registry serves many skills (`git/sources.md` backs 7), and it only ever looked satisfiable because topic-doc pointers kept it dormant. Check 7 already covers the direction that matters. An earlier draft of this comment listed check 8 as a FAIL; that was wrong. 7. **Historical `Basis:` paths.** `core/instructions/git.md` and `commits.md` were deleted in `5deed07`. A `Basis:` bullet annotated `(removed in <sha>)` skips the existence check. The sha is not verified, so the annotation can be written on any bullet to dodge the check; that weakness is accepted for a handful of entries. 8. **The `lint` entry is removed.** `house-vale-3-15-2-repro` (in `vale-config` and `vale-run`) said `none` and claimed six behaviours were established "against purpose-built fixtures in this repo". No such fixture or test exists in the tree or in history (added in `d1afdbe` with no test files; the only vale test ever deleted, `4de5b6b`, guards an unrelated `E100`). The entry and its `source_keys` citations are removed; the behavioural rules stay. ## Result (PR #139) - Corpus sweep: 27 skills checked, passes with no allowlist. - `factory-audit` bats: 272/272. Full `tests/run-tests.sh --strict`: 0 failed, 0 skipped. - All pre-push hooks passed on push, including `apm audit --ci`, `apm pack --check-clean`, the new corpus gate and the ADR-0022 version-bump gate (16 skills bumped by one PATCH). - Vale 3.15.2 via `vale-wrap.sh`: 0 alerts across the 16 changed `SKILL.md` files, which is all the repo's Vale hook covers. - Migrated: 9 git skills, `gitea-releases`, `agentsmd-audit`, `provider-adapter-author`; the two lint entries removed. ## Open - `configuration-reference.md` still says its rows were "reproduced against Vale 3.15.2" and "house-verified". That wording now has no provenance entry behind it. Whether to soften it, or commit real reproduction fixtures, is not decided here. - The ADR, `references/` and `sources.md` files are outside any configured Vale scope, so they are unlinted rather than clean. Extending Vale to `docs/adr/` would be a separate change. Paths follow the earlier rename comment: `factory-audit/scripts/validate-provenance.sh`, `create.md:175`.
Sign in to join this conversation.