.pre-commit-config.yaml

@@ -220,6 +220,21 @@ repos:
         pass_filenames: false
         always_run: true
 
+      - id: check-provenance-corpus
+        name: Check provenance across the skill corpus
+        description: Run factory-audit's validate-provenance.sh over every plugins/*/.apm/skills/*/ that has references/sources.md and fail on any FAIL (ADR-0028, #121)
+        entry: bash scripts/check-provenance-corpus.sh
+        language: system
+        stages: [pre-push]
+        pass_filenames: false
+        always_run: true
+        # Nothing else runs validate-provenance.sh over the real corpus --
+        # check-scope-walkup-sync exercises it against synthetic fixtures only --
+        # so ADR-0028's FAIL tier for a Research doc mismatch would be inert
+        # without this caller. The skill set is globbed, not counted, and
+        # discovering zero skills is an error (exit 2), not a pass. Needs no
+        # network; needs python3, which the validator's own preflight names.
+
       - id: check-skill-version-bump
         name: Check changed skills bump metadata.version
         description: On every push, fail if a skill directory changed (tests/ excluded) since the merge-base with main without its SKILL.md metadata.version rising above both that merge-base's and main's tip's (ADR-0022)

docs/spec/gates.md

@@ -42,7 +42,7 @@ is checked out. Push one ref at a time when the gate matters.
 
 ## The pre-push gate
 
-Eight hooks, grouped below by what they guard rather than by the order `.pre-commit-config.yaml` declares them in.
+Nine hooks, grouped below by what they guard rather than by the order `.pre-commit-config.yaml` declares them in.
 
 **Core checks**
 
@@ -66,6 +66,7 @@ version-blind, so a stale key deploys fine (see [apm gates](#apm-gates)).
 | Hook | Guards |
 |---|---|
 | `check-apm-agents-valid` | runs `factory-audit`'s `validate.sh` over every real `plugins/*/.apm/agents/*.agent.md` (see [Agent files](#agent-files-take-the-description-gates-not-the-body-gate)) |
+| `check-provenance-corpus` | runs `factory-audit`'s `validate-provenance.sh` over every real `plugins/*/.apm/skills/*/` that has a `references/sources.md`, failing on any FAIL (see [The provenance corpus sweep](#the-provenance-corpus-sweep-adr-0028)) |
 
 **apm's own gates**
 
@@ -576,6 +577,38 @@ follows symlinks with `find -L` because vale does.
 on `files:` patterns that match single markdown files, and only the `-d "$arg"` branch mirrors a
 directory. The exposed caller is the hand-invoked `vale-wrap.sh <dir>`.
 
+## The provenance corpus sweep (ADR-0028)
+
+`check-provenance-corpus` runs `validate-provenance.sh` over every real
+`plugins/*/.apm/skills/*/` directory that has a `references/sources.md`, and fails on any FAIL. The set
+is discovered by glob, not counted, so a new skill is covered the moment it grows a `sources.md`, and
+**discovering zero skills is an error, not a pass**.
+
+The hook exists because nothing else ran the validator over the real corpus.
+`check-scope-walkup-sync` invokes it only against synthetic `mktemp` fixtures, and `factory-audit`'s
+bats suite does the same. So a `Research doc:` naming the wrong file, or a slug absent from its
+Research registry, could only be found by hand-running the validator in a loop. That is how 36
+mismatches (#121) reported INFO while every gate stayed green. ADR-0028 promotes "the check ran and
+found a mismatch" from INFO to FAIL; without a caller across the corpus that FAIL tier would be inert.
+
+It reuses the validators' exit contract (see
+[the three exit tiers](#the-three-exit-tiers-of-factory-audits-validators)) and keeps the tiers apart:
+
+| Exit | Means |
+|---|---|
+| **0** | every skill validated. INFO-only findings are printed, never swallowed |
+| **1** | at least one skill FAILed. The summary line names the failing skills |
+| **2** | the gate could not run: the validator is missing, a skill's validator run exited 2 ("not auditable"), or no skill with a `references/sources.md` was found |
+
+A validator exit 2 is reported as a gate error, not as a FAIL about that skill: it says the audit never
+happened, and the skill has not been shown to be wrong.
+
+An unresolvable `Research doc:` path stays INFO by design, because a deployed copy of a skill outside
+this repo will not carry the research docs (see `skill-file-structure.md`'s `sources.md` exemption).
+This repo's own corpus is audited from the authoring source, where every path resolves, so an INFO
+printed here is worth reading. Needs no network; needs `python3`, which the validator's own preflight
+names.
+
 ## Current retrofit status
 
 The ADR-0020 gates ship hot, with no baseline file — a shrinking baseline was considered and

scripts/check-provenance-corpus.sh

@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Corpus-wide provenance sweep: runs factory-audit's validate-provenance.sh over
+# every plugins/*/.apm/skills/*/ directory that has a references/sources.md, and
+# fails on any FAIL.
+#
+# WHY THIS GATE EXISTS (ADR-0028, #121). Nothing else runs the validator over the
+# real corpus. check-scope-walkup-sync.sh invokes it, but only against synthetic
+# mktemp fixtures, and the factory-audit bats suite does the same. So a
+# `Research doc:` that named the wrong file, or a slug absent from its Research
+# registry, could only be found by hand-running the validator in a loop -- which
+# is how 36 mismatches sat unnoticed while every gate stayed green. ADR-0028
+# promotes "the check ran and found a mismatch" from INFO to FAIL; without a
+# caller across the corpus that FAIL tier would be inert.
+#
+# Exit codes, kept distinct on purpose:
+#   0  every skill validated (INFO-only findings are printed, never swallowed)
+#   1  at least one skill FAILed -- a real finding about the corpus
+#   2  the gate itself could not run: validator missing, a validator exit 2
+#      ("not auditable"), or NO skill with a references/sources.md found. A
+#      gate that discovers nothing must not read as a pass, and a skill that
+#      could not be audited must not read as a skill that failed the audit.
+#
+# The skill set is discovered by glob, not hardcoded, so a new skill is covered
+# the moment it grows a references/sources.md. Run from repo root or pass
+# REPO_ROOT as arg.
+
+REPO_ROOT="${1:-$(git rev-parse --show-toplevel 2>/dev/null || pwd)}"
+if [[ ! -d "$REPO_ROOT" ]]; then
+  echo "Provenance corpus check failed: REPO_ROOT '$REPO_ROOT' is not a directory." >&2
+  exit 2
+fi
+REPO_ROOT="$(cd "$REPO_ROOT" && pwd)"
+
+VALIDATOR="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/validate-provenance.sh"
+if [[ ! -f "$VALIDATOR" ]]; then
+  echo "Provenance corpus check failed: $VALIDATOR does not exist, so no skill was audited. If factory-audit's scripts moved, update this path." >&2
+  exit 2
+fi
+
+shopt -s nullglob
+sources_files=("$REPO_ROOT"/plugins/*/.apm/skills/*/references/sources.md)
+shopt -u nullglob
+
+if [[ ${#sources_files[@]} -eq 0 ]]; then
+  echo "Provenance corpus check failed: found no plugins/*/.apm/skills/*/references/sources.md under $REPO_ROOT. Discovering zero skills is an error, not a pass -- the glob has gone stale or the corpus moved." >&2
+  exit 2
+fi
+
+failing=()
+errored=()
+for sources in "${sources_files[@]}"; do
+  refs_dir="${sources%/*}"
+  skill_dir="${refs_dir%/*}"
+  rel="${skill_dir#"$REPO_ROOT"/plugins/}"
+  label="${rel%%/*}/${skill_dir##*/}"
+
+  rc=0
+  out="$(bash "$VALIDATOR" "$skill_dir" 2>&1)" || rc=$?
+
+  case "$rc" in
+    0)
+      # Exit 0 with output means INFO-only: a check that could not run,
+      # announced rather than skipped. Print it so it is not swallowed.
+      if [[ -n "$out" ]]; then
+        echo "== $label"
+        echo "$out"
+      fi
+      ;;
+    1)
+      echo "== $label"
+      echo "$out"
+      failing+=("$label")
+      ;;
+    *)
+      echo "== $label (validator exit $rc)"
+      echo "$out"
+      errored+=("$label")
+      ;;
+  esac
+done
+
+echo ""
+echo "Provenance corpus: ${#sources_files[@]} skill(s) checked."
+
+if [[ ${#errored[@]} -gt 0 ]]; then
+  echo "Provenance corpus check errored (could not audit): ${errored[*]}" >&2
+  if [[ ${#failing[@]} -gt 0 ]]; then
+    echo "Failing skills: ${failing[*]}" >&2
+  fi
+  exit 2
+fi
+
+if [[ ${#failing[@]} -gt 0 ]]; then
+  echo "Failing skills: ${failing[*]}" >&2
+  echo "Fix each FAIL above (see ADR-0028 for the Research doc / Basis grammar); INFO lines do not fail the gate." >&2
+  exit 1
+fi
+
+echo "Provenance corpus check passed."

tests/test-adr0020-contract.sh

@@ -744,6 +744,8 @@ EXPECTED = {
         'apm pack --check-versions --check-clean --dry-run', ['pre-push']),
     'check-scope-walkup-sync': (
         'bash scripts/check-scope-walkup-sync.sh', ['pre-push']),
+    'check-provenance-corpus': (
+        'bash scripts/check-provenance-corpus.sh', ['pre-push']),
     'check-skill-version-bump': (
         'bash scripts/check-skill-version-bump.sh', ['pre-push']),
     'validate-marketplace': (

tests/test-check-provenance-corpus.sh

@@ -0,0 +1,186 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SCRIPT="$REPO_ROOT/scripts/check-provenance-corpus.sh"
+VALIDATOR_DIR="plugins/kyberforge/.apm/skills/factory-audit/scripts"
+PASS=0
+FAIL=0
+
+pass() { echo "  PASS: $1"; PASS=$((PASS + 1)); }
+fail() { echo "  FAIL: $1"; FAIL=$((FAIL + 1)); }
+
+FIXTURES=()
+cleanup() { [[ ${#FIXTURES[@]} -eq 0 ]] || rm -rf "${FIXTURES[@]}"; }
+trap cleanup EXIT
+
+# Per-run scratch for captured output, for the reason check-scope-walkup-sync's
+# test gives: tests/run-tests.sh fans test scripts out concurrently.
+RUN_TMP="$(mktemp -d)"
+FIXTURES+=("$RUN_TMP")
+
+# A minimal REPO_ROOT: a .git entry (the validator's find_repo_root stops at
+# it), a copy of the real validator at its real relative path, and one plugin
+# holding a Research registry. Copying the real validator means the fixtures
+# exercise the actual FAIL/INFO/exit contract rather than a stub of it.
+make_repo() {
+  local dir
+  dir="$(mktemp -d)"
+  FIXTURES+=("$dir")
+  mkdir -p "$dir/.git" "$dir/$VALIDATOR_DIR" "$dir/plugins/p/docs/research/docs/t"
+  cp -R "$REPO_ROOT/$VALIDATOR_DIR/." "$dir/$VALIDATOR_DIR/"
+  cat > "$dir/plugins/p/docs/research/docs/t/sources.md" <<'EOF'
+# Sources
+
+## known-slug
+
+**Status:** `extracted`
+EOF
+  echo "$dir"
+}
+
+# make_skill <repo> <name> <slug> <research-doc-value>
+make_skill() {
+  local repo="$1" name="$2" slug="$3" research="$4"
+  local skill="$repo/plugins/p/.apm/skills/$name"
+  mkdir -p "$skill/references"
+  cat > "$skill/SKILL.md" <<EOF
+---
+name: $name
+description: A valid skill description.
+metadata:
+  source_keys:
+    - $slug
+---
+
+## Step 1
+
+Do the thing.
+EOF
+  cat > "$skill/references/sources.md" <<EOF
+# Sources
+
+## $slug
+
+- **URL:** https://example.com/$slug
+- **Description:** A test source.
+- **Contributing files:** SKILL.md
+- **Research doc:** $research
+- **Status:** \`extracted\`
+EOF
+}
+
+REGISTRY="plugins/p/docs/research/docs/t/sources.md"
+
+# --- 1. A skill whose slug resolves in the registry passes, quietly ---
+echo ""
+echo "--- passing skill ---"
+R="$(make_repo)"
+make_skill "$R" good known-slug "$REGISTRY"
+if bash "$SCRIPT" "$R" > "$RUN_TMP/good.out" 2>&1; then
+  pass "exits 0 when every skill validates"
+else
+  fail "exited non-zero on a clean corpus: $(cat "$RUN_TMP/good.out")"
+fi
+
+# --- 2. A slug missing from the registry is a FAIL and is named ---
+echo ""
+echo "--- failing skill ---"
+R="$(make_repo)"
+make_skill "$R" good known-slug "$REGISTRY"
+make_skill "$R" bad missing-slug "$REGISTRY"
+set +e
+bash "$SCRIPT" "$R" > "$RUN_TMP/bad.out" 2>&1
+rc=$?
+set -e
+if [[ $rc -eq 1 ]]; then
+  pass "exits 1 when one skill has a slug missing from its registry"
+else
+  fail "expected exit 1, got $rc: $(cat "$RUN_TMP/bad.out")"
+fi
+if grep -q "bad" "$RUN_TMP/bad.out" && ! grep -qE "Failing skills:.*good" "$RUN_TMP/bad.out"; then
+  pass "summary line names the failing skill and not the passing one"
+else
+  fail "summary did not name only the failing skill: $(cat "$RUN_TMP/bad.out")"
+fi
+
+# --- 3. INFO-only passes but the INFO is printed, not swallowed ---
+echo ""
+echo "--- INFO-only skill ---"
+R="$(make_repo)"
+make_skill "$R" info-only known-slug "plugins/p/docs/research/docs/gone/sources.md"
+if bash "$SCRIPT" "$R" > "$RUN_TMP/info.out" 2>&1; then
+  pass "exits 0 when the only findings are INFO"
+else
+  fail "INFO-only corpus failed the gate: $(cat "$RUN_TMP/info.out")"
+fi
+if grep -q "INFO" "$RUN_TMP/info.out"; then
+  pass "INFO findings are printed"
+else
+  fail "INFO finding was swallowed: $(cat "$RUN_TMP/info.out")"
+fi
+
+# --- 4. Zero skills discovered is an error, not a pass ---
+echo ""
+echo "--- zero skills ---"
+R="$(make_repo)"
+set +e
+bash "$SCRIPT" "$R" > "$RUN_TMP/zero.out" 2>&1
+rc=$?
+set -e
+if [[ $rc -eq 2 ]]; then
+  pass "exits 2 when no skill with references/sources.md is found"
+else
+  fail "expected exit 2 for an empty corpus, got $rc: $(cat "$RUN_TMP/zero.out")"
+fi
+
+# --- 5. A missing validator is a gate error (exit 2), never a pass ---
+echo ""
+echo "--- missing validator ---"
+R="$(make_repo)"
+make_skill "$R" good known-slug "$REGISTRY"
+rm -rf "${R:?}/$VALIDATOR_DIR"
+set +e
+bash "$SCRIPT" "$R" > "$RUN_TMP/novalidator.out" 2>&1
+rc=$?
+set -e
+if [[ $rc -eq 2 ]]; then
+  pass "exits 2 when the validator is missing"
+else
+  fail "expected exit 2 for a missing validator, got $rc: $(cat "$RUN_TMP/novalidator.out")"
+fi
+
+# --- 6. A validator exit 2 (unauditable input) is a gate error, not a FAIL ---
+echo ""
+echo "--- validator exit 2 ---"
+R="$(make_repo)"
+make_skill "$R" good known-slug "$REGISTRY"
+# Replace the entry point with a stub that reports "not auditable".
+printf '#!/usr/bin/env bash\necho "stub: not auditable" >&2\nexit 2\n' \
+  > "$R/$VALIDATOR_DIR/validate-provenance.sh"
+set +e
+bash "$SCRIPT" "$R" > "$RUN_TMP/exit2.out" 2>&1
+rc=$?
+set -e
+if [[ $rc -eq 2 ]]; then
+  pass "a validator exit 2 surfaces as gate exit 2, not as a skill FAIL"
+else
+  fail "expected exit 2 to propagate, got $rc: $(cat "$RUN_TMP/exit2.out")"
+fi
+
+# --- 7. The real corpus: reported, and the gate agrees with the validator ---
+echo ""
+echo "--- this repo's real corpus ---"
+set +e
+bash "$SCRIPT" "$REPO_ROOT" > "$RUN_TMP/real.out" 2>&1
+rc=$?
+set -e
+if [[ $rc -eq 0 || $rc -eq 1 ]]; then
+  pass "gate runs to a verdict (0 or 1) against the real corpus (exit $rc)"
+else
+  fail "gate errored (exit $rc) against the real corpus: $(cat "$RUN_TMP/real.out")"
+fi
+
+echo ""
+echo "Results: $PASS passed, $FAIL failed"
+[[ $FAIL -eq 0 ]]