diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index caa074e..e4da5c4 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -220,6 +220,21 @@ repos: pass_filenames: false always_run: true + - id: check-provenance-corpus + name: Check provenance across the skill corpus + description: Run factory-audit's validate-provenance.sh over every plugins/*/.apm/skills/*/ that has references/sources.md and fail on any FAIL (ADR-0028, #121) + entry: bash scripts/check-provenance-corpus.sh + language: system + stages: [pre-push] + pass_filenames: false + always_run: true + # Nothing else runs validate-provenance.sh over the real corpus -- + # check-scope-walkup-sync exercises it against synthetic fixtures only -- + # so ADR-0028's FAIL tier for a Research doc mismatch would be inert + # without this caller. The skill set is globbed, not counted, and + # discovering zero skills is an error (exit 2), not a pass. Needs no + # network; needs python3, which the validator's own preflight names. + - id: check-skill-version-bump name: Check changed skills bump metadata.version description: On every push, fail if a skill directory changed (tests/ excluded) since the merge-base with main without its SKILL.md metadata.version rising above both that merge-base's and main's tip's (ADR-0022) diff --git a/docs/spec/gates.md b/docs/spec/gates.md index a25124c..ad2a683 100644 --- a/docs/spec/gates.md +++ b/docs/spec/gates.md @@ -42,7 +42,7 @@ is checked out. Push one ref at a time when the gate matters. ## The pre-push gate -Eight hooks, grouped below by what they guard rather than by the order `.pre-commit-config.yaml` declares them in. +Nine hooks, grouped below by what they guard rather than by the order `.pre-commit-config.yaml` declares them in. **Core checks** @@ -66,6 +66,7 @@ version-blind, so a stale key deploys fine (see [apm gates](#apm-gates)). | Hook | Guards | |---|---| | `check-apm-agents-valid` | runs `factory-audit`'s `validate.sh` over every real `plugins/*/.apm/agents/*.agent.md` (see [Agent files](#agent-files-take-the-description-gates-not-the-body-gate)) | +| `check-provenance-corpus` | runs `factory-audit`'s `validate-provenance.sh` over every real `plugins/*/.apm/skills/*/` that has a `references/sources.md`, failing on any FAIL (see [The provenance corpus sweep](#the-provenance-corpus-sweep-adr-0028)) | **apm's own gates** @@ -576,6 +577,38 @@ follows symlinks with `find -L` because vale does. on `files:` patterns that match single markdown files, and only the `-d "$arg"` branch mirrors a directory. The exposed caller is the hand-invoked `vale-wrap.sh `. +## The provenance corpus sweep (ADR-0028) + +`check-provenance-corpus` runs `validate-provenance.sh` over every real +`plugins/*/.apm/skills/*/` directory that has a `references/sources.md`, and fails on any FAIL. The set +is discovered by glob, not counted, so a new skill is covered the moment it grows a `sources.md`, and +**discovering zero skills is an error, not a pass**. + +The hook exists because nothing else ran the validator over the real corpus. +`check-scope-walkup-sync` invokes it only against synthetic `mktemp` fixtures, and `factory-audit`'s +bats suite does the same. So a `Research doc:` naming the wrong file, or a slug absent from its +Research registry, could only be found by hand-running the validator in a loop. That is how 36 +mismatches (#121) reported INFO while every gate stayed green. ADR-0028 promotes "the check ran and +found a mismatch" from INFO to FAIL; without a caller across the corpus that FAIL tier would be inert. + +It reuses the validators' exit contract (see +[the three exit tiers](#the-three-exit-tiers-of-factory-audits-validators)) and keeps the tiers apart: + +| Exit | Means | +|---|---| +| **0** | every skill validated. INFO-only findings are printed, never swallowed | +| **1** | at least one skill FAILed. The summary line names the failing skills | +| **2** | the gate could not run: the validator is missing, a skill's validator run exited 2 ("not auditable"), or no skill with a `references/sources.md` was found | + +A validator exit 2 is reported as a gate error, not as a FAIL about that skill: it says the audit never +happened, and the skill has not been shown to be wrong. + +An unresolvable `Research doc:` path stays INFO by design, because a deployed copy of a skill outside +this repo will not carry the research docs (see `skill-file-structure.md`'s `sources.md` exemption). +This repo's own corpus is audited from the authoring source, where every path resolves, so an INFO +printed here is worth reading. Needs no network; needs `python3`, which the validator's own preflight +names. + ## Current retrofit status The ADR-0020 gates ship hot, with no baseline file — a shrinking baseline was considered and diff --git a/scripts/check-provenance-corpus.sh b/scripts/check-provenance-corpus.sh new file mode 100755 index 0000000..51c9c46 --- /dev/null +++ b/scripts/check-provenance-corpus.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Corpus-wide provenance sweep: runs factory-audit's validate-provenance.sh over +# every plugins/*/.apm/skills/*/ directory that has a references/sources.md, and +# fails on any FAIL. +# +# WHY THIS GATE EXISTS (ADR-0028, #121). Nothing else runs the validator over the +# real corpus. check-scope-walkup-sync.sh invokes it, but only against synthetic +# mktemp fixtures, and the factory-audit bats suite does the same. So a +# `Research doc:` that named the wrong file, or a slug absent from its Research +# registry, could only be found by hand-running the validator in a loop -- which +# is how 36 mismatches sat unnoticed while every gate stayed green. ADR-0028 +# promotes "the check ran and found a mismatch" from INFO to FAIL; without a +# caller across the corpus that FAIL tier would be inert. +# +# Exit codes, kept distinct on purpose: +# 0 every skill validated (INFO-only findings are printed, never swallowed) +# 1 at least one skill FAILed -- a real finding about the corpus +# 2 the gate itself could not run: validator missing, a validator exit 2 +# ("not auditable"), or NO skill with a references/sources.md found. A +# gate that discovers nothing must not read as a pass, and a skill that +# could not be audited must not read as a skill that failed the audit. +# +# The skill set is discovered by glob, not hardcoded, so a new skill is covered +# the moment it grows a references/sources.md. Run from repo root or pass +# REPO_ROOT as arg. + +REPO_ROOT="${1:-$(git rev-parse --show-toplevel 2>/dev/null || pwd)}" +if [[ ! -d "$REPO_ROOT" ]]; then + echo "Provenance corpus check failed: REPO_ROOT '$REPO_ROOT' is not a directory." >&2 + exit 2 +fi +REPO_ROOT="$(cd "$REPO_ROOT" && pwd)" + +VALIDATOR="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/validate-provenance.sh" +if [[ ! -f "$VALIDATOR" ]]; then + echo "Provenance corpus check failed: $VALIDATOR does not exist, so no skill was audited. If factory-audit's scripts moved, update this path." >&2 + exit 2 +fi + +shopt -s nullglob +sources_files=("$REPO_ROOT"/plugins/*/.apm/skills/*/references/sources.md) +shopt -u nullglob + +if [[ ${#sources_files[@]} -eq 0 ]]; then + echo "Provenance corpus check failed: found no plugins/*/.apm/skills/*/references/sources.md under $REPO_ROOT. Discovering zero skills is an error, not a pass -- the glob has gone stale or the corpus moved." >&2 + exit 2 +fi + +failing=() +errored=() +for sources in "${sources_files[@]}"; do + refs_dir="${sources%/*}" + skill_dir="${refs_dir%/*}" + rel="${skill_dir#"$REPO_ROOT"/plugins/}" + label="${rel%%/*}/${skill_dir##*/}" + + rc=0 + out="$(bash "$VALIDATOR" "$skill_dir" 2>&1)" || rc=$? + + case "$rc" in + 0) + # Exit 0 with output means INFO-only: a check that could not run, + # announced rather than skipped. Print it so it is not swallowed. + if [[ -n "$out" ]]; then + echo "== $label" + echo "$out" + fi + ;; + 1) + echo "== $label" + echo "$out" + failing+=("$label") + ;; + *) + echo "== $label (validator exit $rc)" + echo "$out" + errored+=("$label") + ;; + esac +done + +echo "" +echo "Provenance corpus: ${#sources_files[@]} skill(s) checked." + +if [[ ${#errored[@]} -gt 0 ]]; then + echo "Provenance corpus check errored (could not audit): ${errored[*]}" >&2 + if [[ ${#failing[@]} -gt 0 ]]; then + echo "Failing skills: ${failing[*]}" >&2 + fi + exit 2 +fi + +if [[ ${#failing[@]} -gt 0 ]]; then + echo "Failing skills: ${failing[*]}" >&2 + echo "Fix each FAIL above (see ADR-0028 for the Research doc / Basis grammar); INFO lines do not fail the gate." >&2 + exit 1 +fi + +echo "Provenance corpus check passed." diff --git a/tests/test-adr0020-contract.sh b/tests/test-adr0020-contract.sh index f74f79d..6b97779 100755 --- a/tests/test-adr0020-contract.sh +++ b/tests/test-adr0020-contract.sh @@ -744,6 +744,8 @@ EXPECTED = { 'apm pack --check-versions --check-clean --dry-run', ['pre-push']), 'check-scope-walkup-sync': ( 'bash scripts/check-scope-walkup-sync.sh', ['pre-push']), + 'check-provenance-corpus': ( + 'bash scripts/check-provenance-corpus.sh', ['pre-push']), 'check-skill-version-bump': ( 'bash scripts/check-skill-version-bump.sh', ['pre-push']), 'validate-marketplace': ( diff --git a/tests/test-check-provenance-corpus.sh b/tests/test-check-provenance-corpus.sh new file mode 100755 index 0000000..8397b85 --- /dev/null +++ b/tests/test-check-provenance-corpus.sh @@ -0,0 +1,186 @@ +#!/usr/bin/env bash +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SCRIPT="$REPO_ROOT/scripts/check-provenance-corpus.sh" +VALIDATOR_DIR="plugins/kyberforge/.apm/skills/factory-audit/scripts" +PASS=0 +FAIL=0 + +pass() { echo " PASS: $1"; PASS=$((PASS + 1)); } +fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); } + +FIXTURES=() +cleanup() { [[ ${#FIXTURES[@]} -eq 0 ]] || rm -rf "${FIXTURES[@]}"; } +trap cleanup EXIT + +# Per-run scratch for captured output, for the reason check-scope-walkup-sync's +# test gives: tests/run-tests.sh fans test scripts out concurrently. +RUN_TMP="$(mktemp -d)" +FIXTURES+=("$RUN_TMP") + +# A minimal REPO_ROOT: a .git entry (the validator's find_repo_root stops at +# it), a copy of the real validator at its real relative path, and one plugin +# holding a Research registry. Copying the real validator means the fixtures +# exercise the actual FAIL/INFO/exit contract rather than a stub of it. +make_repo() { + local dir + dir="$(mktemp -d)" + FIXTURES+=("$dir") + mkdir -p "$dir/.git" "$dir/$VALIDATOR_DIR" "$dir/plugins/p/docs/research/docs/t" + cp -R "$REPO_ROOT/$VALIDATOR_DIR/." "$dir/$VALIDATOR_DIR/" + cat > "$dir/plugins/p/docs/research/docs/t/sources.md" <<'EOF' +# Sources + +## known-slug + +**Status:** `extracted` +EOF + echo "$dir" +} + +# make_skill +make_skill() { + local repo="$1" name="$2" slug="$3" research="$4" + local skill="$repo/plugins/p/.apm/skills/$name" + mkdir -p "$skill/references" + cat > "$skill/SKILL.md" < "$skill/references/sources.md" < "$RUN_TMP/good.out" 2>&1; then + pass "exits 0 when every skill validates" +else + fail "exited non-zero on a clean corpus: $(cat "$RUN_TMP/good.out")" +fi + +# --- 2. A slug missing from the registry is a FAIL and is named --- +echo "" +echo "--- failing skill ---" +R="$(make_repo)" +make_skill "$R" good known-slug "$REGISTRY" +make_skill "$R" bad missing-slug "$REGISTRY" +set +e +bash "$SCRIPT" "$R" > "$RUN_TMP/bad.out" 2>&1 +rc=$? +set -e +if [[ $rc -eq 1 ]]; then + pass "exits 1 when one skill has a slug missing from its registry" +else + fail "expected exit 1, got $rc: $(cat "$RUN_TMP/bad.out")" +fi +if grep -q "bad" "$RUN_TMP/bad.out" && ! grep -qE "Failing skills:.*good" "$RUN_TMP/bad.out"; then + pass "summary line names the failing skill and not the passing one" +else + fail "summary did not name only the failing skill: $(cat "$RUN_TMP/bad.out")" +fi + +# --- 3. INFO-only passes but the INFO is printed, not swallowed --- +echo "" +echo "--- INFO-only skill ---" +R="$(make_repo)" +make_skill "$R" info-only known-slug "plugins/p/docs/research/docs/gone/sources.md" +if bash "$SCRIPT" "$R" > "$RUN_TMP/info.out" 2>&1; then + pass "exits 0 when the only findings are INFO" +else + fail "INFO-only corpus failed the gate: $(cat "$RUN_TMP/info.out")" +fi +if grep -q "INFO" "$RUN_TMP/info.out"; then + pass "INFO findings are printed" +else + fail "INFO finding was swallowed: $(cat "$RUN_TMP/info.out")" +fi + +# --- 4. Zero skills discovered is an error, not a pass --- +echo "" +echo "--- zero skills ---" +R="$(make_repo)" +set +e +bash "$SCRIPT" "$R" > "$RUN_TMP/zero.out" 2>&1 +rc=$? +set -e +if [[ $rc -eq 2 ]]; then + pass "exits 2 when no skill with references/sources.md is found" +else + fail "expected exit 2 for an empty corpus, got $rc: $(cat "$RUN_TMP/zero.out")" +fi + +# --- 5. A missing validator is a gate error (exit 2), never a pass --- +echo "" +echo "--- missing validator ---" +R="$(make_repo)" +make_skill "$R" good known-slug "$REGISTRY" +rm -rf "${R:?}/$VALIDATOR_DIR" +set +e +bash "$SCRIPT" "$R" > "$RUN_TMP/novalidator.out" 2>&1 +rc=$? +set -e +if [[ $rc -eq 2 ]]; then + pass "exits 2 when the validator is missing" +else + fail "expected exit 2 for a missing validator, got $rc: $(cat "$RUN_TMP/novalidator.out")" +fi + +# --- 6. A validator exit 2 (unauditable input) is a gate error, not a FAIL --- +echo "" +echo "--- validator exit 2 ---" +R="$(make_repo)" +make_skill "$R" good known-slug "$REGISTRY" +# Replace the entry point with a stub that reports "not auditable". +printf '#!/usr/bin/env bash\necho "stub: not auditable" >&2\nexit 2\n' \ + > "$R/$VALIDATOR_DIR/validate-provenance.sh" +set +e +bash "$SCRIPT" "$R" > "$RUN_TMP/exit2.out" 2>&1 +rc=$? +set -e +if [[ $rc -eq 2 ]]; then + pass "a validator exit 2 surfaces as gate exit 2, not as a skill FAIL" +else + fail "expected exit 2 to propagate, got $rc: $(cat "$RUN_TMP/exit2.out")" +fi + +# --- 7. The real corpus: reported, and the gate agrees with the validator --- +echo "" +echo "--- this repo's real corpus ---" +set +e +bash "$SCRIPT" "$REPO_ROOT" > "$RUN_TMP/real.out" 2>&1 +rc=$? +set -e +if [[ $rc -eq 0 || $rc -eq 1 ]]; then + pass "gate runs to a verdict (0 or 1) against the real corpus (exit $rc)" +else + fail "gate errored (exit $rc) against the real corpus: $(cat "$RUN_TMP/real.out")" +fi + +echo "" +echo "Results: $PASS passed, $FAIL failed" +[[ $FAIL -eq 0 ]]