feat(gates): sweep the provenance corpus on pre-push
Nothing ran validate-provenance.sh across the real corpus, so the 36 INFOs it reported for Research doc mismatches were found only by a manual loop, and a FAIL tier would have been inert. Add scripts/check-provenance-corpus.sh, which runs the validator over every plugins/*/.apm/skills/*/ that has references/sources.md. Exit 1 when any skill FAILs, naming them; INFO lines are printed but do not fail; exit 2 when the gate cannot run (missing validator, validator exit 2, or no skills found). Registered as a pre-push hook shaped like check-scope-walkup-sync, documented in docs/spec/gates.md, and pinned in test-adr0020-contract.sh's list of repo-authored hooks. Refs: #121 ADR: 0028 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EGHFJextYtVQseaHPDDhxB
This commit is contained in:
@@ -220,6 +220,21 @@ repos:
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
|
||||
- id: check-provenance-corpus
|
||||
name: Check provenance across the skill corpus
|
||||
description: Run factory-audit's validate-provenance.sh over every plugins/*/.apm/skills/*/ that has references/sources.md and fail on any FAIL (ADR-0028, #121)
|
||||
entry: bash scripts/check-provenance-corpus.sh
|
||||
language: system
|
||||
stages: [pre-push]
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
# Nothing else runs validate-provenance.sh over the real corpus --
|
||||
# check-scope-walkup-sync exercises it against synthetic fixtures only --
|
||||
# so ADR-0028's FAIL tier for a Research doc mismatch would be inert
|
||||
# without this caller. The skill set is globbed, not counted, and
|
||||
# discovering zero skills is an error (exit 2), not a pass. Needs no
|
||||
# network; needs python3, which the validator's own preflight names.
|
||||
|
||||
- id: check-skill-version-bump
|
||||
name: Check changed skills bump metadata.version
|
||||
description: On every push, fail if a skill directory changed (tests/ excluded) since the merge-base with main without its SKILL.md metadata.version rising above both that merge-base's and main's tip's (ADR-0022)
|
||||
|
||||
@@ -42,7 +42,7 @@ is checked out. Push one ref at a time when the gate matters.
|
||||
|
||||
## The pre-push gate
|
||||
|
||||
Eight hooks, grouped below by what they guard rather than by the order `.pre-commit-config.yaml` declares them in.
|
||||
Nine hooks, grouped below by what they guard rather than by the order `.pre-commit-config.yaml` declares them in.
|
||||
|
||||
**Core checks**
|
||||
|
||||
@@ -66,6 +66,7 @@ version-blind, so a stale key deploys fine (see [apm gates](#apm-gates)).
|
||||
| Hook | Guards |
|
||||
|---|---|
|
||||
| `check-apm-agents-valid` | runs `factory-audit`'s `validate.sh` over every real `plugins/*/.apm/agents/*.agent.md` (see [Agent files](#agent-files-take-the-description-gates-not-the-body-gate)) |
|
||||
| `check-provenance-corpus` | runs `factory-audit`'s `validate-provenance.sh` over every real `plugins/*/.apm/skills/*/` that has a `references/sources.md`, failing on any FAIL (see [The provenance corpus sweep](#the-provenance-corpus-sweep-adr-0028)) |
|
||||
|
||||
**apm's own gates**
|
||||
|
||||
@@ -576,6 +577,38 @@ follows symlinks with `find -L` because vale does.
|
||||
on `files:` patterns that match single markdown files, and only the `-d "$arg"` branch mirrors a
|
||||
directory. The exposed caller is the hand-invoked `vale-wrap.sh <dir>`.
|
||||
|
||||
## The provenance corpus sweep (ADR-0028)
|
||||
|
||||
`check-provenance-corpus` runs `validate-provenance.sh` over every real
|
||||
`plugins/*/.apm/skills/*/` directory that has a `references/sources.md`, and fails on any FAIL. The set
|
||||
is discovered by glob, not counted, so a new skill is covered the moment it grows a `sources.md`, and
|
||||
**discovering zero skills is an error, not a pass**.
|
||||
|
||||
The hook exists because nothing else ran the validator over the real corpus.
|
||||
`check-scope-walkup-sync` invokes it only against synthetic `mktemp` fixtures, and `factory-audit`'s
|
||||
bats suite does the same. So a `Research doc:` naming the wrong file, or a slug absent from its
|
||||
Research registry, could only be found by hand-running the validator in a loop. That is how 36
|
||||
mismatches (#121) reported INFO while every gate stayed green. ADR-0028 promotes "the check ran and
|
||||
found a mismatch" from INFO to FAIL; without a caller across the corpus that FAIL tier would be inert.
|
||||
|
||||
It reuses the validators' exit contract (see
|
||||
[the three exit tiers](#the-three-exit-tiers-of-factory-audits-validators)) and keeps the tiers apart:
|
||||
|
||||
| Exit | Means |
|
||||
|---|---|
|
||||
| **0** | every skill validated. INFO-only findings are printed, never swallowed |
|
||||
| **1** | at least one skill FAILed. The summary line names the failing skills |
|
||||
| **2** | the gate could not run: the validator is missing, a skill's validator run exited 2 ("not auditable"), or no skill with a `references/sources.md` was found |
|
||||
|
||||
A validator exit 2 is reported as a gate error, not as a FAIL about that skill: it says the audit never
|
||||
happened, and the skill has not been shown to be wrong.
|
||||
|
||||
An unresolvable `Research doc:` path stays INFO by design, because a deployed copy of a skill outside
|
||||
this repo will not carry the research docs (see `skill-file-structure.md`'s `sources.md` exemption).
|
||||
This repo's own corpus is audited from the authoring source, where every path resolves, so an INFO
|
||||
printed here is worth reading. Needs no network; needs `python3`, which the validator's own preflight
|
||||
names.
|
||||
|
||||
## Current retrofit status
|
||||
|
||||
The ADR-0020 gates ship hot, with no baseline file — a shrinking baseline was considered and
|
||||
|
||||
101
scripts/check-provenance-corpus.sh
Executable file
101
scripts/check-provenance-corpus.sh
Executable file
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Corpus-wide provenance sweep: runs factory-audit's validate-provenance.sh over
|
||||
# every plugins/*/.apm/skills/*/ directory that has a references/sources.md, and
|
||||
# fails on any FAIL.
|
||||
#
|
||||
# WHY THIS GATE EXISTS (ADR-0028, #121). Nothing else runs the validator over the
|
||||
# real corpus. check-scope-walkup-sync.sh invokes it, but only against synthetic
|
||||
# mktemp fixtures, and the factory-audit bats suite does the same. So a
|
||||
# `Research doc:` that named the wrong file, or a slug absent from its Research
|
||||
# registry, could only be found by hand-running the validator in a loop -- which
|
||||
# is how 36 mismatches sat unnoticed while every gate stayed green. ADR-0028
|
||||
# promotes "the check ran and found a mismatch" from INFO to FAIL; without a
|
||||
# caller across the corpus that FAIL tier would be inert.
|
||||
#
|
||||
# Exit codes, kept distinct on purpose:
|
||||
# 0 every skill validated (INFO-only findings are printed, never swallowed)
|
||||
# 1 at least one skill FAILed -- a real finding about the corpus
|
||||
# 2 the gate itself could not run: validator missing, a validator exit 2
|
||||
# ("not auditable"), or NO skill with a references/sources.md found. A
|
||||
# gate that discovers nothing must not read as a pass, and a skill that
|
||||
# could not be audited must not read as a skill that failed the audit.
|
||||
#
|
||||
# The skill set is discovered by glob, not hardcoded, so a new skill is covered
|
||||
# the moment it grows a references/sources.md. Run from repo root or pass
|
||||
# REPO_ROOT as arg.
|
||||
|
||||
REPO_ROOT="${1:-$(git rev-parse --show-toplevel 2>/dev/null || pwd)}"
|
||||
if [[ ! -d "$REPO_ROOT" ]]; then
|
||||
echo "Provenance corpus check failed: REPO_ROOT '$REPO_ROOT' is not a directory." >&2
|
||||
exit 2
|
||||
fi
|
||||
REPO_ROOT="$(cd "$REPO_ROOT" && pwd)"
|
||||
|
||||
VALIDATOR="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/validate-provenance.sh"
|
||||
if [[ ! -f "$VALIDATOR" ]]; then
|
||||
echo "Provenance corpus check failed: $VALIDATOR does not exist, so no skill was audited. If factory-audit's scripts moved, update this path." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
shopt -s nullglob
|
||||
sources_files=("$REPO_ROOT"/plugins/*/.apm/skills/*/references/sources.md)
|
||||
shopt -u nullglob
|
||||
|
||||
if [[ ${#sources_files[@]} -eq 0 ]]; then
|
||||
echo "Provenance corpus check failed: found no plugins/*/.apm/skills/*/references/sources.md under $REPO_ROOT. Discovering zero skills is an error, not a pass -- the glob has gone stale or the corpus moved." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
failing=()
|
||||
errored=()
|
||||
for sources in "${sources_files[@]}"; do
|
||||
refs_dir="${sources%/*}"
|
||||
skill_dir="${refs_dir%/*}"
|
||||
rel="${skill_dir#"$REPO_ROOT"/plugins/}"
|
||||
label="${rel%%/*}/${skill_dir##*/}"
|
||||
|
||||
rc=0
|
||||
out="$(bash "$VALIDATOR" "$skill_dir" 2>&1)" || rc=$?
|
||||
|
||||
case "$rc" in
|
||||
0)
|
||||
# Exit 0 with output means INFO-only: a check that could not run,
|
||||
# announced rather than skipped. Print it so it is not swallowed.
|
||||
if [[ -n "$out" ]]; then
|
||||
echo "== $label"
|
||||
echo "$out"
|
||||
fi
|
||||
;;
|
||||
1)
|
||||
echo "== $label"
|
||||
echo "$out"
|
||||
failing+=("$label")
|
||||
;;
|
||||
*)
|
||||
echo "== $label (validator exit $rc)"
|
||||
echo "$out"
|
||||
errored+=("$label")
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Provenance corpus: ${#sources_files[@]} skill(s) checked."
|
||||
|
||||
if [[ ${#errored[@]} -gt 0 ]]; then
|
||||
echo "Provenance corpus check errored (could not audit): ${errored[*]}" >&2
|
||||
if [[ ${#failing[@]} -gt 0 ]]; then
|
||||
echo "Failing skills: ${failing[*]}" >&2
|
||||
fi
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [[ ${#failing[@]} -gt 0 ]]; then
|
||||
echo "Failing skills: ${failing[*]}" >&2
|
||||
echo "Fix each FAIL above (see ADR-0028 for the Research doc / Basis grammar); INFO lines do not fail the gate." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Provenance corpus check passed."
|
||||
@@ -744,6 +744,8 @@ EXPECTED = {
|
||||
'apm pack --check-versions --check-clean --dry-run', ['pre-push']),
|
||||
'check-scope-walkup-sync': (
|
||||
'bash scripts/check-scope-walkup-sync.sh', ['pre-push']),
|
||||
'check-provenance-corpus': (
|
||||
'bash scripts/check-provenance-corpus.sh', ['pre-push']),
|
||||
'check-skill-version-bump': (
|
||||
'bash scripts/check-skill-version-bump.sh', ['pre-push']),
|
||||
'validate-marketplace': (
|
||||
|
||||
186
tests/test-check-provenance-corpus.sh
Executable file
186
tests/test-check-provenance-corpus.sh
Executable file
@@ -0,0 +1,186 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="$REPO_ROOT/scripts/check-provenance-corpus.sh"
|
||||
VALIDATOR_DIR="plugins/kyberforge/.apm/skills/factory-audit/scripts"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
|
||||
FIXTURES=()
|
||||
cleanup() { [[ ${#FIXTURES[@]} -eq 0 ]] || rm -rf "${FIXTURES[@]}"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
# Per-run scratch for captured output, for the reason check-scope-walkup-sync's
|
||||
# test gives: tests/run-tests.sh fans test scripts out concurrently.
|
||||
RUN_TMP="$(mktemp -d)"
|
||||
FIXTURES+=("$RUN_TMP")
|
||||
|
||||
# A minimal REPO_ROOT: a .git entry (the validator's find_repo_root stops at
|
||||
# it), a copy of the real validator at its real relative path, and one plugin
|
||||
# holding a Research registry. Copying the real validator means the fixtures
|
||||
# exercise the actual FAIL/INFO/exit contract rather than a stub of it.
|
||||
make_repo() {
|
||||
local dir
|
||||
dir="$(mktemp -d)"
|
||||
FIXTURES+=("$dir")
|
||||
mkdir -p "$dir/.git" "$dir/$VALIDATOR_DIR" "$dir/plugins/p/docs/research/docs/t"
|
||||
cp -R "$REPO_ROOT/$VALIDATOR_DIR/." "$dir/$VALIDATOR_DIR/"
|
||||
cat > "$dir/plugins/p/docs/research/docs/t/sources.md" <<'EOF'
|
||||
# Sources
|
||||
|
||||
## known-slug
|
||||
|
||||
**Status:** `extracted`
|
||||
EOF
|
||||
echo "$dir"
|
||||
}
|
||||
|
||||
# make_skill <repo> <name> <slug> <research-doc-value>
|
||||
make_skill() {
|
||||
local repo="$1" name="$2" slug="$3" research="$4"
|
||||
local skill="$repo/plugins/p/.apm/skills/$name"
|
||||
mkdir -p "$skill/references"
|
||||
cat > "$skill/SKILL.md" <<EOF
|
||||
---
|
||||
name: $name
|
||||
description: A valid skill description.
|
||||
metadata:
|
||||
source_keys:
|
||||
- $slug
|
||||
---
|
||||
|
||||
## Step 1
|
||||
|
||||
Do the thing.
|
||||
EOF
|
||||
cat > "$skill/references/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## $slug
|
||||
|
||||
- **URL:** https://example.com/$slug
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** SKILL.md
|
||||
- **Research doc:** $research
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
}
|
||||
|
||||
REGISTRY="plugins/p/docs/research/docs/t/sources.md"
|
||||
|
||||
# --- 1. A skill whose slug resolves in the registry passes, quietly ---
|
||||
echo ""
|
||||
echo "--- passing skill ---"
|
||||
R="$(make_repo)"
|
||||
make_skill "$R" good known-slug "$REGISTRY"
|
||||
if bash "$SCRIPT" "$R" > "$RUN_TMP/good.out" 2>&1; then
|
||||
pass "exits 0 when every skill validates"
|
||||
else
|
||||
fail "exited non-zero on a clean corpus: $(cat "$RUN_TMP/good.out")"
|
||||
fi
|
||||
|
||||
# --- 2. A slug missing from the registry is a FAIL and is named ---
|
||||
echo ""
|
||||
echo "--- failing skill ---"
|
||||
R="$(make_repo)"
|
||||
make_skill "$R" good known-slug "$REGISTRY"
|
||||
make_skill "$R" bad missing-slug "$REGISTRY"
|
||||
set +e
|
||||
bash "$SCRIPT" "$R" > "$RUN_TMP/bad.out" 2>&1
|
||||
rc=$?
|
||||
set -e
|
||||
if [[ $rc -eq 1 ]]; then
|
||||
pass "exits 1 when one skill has a slug missing from its registry"
|
||||
else
|
||||
fail "expected exit 1, got $rc: $(cat "$RUN_TMP/bad.out")"
|
||||
fi
|
||||
if grep -q "bad" "$RUN_TMP/bad.out" && ! grep -qE "Failing skills:.*good" "$RUN_TMP/bad.out"; then
|
||||
pass "summary line names the failing skill and not the passing one"
|
||||
else
|
||||
fail "summary did not name only the failing skill: $(cat "$RUN_TMP/bad.out")"
|
||||
fi
|
||||
|
||||
# --- 3. INFO-only passes but the INFO is printed, not swallowed ---
|
||||
echo ""
|
||||
echo "--- INFO-only skill ---"
|
||||
R="$(make_repo)"
|
||||
make_skill "$R" info-only known-slug "plugins/p/docs/research/docs/gone/sources.md"
|
||||
if bash "$SCRIPT" "$R" > "$RUN_TMP/info.out" 2>&1; then
|
||||
pass "exits 0 when the only findings are INFO"
|
||||
else
|
||||
fail "INFO-only corpus failed the gate: $(cat "$RUN_TMP/info.out")"
|
||||
fi
|
||||
if grep -q "INFO" "$RUN_TMP/info.out"; then
|
||||
pass "INFO findings are printed"
|
||||
else
|
||||
fail "INFO finding was swallowed: $(cat "$RUN_TMP/info.out")"
|
||||
fi
|
||||
|
||||
# --- 4. Zero skills discovered is an error, not a pass ---
|
||||
echo ""
|
||||
echo "--- zero skills ---"
|
||||
R="$(make_repo)"
|
||||
set +e
|
||||
bash "$SCRIPT" "$R" > "$RUN_TMP/zero.out" 2>&1
|
||||
rc=$?
|
||||
set -e
|
||||
if [[ $rc -eq 2 ]]; then
|
||||
pass "exits 2 when no skill with references/sources.md is found"
|
||||
else
|
||||
fail "expected exit 2 for an empty corpus, got $rc: $(cat "$RUN_TMP/zero.out")"
|
||||
fi
|
||||
|
||||
# --- 5. A missing validator is a gate error (exit 2), never a pass ---
|
||||
echo ""
|
||||
echo "--- missing validator ---"
|
||||
R="$(make_repo)"
|
||||
make_skill "$R" good known-slug "$REGISTRY"
|
||||
rm -rf "${R:?}/$VALIDATOR_DIR"
|
||||
set +e
|
||||
bash "$SCRIPT" "$R" > "$RUN_TMP/novalidator.out" 2>&1
|
||||
rc=$?
|
||||
set -e
|
||||
if [[ $rc -eq 2 ]]; then
|
||||
pass "exits 2 when the validator is missing"
|
||||
else
|
||||
fail "expected exit 2 for a missing validator, got $rc: $(cat "$RUN_TMP/novalidator.out")"
|
||||
fi
|
||||
|
||||
# --- 6. A validator exit 2 (unauditable input) is a gate error, not a FAIL ---
|
||||
echo ""
|
||||
echo "--- validator exit 2 ---"
|
||||
R="$(make_repo)"
|
||||
make_skill "$R" good known-slug "$REGISTRY"
|
||||
# Replace the entry point with a stub that reports "not auditable".
|
||||
printf '#!/usr/bin/env bash\necho "stub: not auditable" >&2\nexit 2\n' \
|
||||
> "$R/$VALIDATOR_DIR/validate-provenance.sh"
|
||||
set +e
|
||||
bash "$SCRIPT" "$R" > "$RUN_TMP/exit2.out" 2>&1
|
||||
rc=$?
|
||||
set -e
|
||||
if [[ $rc -eq 2 ]]; then
|
||||
pass "a validator exit 2 surfaces as gate exit 2, not as a skill FAIL"
|
||||
else
|
||||
fail "expected exit 2 to propagate, got $rc: $(cat "$RUN_TMP/exit2.out")"
|
||||
fi
|
||||
|
||||
# --- 7. The real corpus: reported, and the gate agrees with the validator ---
|
||||
echo ""
|
||||
echo "--- this repo's real corpus ---"
|
||||
set +e
|
||||
bash "$SCRIPT" "$REPO_ROOT" > "$RUN_TMP/real.out" 2>&1
|
||||
rc=$?
|
||||
set -e
|
||||
if [[ $rc -eq 0 || $rc -eq 1 ]]; then
|
||||
pass "gate runs to a verdict (0 or 1) against the real corpus (exit $rc)"
|
||||
else
|
||||
fail "gate errored (exit $rc) against the real corpus: $(cat "$RUN_TMP/real.out")"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
Reference in New Issue
Block a user