refactor(kyberforge)!: merge skill-audit and agent-audit into factory-audit
Why The two audit skills carried 1,724 lines of byte-identical duplication: the ADR-0020 boundary resolver (1,061), vale-wrap.sh (526), the Vale style rules (44) and the Contributing-files parser (93). Nothing shared them — they were held in sync by a 413-line pre-push gate and its 797-line test suite. Sync-by-gate had already failed once: at484357athe two parser copies drifted into different spellings of the bullet loop while a docstring asserted they were identical. That drift was behaviour-neutral and was re-unified by hand at598a7c3, so the copies were identical at merge time — but nothing had caught it, and the next drift need not be neutral. Implementation Notes Self-containment binds BETWEEN skills, not within one. The agentskills.io spec forbids reaching across skill directories, which is why two separate skills needed embedded copies; two files inside ONE skill may source a third. That is the whole reason the merge removes duplication rather than relocating it. The union of both bodies measured 1,532 words against BODY_MAX_WORDS=900, and only 211 of those words were shared, so SKILL.md is a dispatch body. Step 0 resolves the flow from the target path before any validation, and its table mirrors validate.sh's detection exactly: a directory holding SKILL.md or a SKILL.md file (skill); a *.agent.md, or a .md directly under an agents/ directory (agent); anything else stops without running a validator. Steps 1-3 live in references/skill-flow.md and references/agent-flow.md, and gotchas that apply to one flow live in that flow's file, since it is loaded on every invocation anyway. If validate.sh reports on the other artifact type, the body restarts at Step 0. Named factory-audit rather than forge-audit because forge is a live skill, and a family prefix that matches a live sibling reads as ownership rather than membership. The description carries one arrow per boundary target, because ADR-0020 resolves only the first target after an arrow. It drops the quoted "audit this skill"-style phrases, which restated "audited" in a second register (ADR-0020's duplicate-register rule). 241 characters, Gotchas 16% of the body: no size SUGGESTIONs. The boundary resolver stays embedded in two files rather than imported: a cache-installed plugin cannot read outside its own directory, and the repo-root hook resolves via .pre-commit-hooks.yaml where entry[0] is the only token pre-commit rewrites, so no single file is reachable by both. tests/test-adr0020-contract.sh hashes both copies for byte-identity, and asserts validate.sh sources the resolver and that no third copy exists. The entry scripts classify the target from its resolved parent directory, so a bare agent filename typed inside agents/ works; resolve SCRIPT_DIR CDPATH-safely; and exit 2 when a lib-*.sh is missing, rather than dying with exit 1, the tier the flows relay as real findings. The provenance run functions stash their findings code in KYBERFORGE_PROV_RC and return 0, so validate-provenance.sh calls them UNTESTED. Testing a function's status (`f || RC=$?`) disables errexit for its entire body, and no subshell or `set -e` inside can re-arm it once the call sits in a condition context (measured, both spellings). Their error paths use `exit`, which is unaffected either way; this keeps errexit armed for anything added later. Case 0's readability guard reads the file instead of asking `[[ -r ]]`. `-r` is access(2), which answers yes for uid 0 even on a mode-000 file, and this repo's dev environment is root -- so the guard could never fire where it exists to fire. A read attempt is also the stricter question, catching EIO. This is the reasoning scripts/check-vale-style-sync.sh carried before this commit deleted it; the hazard did not go with it. All three entry scripts are CDPATH-safe, vale-wrap.sh included: both of its cd sites are cleared, the --config resolution and the directory-mirror walk, where an exported CDPATH would otherwise print a decoy path into the -print0 stream and build the mirror from the decoy's files. The two remaining bare cd calls take absolute paths, which CDPATH is never consulted for. Impact BREAKING: skill-audit and agent-audit no longer exist as invocable skills. kyberforge goes to 2.0.0 (catalog 0.4.7). Check logic is unchanged: differential runs of the old and new validators across every skill and agent produced byte-identical stdout, stderr and exit codes, and the reconstructed Python payloads differ only in comments and the references/field-inventory.md -> agent-field-inventory.md rename. One doctrine governs the tiers: exit 0 is audited and clean, exit 1 is audited with findings OR a target present but unreadable, exit 2 is that nothing was audited at all. Edge paths DID change, deliberately (full table in ADR-0025): - a missing target exits 2 (never ran), not 1, under its own "does not exist" message; detection is by path shape, so a shape-matching path that is simply absent used to reach the validator and come back as a FAIL against a file that never existed; - an unshaped target exits 2 under the generic "matches neither" message, and a directory with no SKILL.md under a third, distinct one -- three exit-2 messages, not one; - a dangling symlink or a symlink loop stays exit 1: it is present but broken, which is a finding about the artifact rather than a usage error; - a SKILL.md file path is audited as its skill directory instead of refused; - a .md agent outside an agents/ directory is refused rather than audited; - a missing script library, a missing python3, a missing PyYAML, and no argument at all each exit 2. validate-provenance.sh already exited 2 for the last two; validate.sh now matches it. .pre-commit-hooks.yaml is a published contract consumed by external repos. Both hook IDs and both files: regexes are unchanged; only entry: and description: moved. scripts/check-vale-style-sync.sh (413), scripts/sync-vale-styles.sh (21), tests/test-check-vale-style-sync.sh (797) and agent-audit/scripts/README.md (47) are deleted. The checker made 17 assertions: 6 compared the two Vale copies and are moot; 10 are rehomed into tests/test-vale-wrap.sh (case 0, cases 28-31, and the suite's Vale-absent skip); and the cross-manifest files: agreement check, which selected hooks by entry: and so could not survive both hooks sharing one, is ported as case 33 pairing hooks by id:. Cases 28, 30 and 33 carry mutation self-tests; narrowing the local skill prefilter to 6 of 38 SKILL.md files now fails the suite. Skills go 39 to 38. Pre-push goes 9 repo-authored hooks to 8. ADR: 0025 BREAKING-CHANGE: the skill-audit and agent-audit skills are removed. Both flows are served by factory-audit, which auto-detects whether it was handed a skill directory or an agent file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YR2CjVumUbEGWcMikcoXBD
This commit is contained in:
@@ -22,11 +22,13 @@
|
||||
# green gate having verified 15 of the 17 suites that existed then.
|
||||
# Exactly the vacuous-pass class the rest of this file exists to close.
|
||||
#
|
||||
# Deliberately its own switch, NOT folded into
|
||||
# CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE. That one governs whether
|
||||
# check-vale-style-sync may downgrade itself; this one governs whether the test
|
||||
# dispatcher tolerates an unrunnable suite. They are separate decisions and one
|
||||
# flag disarming both gates is how an opt-out quietly grows blast radius.
|
||||
# Deliberately its own switch, scoped to this dispatcher alone: it governs
|
||||
# whether an unrunnable suite is tolerated, nothing else. It was once kept
|
||||
# separate from CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE, which governed whether
|
||||
# check-vale-style-sync could downgrade itself; that gate is retired (ADR-0025
|
||||
# merged the two Vale copies it diffed), but the rule that retired it does not
|
||||
# apply here. Keep any future vale-related opt-out separate too — one flag
|
||||
# disarming several gates is how an opt-out quietly grows blast radius.
|
||||
#
|
||||
# TEST_DIR — override root to search for test-*.sh (default: REPO_ROOT); used by tests.
|
||||
set -euo pipefail
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
# child bullets were counted as top-level entries, `## Gotcha handling` was read
|
||||
# as the Gotchas section, and a documented-then-removed references/ file became a
|
||||
# hard ERROR. The skills most likely to carry such an example are skill-author and
|
||||
# skill-audit — the two that DOCUMENT these conventions — so a gate that fires on
|
||||
# factory-audit — the two that DOCUMENT these conventions — so a gate that fires on
|
||||
# them is a gate nobody can turn on.
|
||||
#
|
||||
# Every case is a matched pair: the check fires just over its boundary, and stays
|
||||
@@ -198,7 +198,7 @@ expect "a Gotchas section at 26% of the body raises a SUGGESTION" \
|
||||
# ---------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- a ## Gotchas heading inside a fenced block is not the Gotchas section ---"
|
||||
# skill-author and skill-audit both document this convention by showing it. If a
|
||||
# skill-author and factory-audit both document this convention by showing it. If a
|
||||
# fenced example counted, the two skills that define the rule would be the two
|
||||
# most likely to fail it.
|
||||
F_FENCED_HEADING="$(make_skill gotchas-fenced-heading "$CLEAN_DESC" <<EOF
|
||||
|
||||
@@ -3,21 +3,41 @@
|
||||
# itself. None of them was pinned anywhere before this file, and each one fails
|
||||
# silently — which is the whole reason they need a test rather than a comment:
|
||||
#
|
||||
# 1. "ONE resolver, embedded VERBATIM in three scripts." The block between the
|
||||
# 1. "ONE resolver, embedded VERBATIM in two scripts." The block between the
|
||||
# BEGIN/END markers is copied, not imported, because a cache-installed
|
||||
# plugin's scripts cannot read files outside their own plugin directory.
|
||||
# Nothing but this file asserts the three copies are still identical, and a
|
||||
# one-line edit to a single copy is invisible: every constant-agreement
|
||||
# ADR-0025 merged skill-audit and agent-audit, which dropped the count from
|
||||
# three copies to two: factory-audit now holds ONE copy in a sourced
|
||||
# lib-boundary-resolver.sh, and scripts/skill-size-check.sh keeps its
|
||||
# embedded copy because it is a repo-root hook consumed through
|
||||
# .pre-commit-hooks.yaml, where entry[0] is the only token pre-commit
|
||||
# rewrites — it cannot reach a file inside the plugin at a path any consumer
|
||||
# has. Nothing but this file asserts the two copies are still identical, and
|
||||
# a one-line edit to a single copy is invisible: every constant-agreement
|
||||
# assertion in tests/test-skill-size-check.sh still passes, because the
|
||||
# CONSTANTS are not what drifted.
|
||||
# 1b. The same claim, one directory over, for the Contributing-files parser
|
||||
# embedded in both validate-provenance.sh copies. That one was worse: the
|
||||
# agent-audit copy's docstring ASSERTED it was kept behaviourally identical
|
||||
# to skill-audit's, and the two had already drifted.
|
||||
# 2. Both interpreter preflights, in all three scripts. python3 and PyYAML are
|
||||
# declared HARD dependencies precisely so a missing one cannot turn into a
|
||||
# vacuous pass, and the two are checked separately so the message names the
|
||||
# thing to install rather than the wrong one.
|
||||
# 1a. The resolver's two copies are the ONLY two, and validate.sh sources
|
||||
# factory-audit's in both mode branches — the same authority checks 1b
|
||||
# makes for the parser, which byte-identity alone cannot make.
|
||||
# 1b. The same claim, one directory over, for the Contributing-files parser.
|
||||
# That one was worse: it was embedded in both validate-provenance.sh copies,
|
||||
# the agent-audit copy's docstring ASSERTED it was kept behaviourally
|
||||
# identical to skill-audit's, and the two had drifted (cosmetically, at
|
||||
# 484357a, re-unified at 598a7c3 — nothing had caught it). ADR-0025
|
||||
# removed the second copy, so there is no longer a pair to hash — but
|
||||
# deleting the assertion would restore exactly the condition that let the
|
||||
# drift happen, so it is CONVERTED rather than dropped: it now pins that
|
||||
# lib-contributing-files.sh is the single authority (one marker pair), that
|
||||
# validate-provenance.sh actually SOURCES it, and that nobody has re-inlined
|
||||
# the parser into a mode library or anywhere else in the tree.
|
||||
# 2. Both interpreter preflights, in both scripts and both of the merged
|
||||
# entry point's modes. python3 and PyYAML are declared HARD dependencies
|
||||
# precisely so a missing one cannot turn into a vacuous pass, and the two
|
||||
# are checked separately so the message names the thing to install rather
|
||||
# than the wrong one. factory-audit/scripts/validate.sh detects its mode
|
||||
# first and only then calls the preflight, which lives in the mode library
|
||||
# it sources (lib-checks-skill.sh / lib-checks-agent.sh) — so each mode is
|
||||
# probed with its own target shape, not just one of them.
|
||||
# 3. `verbose: true` on the skill-size-check hook. It is the ENTIRE delivery
|
||||
# mechanism for the SUGGESTION tier: pre-commit prints nothing at all for a
|
||||
# passing hook, and a SUGGESTION deliberately does not fail, so dropping
|
||||
@@ -27,8 +47,12 @@ set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||
FACTORY_AUDIT="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit"
|
||||
# ADR-0025: one auto-detecting entry point, and one sourced copy of the resolver
|
||||
# behind it. The entry point is what the preflight assertions run; the library is
|
||||
# what the byte-identity assertion hashes.
|
||||
FACTORY_VALIDATE="$FACTORY_AUDIT/scripts/validate.sh"
|
||||
FACTORY_RESOLVER="$FACTORY_AUDIT/scripts/lib-boundary-resolver.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
@@ -42,18 +66,23 @@ BEGIN_MARKER='# ===== BEGIN ADR-0020 SHARED BOUNDARY RESOLVER ====='
|
||||
END_MARKER='# ===== END ADR-0020 SHARED BOUNDARY RESOLVER ====='
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The shared resolver block is byte-identical in all three scripts
|
||||
# 1. The shared resolver block is byte-identical in both scripts
|
||||
# ---------------------------------------------------------------------------
|
||||
# Two copies, not three, since ADR-0025 merged skill-audit and agent-audit:
|
||||
# scripts/skill-size-check.sh (embedded — sourcing the plugin's copy was
|
||||
# considered and refuted, see the header) and factory-audit's single sourced
|
||||
# lib-boundary-resolver.sh. The claim is unchanged and so is the check: the
|
||||
# span between the markers must be byte-identical wherever it appears.
|
||||
echo ""
|
||||
echo "--- the ADR-0020 shared resolver block is byte-identical in all three scripts ---"
|
||||
echo "--- the ADR-0020 shared resolver block is byte-identical in both scripts ---"
|
||||
|
||||
# Marker discipline first. An unbalanced or duplicated marker pair makes the
|
||||
# extraction below silently measure the wrong span — a sed range that never
|
||||
# closes swallows the rest of the file, and one that opens twice concatenates
|
||||
# two spans. Both would still compare "equal" if all three were mangled the
|
||||
# two spans. Both would still compare "equal" if both were mangled the
|
||||
# same way, so the shape is asserted before the contents.
|
||||
MARKERS_OK=true
|
||||
for f in "$HOOK" "$SKILL_VALIDATE" "$AGENT_VALIDATE"; do
|
||||
for f in "$HOOK" "$FACTORY_RESOLVER"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
fail "script not found: $f"
|
||||
MARKERS_OK=false
|
||||
@@ -74,97 +103,337 @@ if ! $MARKERS_OK; then
|
||||
else
|
||||
HASHES=()
|
||||
LINECOUNTS=()
|
||||
for f in "$HOOK" "$SKILL_VALIDATE" "$AGENT_VALIDATE"; do
|
||||
for f in "$HOOK" "$FACTORY_RESOLVER"; do
|
||||
out="$TMPDIR_T/block-$(echo "$f" | md5sum | cut -c1-8).txt"
|
||||
sed -n "/^${BEGIN_MARKER}\$/,/^${END_MARKER}\$/p" "$f" > "$out"
|
||||
HASHES+=("$(md5sum < "$out" | cut -d' ' -f1)")
|
||||
LINECOUNTS+=("$(wc -l < "$out" | tr -d ' ')")
|
||||
done
|
||||
if [[ "${HASHES[0]}" == "${HASHES[1]}" && "${HASHES[1]}" == "${HASHES[2]}" ]]; then
|
||||
pass "all three copies hash to ${HASHES[0]} (${LINECOUNTS[0]} lines) — agreement by construction, not by coincidence"
|
||||
if [[ "${HASHES[0]}" == "${HASHES[1]}" ]]; then
|
||||
pass "both copies hash to ${HASHES[0]} (${LINECOUNTS[0]} lines) — agreement by construction, not by coincidence"
|
||||
else
|
||||
fail "the shared resolver has DRIFTED: skill-size-check=${HASHES[0]} (${LINECOUNTS[0]} lines), skill-audit=${HASHES[1]} (${LINECOUNTS[1]} lines), agent-audit=${HASHES[2]} (${LINECOUNTS[2]} lines). Edit one copy, then paste it over the other two."
|
||||
fail "the shared resolver has DRIFTED: skill-size-check=${HASHES[0]} (${LINECOUNTS[0]} lines), factory-audit/scripts/lib-boundary-resolver.sh=${HASHES[1]} (${LINECOUNTS[1]} lines). Edit one copy, then paste it over the other."
|
||||
fi
|
||||
# A block that has been emptied out would hash equal in all three and pass the
|
||||
# comparison above while enforcing nothing. The resolver is ~570 lines; 100 is
|
||||
# a floor low enough never to need maintenance and high enough that a gutted
|
||||
# block cannot sneak past.
|
||||
# A block that has been emptied out would hash equal in both and pass the
|
||||
# comparison above while enforcing nothing. The resolver is ~1,060 lines; 100
|
||||
# is a floor low enough never to need maintenance and high enough that a
|
||||
# gutted block cannot sneak past.
|
||||
if [[ "${LINECOUNTS[0]}" -gt 100 ]]; then
|
||||
pass "the extracted block is ${LINECOUNTS[0]} lines — the comparison is over real content, not an empty span"
|
||||
else
|
||||
fail "the extracted shared block is only ${LINECOUNTS[0]} lines — three identical empty spans would compare equal and assert nothing"
|
||||
fail "the extracted shared block is only ${LINECOUNTS[0]} lines — two identical empty spans would compare equal and assert nothing"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1b. The shared Contributing-files parser is byte-identical in both copies
|
||||
# 1a. The resolver copies are the ONLY two, and validate.sh sources its one
|
||||
# ---------------------------------------------------------------------------
|
||||
# Same defect class, one directory over. parse_contributing_files() is embedded
|
||||
# in both validate-provenance.sh copies for the same reason the resolver is
|
||||
# embedded three times, and until this assertion existed the agent-audit copy's
|
||||
# Byte-identity between two named files says nothing about a THIRD copy, and
|
||||
# nothing about whether factory-audit's copy is the one that runs. Assertion 1b
|
||||
# pins both of those for the Contributing-files parser; the resolver is the same
|
||||
# defect class and gets the same two checks:
|
||||
#
|
||||
# a. validate.sh actually SOURCES lib-boundary-resolver.sh, in BOTH mode
|
||||
# branches — asserted inside each arm of `case "$MODE" in`, not by counting
|
||||
# source lines file-wide, because a count cannot see a branch. A library
|
||||
# that is identical, unique and never sourced is a copy that has quietly
|
||||
# been replaced by an inline one — and the byte-identity check above would
|
||||
# stay green over it.
|
||||
# b. Nothing has re-inlined it. The BEGIN marker and a def unique to the
|
||||
# resolver (`_authoring_root`) appear in exactly the two authorities —
|
||||
# scripts/skill-size-check.sh and lib-boundary-resolver.sh — and nowhere
|
||||
# else under the tree. A mode library that grows a "just this once" copy
|
||||
# would otherwise escape assertion 1 entirely, because 1 hashes only the
|
||||
# two files it names.
|
||||
echo ""
|
||||
echo "--- the ADR-0020 resolver has exactly two authorities, and validate.sh sources factory-audit's ---"
|
||||
|
||||
# Deployed and vendored trees are generated copies, not authorities: .claude/ is
|
||||
# apm install output, apm_modules/ is resolved dependencies, build/ is release
|
||||
# artifacts. This file is excluded because it necessarily quotes what it
|
||||
# searches for. Markdown is excluded because an authority is code that runs:
|
||||
# ADR-0025 and gates.md quote these needles to document this very check, and a
|
||||
# prose mention is not a re-inlined copy.
|
||||
SELF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
|
||||
tree_scan() {
|
||||
grep -rlF --binary-files=without-match \
|
||||
--exclude-dir=.git --exclude-dir=build --exclude-dir=.claude \
|
||||
--exclude-dir=apm_modules --exclude-dir=node_modules --exclude='*.md' \
|
||||
-- "$1" "$REPO_ROOT" 2>/dev/null | grep -vFx "$SELF" | sort || true
|
||||
}
|
||||
|
||||
# (a) Sourced, once per mode branch — asserted PER BRANCH, not by counting.
|
||||
#
|
||||
# This used to be a file-wide `grep -c ... >= 2`, and a count cannot see a
|
||||
# branch. Proven by mutation: moving the agent arm's source line out and
|
||||
# duplicating the skill arm's leaves the count at 2 and the old assertion
|
||||
# printed PASS while claiming "in both mode branches" — so a branch that lost
|
||||
# its resolver gracefully (a defaulted variable, a `set +u` region, an optional
|
||||
# resolver) stayed green and lying. The claim is per-branch, so the check is
|
||||
# too: each arm of `case "$MODE" in` must source the resolver inside its own
|
||||
# body. The mutation that fooled the old form is run against the new one below,
|
||||
# because an assertion about branches that has never been shown to fail on a
|
||||
# count-preserving branch edit is exactly the assertion that was here before.
|
||||
RESOLVER_SOURCE_RE='^[[:space:]]*(\.|source)[[:space:]]+.*lib-boundary-resolver\.sh'
|
||||
|
||||
# The body of one arm of the `case "$MODE" in` block: everything between the
|
||||
# arm's label and its `;;`. Structural, not positional — the file is edited
|
||||
# often and line numbers or a whole-file hash would pin the wrong thing.
|
||||
mode_arm_body() {
|
||||
awk -v arm="$2" '
|
||||
!incase && $0 ~ /^[[:space:]]*case[[:space:]]+"?\$MODE"?[[:space:]]+in[[:space:]]*$/ { incase = 1; next }
|
||||
incase && $0 ~ /^[[:space:]]*esac([[:space:]]|$)/ { incase = 0; next }
|
||||
incase && !inarm && $0 ~ "^[[:space:]]*\\(?" arm "\\)[[:space:]]*$" { inarm = 1; next }
|
||||
inarm && $0 ~ /^[[:space:]]*;;[[:space:]]*$/ { inarm = 0; next }
|
||||
inarm { print }
|
||||
' "$1"
|
||||
}
|
||||
|
||||
# Every mode arm sources the resolver exactly once, inside its own body.
|
||||
# Returns 0/1 and leaves the reason in ARM_DETAIL, so the same function can be
|
||||
# run against the real file and against the mutant below.
|
||||
ARM_DETAIL=""
|
||||
check_resolver_per_arm() {
|
||||
local file="$1" arm body n rc=0
|
||||
ARM_DETAIL=""
|
||||
for arm in skill agent; do
|
||||
body="$(mode_arm_body "$file" "$arm")"
|
||||
if [[ -z "$body" ]]; then
|
||||
ARM_DETAIL+="the $arm) arm of the case \"\$MODE\" block was not found or is empty; "
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
n="$(grep -Ec "$RESOLVER_SOURCE_RE" <<< "$body" || true)"
|
||||
if [[ "$n" -eq 0 ]]; then
|
||||
ARM_DETAIL+="the $arm) arm never sources lib-boundary-resolver.sh, so that mode runs some other resolver or none; "
|
||||
rc=1
|
||||
elif [[ "$n" -ne 1 ]]; then
|
||||
ARM_DETAIL+="the $arm) arm sources lib-boundary-resolver.sh $n times; "
|
||||
rc=1
|
||||
fi
|
||||
done
|
||||
return $rc
|
||||
}
|
||||
|
||||
if [[ ! -f "$FACTORY_VALIDATE" ]]; then
|
||||
fail "entry point not found: ${FACTORY_VALIDATE#"$REPO_ROOT/"}"
|
||||
else
|
||||
RESOLVER_SOURCES="$(grep -Ec "$RESOLVER_SOURCE_RE" "$FACTORY_VALIDATE" || true)"
|
||||
if check_resolver_per_arm "$FACTORY_VALIDATE"; then
|
||||
pass "${FACTORY_VALIDATE#"$REPO_ROOT/"}: the skill) and agent) arms of its case \"\$MODE\" block EACH source lib-boundary-resolver.sh inside their own body, exactly once"
|
||||
else
|
||||
fail "${FACTORY_VALIDATE#"$REPO_ROOT/"} does not source lib-boundary-resolver.sh once per mode arm: ${ARM_DETAIL%; }"
|
||||
fi
|
||||
# Secondary, and deliberately not the verdict: with one source per arm proven
|
||||
# above, a file-wide total of exactly 2 says there is no third source line
|
||||
# sitting outside both arms.
|
||||
if [[ "$RESOLVER_SOURCES" -eq 2 ]]; then
|
||||
pass "${FACTORY_VALIDATE#"$REPO_ROOT/"} carries exactly 2 resolver source lines file-wide — the two arm sources and nothing else"
|
||||
else
|
||||
fail "${FACTORY_VALIDATE#"$REPO_ROOT/"} carries $RESOLVER_SOURCES resolver source lines file-wide, expected the 2 that belong to the mode arms"
|
||||
fi
|
||||
|
||||
# Mutation self-test. The mutation is the one the retired count could not
|
||||
# see: the agent arm's source line is removed and the skill arm's duplicated,
|
||||
# so the FILE-WIDE COUNT IS UNCHANGED. Written into a copy; the real file is
|
||||
# never touched.
|
||||
MUT_DIR="$(mktemp -d "$TMPDIR_T/resolver-mutant.XXXXXX")"
|
||||
MUT="$MUT_DIR/validate.sh"
|
||||
cp "$FACTORY_VALIDATE" "$MUT"
|
||||
if python3 - "$MUT" <<'PY'
|
||||
import re
|
||||
import sys
|
||||
|
||||
path = sys.argv[1]
|
||||
with open(path, encoding='utf-8') as fh:
|
||||
lines = fh.read().split('\n')
|
||||
|
||||
case_re = re.compile(r'^\s*case\s+"?\$MODE"?\s+in\s*$')
|
||||
esac_re = re.compile(r'^\s*esac(\s|$)')
|
||||
term_re = re.compile(r'^\s*;;\s*$')
|
||||
src_re = re.compile(r'^\s*(\.|source)\s+.*lib-boundary-resolver\.sh')
|
||||
|
||||
starts = [i for i, l in enumerate(lines) if case_re.match(l)]
|
||||
assert len(starts) == 1, 'expected exactly one `case "$MODE" in`, found %d' % len(starts)
|
||||
ci = starts[0]
|
||||
ends = [i for i in range(ci + 1, len(lines)) if esac_re.match(lines[i])]
|
||||
assert ends, 'the case "$MODE" block has no esac'
|
||||
ei = ends[0]
|
||||
|
||||
|
||||
def arm_sources(name):
|
||||
for i in range(ci + 1, ei):
|
||||
if re.match(r'^\s*\(?%s\)\s*$' % name, lines[i]):
|
||||
for j in range(i + 1, ei):
|
||||
if term_re.match(lines[j]):
|
||||
return [k for k in range(i + 1, j) if src_re.match(lines[k])]
|
||||
raise AssertionError('the %s) arm has no ;;' % name)
|
||||
raise AssertionError('no %s) arm in the case "$MODE" block' % name)
|
||||
|
||||
|
||||
skill = arm_sources('skill')
|
||||
agent = arm_sources('agent')
|
||||
assert len(skill) == 1 and len(agent) == 1, \
|
||||
'expected one resolver source per arm before mutating, got skill=%d agent=%d' % (len(skill), len(agent))
|
||||
|
||||
out = list(lines)
|
||||
del out[agent[0]] # the agent arm loses its resolver ...
|
||||
out.insert(skill[0], lines[skill[0]]) # ... and the skill arm gains a duplicate
|
||||
with open(path, 'w', encoding='utf-8') as fh:
|
||||
fh.write('\n'.join(out))
|
||||
PY
|
||||
then
|
||||
MUT_SOURCES="$(grep -Ec "$RESOLVER_SOURCE_RE" "$MUT" || true)"
|
||||
MUT_AGENT="$(grep -Ec "$RESOLVER_SOURCE_RE" <<< "$(mode_arm_body "$MUT" agent)" || true)"
|
||||
# Guard the fixture before trusting its verdict: the mutation must have
|
||||
# actually emptied the agent arm AND left the file-wide count where it was,
|
||||
# or the case below proves nothing about the defect it stands for.
|
||||
if [[ "$MUT_SOURCES" -eq "$RESOLVER_SOURCES" && "$MUT_AGENT" -eq 0 ]]; then
|
||||
pass "fixture check: the mutant's agent arm sources no resolver while the file-wide count is still $MUT_SOURCES — the retired 'count >= 2' assertion would have passed it"
|
||||
else
|
||||
fail "the resolver mutation did not land as intended (file-wide $MUT_SOURCES vs $RESOLVER_SOURCES, agent arm $MUT_AGENT) — the case below would prove nothing"
|
||||
fi
|
||||
if check_resolver_per_arm "$MUT"; then
|
||||
fail "the per-arm check PASSED a validate.sh whose agent arm has no resolver source — it is still counting, not reading branches"
|
||||
else
|
||||
pass "the per-arm check FAILS the count-preserving mutant (${ARM_DETAIL%; }) — it reads the branches, not a total"
|
||||
fi
|
||||
else
|
||||
fail "could not build the resolver mutation fixture — validate.sh's case \"\$MODE\" structure is not the shape this self-test knows, so the per-arm check is unproven"
|
||||
fi
|
||||
fi
|
||||
|
||||
# (b) Exactly the two authorities, for both spellings of a copy.
|
||||
EXPECTED_RESOLVERS="$(printf '%s\n' "$HOOK" "$FACTORY_RESOLVER" | sort)"
|
||||
check_resolver_authorities() {
|
||||
local label="$1" needle="$2"
|
||||
local found
|
||||
found="$(tree_scan "$needle")"
|
||||
if [[ "$found" == "$EXPECTED_RESOLVERS" ]]; then
|
||||
pass "$label appears in exactly the two resolver authorities and nowhere else"
|
||||
elif [[ -z "$found" ]]; then
|
||||
fail "$label was found in NO file at all — the scan is looking for the wrong text"
|
||||
else
|
||||
fail "$label appears in an unexpected set of files, so the resolver has been re-inlined or lost: $(echo "$found" | tr '\n' ' ')— expected exactly ${HOOK#"$REPO_ROOT/"} and ${FACTORY_RESOLVER#"$REPO_ROOT/"}"
|
||||
fi
|
||||
}
|
||||
check_resolver_authorities "the resolver's BEGIN marker" "$BEGIN_MARKER"
|
||||
check_resolver_authorities "a 'def _authoring_root' definition" "def _authoring_root("
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1b. The Contributing-files parser has exactly ONE authority, and it is sourced
|
||||
# ---------------------------------------------------------------------------
|
||||
# Same defect class, one directory over. parse_contributing_files() used to be
|
||||
# embedded in both validate-provenance.sh copies for the same reason the resolver
|
||||
# is embedded twice, and until this assertion existed the agent-audit copy's
|
||||
# docstring merely CLAIMED it was "kept behaviourally identical to skill-audit's
|
||||
# copy" — an invariant nothing checked, and the two had already drifted into
|
||||
# different spellings of the bullet loop. The parser decides whether checks 4,
|
||||
# copy" — an invariant nothing checked, and the two did drift into different
|
||||
# spellings of the bullet loop at 484357a. That drift happened to be
|
||||
# behaviour-neutral and was re-unified by hand at 598a7c3; the next one need
|
||||
# not be. The parser decides whether checks 4,
|
||||
# 5 and 8 run at all, so a one-sided edit disables a check in one script while
|
||||
# every other test stays green.
|
||||
#
|
||||
# ADR-0025 merged the two skills, so there is now ONE copy and nothing left to
|
||||
# hash against. That does NOT retire the assertion: a byte-identity check over a
|
||||
# single copy is vacuous, and deleting it outright would restore exactly the
|
||||
# condition that allowed the original drift — a parser with no pinned authority.
|
||||
# So the claim is CONVERTED, not dropped. It is the same claim ("the parser has
|
||||
# exactly one authority") stated against the new structure:
|
||||
#
|
||||
# a. factory-audit/scripts/lib-contributing-files.sh exists and carries exactly
|
||||
# one BEGIN/END marker pair, over a span of real content.
|
||||
# b. validate-provenance.sh actually SOURCES it. A library nobody sources is a
|
||||
# copy that has silently been replaced by an inline one somewhere else.
|
||||
# c. Nothing has re-inlined it. No other file in the tree carries the marker
|
||||
# pair, and no other file defines parse_contributing_files. This is the part
|
||||
# that fails if the merge is ever partially reverted, or if a mode library
|
||||
# grows its own "just this once" copy — which is precisely how the drift
|
||||
# this assertion was written for got in.
|
||||
echo ""
|
||||
echo "--- the shared Contributing-files parser is byte-identical in both validate-provenance.sh copies ---"
|
||||
echo "--- the Contributing-files parser has exactly one authority, and validate-provenance.sh sources it ---"
|
||||
|
||||
CF_BEGIN='# ===== BEGIN SHARED CONTRIBUTING-FILES PARSER ====='
|
||||
CF_END='# ===== END SHARED CONTRIBUTING-FILES PARSER ====='
|
||||
SKILL_PROV="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh"
|
||||
AGENT_PROV="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate-provenance.sh"
|
||||
CF_LIB="$FACTORY_AUDIT/scripts/lib-contributing-files.sh"
|
||||
FACTORY_PROV="$FACTORY_AUDIT/scripts/validate-provenance.sh"
|
||||
|
||||
# (a) One library, one well-formed marker pair, over real content.
|
||||
CF_MARKERS_OK=true
|
||||
for f in "$SKILL_PROV" "$AGENT_PROV"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
fail "script not found: $f"
|
||||
CF_MARKERS_OK=false
|
||||
continue
|
||||
fi
|
||||
b="$(grep -cFx "$CF_BEGIN" "$f" || true)"
|
||||
e="$(grep -cFx "$CF_END" "$f" || true)"
|
||||
if [[ "$b" == "1" && "$e" == "1" ]]; then
|
||||
pass "${f#"$REPO_ROOT/"} carries exactly one BEGIN and one END parser marker"
|
||||
else
|
||||
fail "${f#"$REPO_ROOT/"} has $b BEGIN and $e END parser markers, expected 1 and 1"
|
||||
CF_MARKERS_OK=false
|
||||
fi
|
||||
done
|
||||
|
||||
if ! $CF_MARKERS_OK; then
|
||||
fail "skipping the parser byte-identity comparison — the marker pairs are not well-formed, so any extraction would measure the wrong span"
|
||||
if [[ ! -f "$CF_LIB" ]]; then
|
||||
fail "the single parser authority is missing: ${CF_LIB#"$REPO_ROOT/"}"
|
||||
CF_MARKERS_OK=false
|
||||
else
|
||||
CF_HASHES=()
|
||||
CF_LINECOUNTS=()
|
||||
for f in "$SKILL_PROV" "$AGENT_PROV"; do
|
||||
out="$TMPDIR_T/cfblock-$(echo "$f" | md5sum | cut -c1-8).txt"
|
||||
sed -n "/^${CF_BEGIN}\$/,/^${CF_END}\$/p" "$f" > "$out"
|
||||
CF_HASHES+=("$(md5sum < "$out" | cut -d' ' -f1)")
|
||||
CF_LINECOUNTS+=("$(wc -l < "$out" | tr -d ' ')")
|
||||
done
|
||||
if [[ "${CF_HASHES[0]}" == "${CF_HASHES[1]}" ]]; then
|
||||
pass "both copies hash to ${CF_HASHES[0]} (${CF_LINECOUNTS[0]} lines) — agreement by construction, not by coincidence"
|
||||
b="$(grep -cFx "$CF_BEGIN" "$CF_LIB" || true)"
|
||||
e="$(grep -cFx "$CF_END" "$CF_LIB" || true)"
|
||||
if [[ "$b" == "1" && "$e" == "1" ]]; then
|
||||
pass "${CF_LIB#"$REPO_ROOT/"} carries exactly one BEGIN and one END parser marker"
|
||||
else
|
||||
fail "the shared Contributing-files parser has DRIFTED: skill-audit=${CF_HASHES[0]} (${CF_LINECOUNTS[0]} lines), agent-audit=${CF_HASHES[1]} (${CF_LINECOUNTS[1]} lines). Edit one copy, then paste it over the other."
|
||||
fi
|
||||
# Two identical EMPTY spans would hash equal and assert nothing, exactly as
|
||||
# for the resolver above. The parser block is ~93 lines; 40 is a floor low
|
||||
# enough never to need maintenance and high enough that a gutted block — or
|
||||
# one reduced to its docstring — cannot sneak past.
|
||||
if [[ "${CF_LINECOUNTS[0]}" -gt 40 ]]; then
|
||||
pass "the extracted parser block is ${CF_LINECOUNTS[0]} lines — the comparison is over real content, not an empty span"
|
||||
else
|
||||
fail "the extracted parser block is only ${CF_LINECOUNTS[0]} lines — two identical empty spans would compare equal and assert nothing"
|
||||
fail "${CF_LIB#"$REPO_ROOT/"} has $b BEGIN and $e END parser markers, expected 1 and 1"
|
||||
CF_MARKERS_OK=false
|
||||
fi
|
||||
fi
|
||||
|
||||
if ! $CF_MARKERS_OK; then
|
||||
fail "skipping the parser content check — the marker pair is not well-formed, so any extraction would measure the wrong span"
|
||||
else
|
||||
# A span gutted down to its docstring would still satisfy every structural
|
||||
# check above while enforcing nothing, exactly as for the resolver. The parser
|
||||
# block is ~93 lines; 40 is a floor low enough never to need maintenance and
|
||||
# high enough that a gutted block cannot sneak past.
|
||||
CF_LINECOUNT="$(sed -n "/^${CF_BEGIN}\$/,/^${CF_END}\$/p" "$CF_LIB" | wc -l | tr -d ' ')"
|
||||
if [[ "$CF_LINECOUNT" -gt 40 ]]; then
|
||||
pass "the extracted parser block is $CF_LINECOUNT lines — a real parser, not an empty or docstring-only span"
|
||||
else
|
||||
fail "the extracted parser block is only $CF_LINECOUNT lines — a gutted span asserts nothing"
|
||||
fi
|
||||
fi
|
||||
|
||||
# (b) The one entry point sources it. Without this, (a) and (c) are satisfied by
|
||||
# a library that is present, unique and entirely unused.
|
||||
if [[ ! -f "$FACTORY_PROV" ]]; then
|
||||
fail "entry point not found: ${FACTORY_PROV#"$REPO_ROOT/"}"
|
||||
elif grep -Eq '^[[:space:]]*(\.|source)[[:space:]]+.*lib-contributing-files\.sh' "$FACTORY_PROV"; then
|
||||
pass "${FACTORY_PROV#"$REPO_ROOT/"} sources lib-contributing-files.sh — the single copy is the one that actually runs"
|
||||
else
|
||||
fail "${FACTORY_PROV#"$REPO_ROOT/"} never sources lib-contributing-files.sh — the library is dead code and the parser that runs is some other copy"
|
||||
fi
|
||||
|
||||
# (c) Nobody re-inlined it. Both spellings are scanned: the marker pair (a
|
||||
# copy-paste of the block) and a second `def parse_contributing_files` (a
|
||||
# re-implementation that skipped the markers). tree_scan (defined in 1a)
|
||||
# excludes the same generated trees and this file.
|
||||
cf_scan() { tree_scan "$1"; }
|
||||
check_sole_authority() {
|
||||
local label="$1" needle="$2"
|
||||
local found extra
|
||||
found="$(cf_scan "$needle")"
|
||||
extra="$(printf '%s\n' "$found" | grep -vFx "$CF_LIB" | grep -v '^$' || true)"
|
||||
if [[ -z "$found" ]]; then
|
||||
fail "$label was found in NO file at all — the parser authority has vanished, or the scan is looking for the wrong text"
|
||||
elif [[ -n "$extra" ]]; then
|
||||
fail "$label appears outside the single authority, so the parser has been re-inlined: $(echo "$extra" | tr '\n' ' ')— delete the copy and source lib-contributing-files.sh instead"
|
||||
else
|
||||
pass "$label appears only in ${CF_LIB#"$REPO_ROOT/"} — one authority, no re-inlined copies"
|
||||
fi
|
||||
}
|
||||
check_sole_authority "the parser's BEGIN marker" "$CF_BEGIN"
|
||||
check_sole_authority "a 'def parse_contributing_files' definition" "def parse_contributing_files("
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Both interpreter preflights, in all three scripts
|
||||
# 2. Both interpreter preflights, in both scripts and both modes
|
||||
# ---------------------------------------------------------------------------
|
||||
# The two are checked separately on purpose: `python3 -c 'import yaml'` fails
|
||||
# identically whether python3 is missing or PyYAML is, and naming the wrong one
|
||||
# sends the reader to install the wrong thing.
|
||||
#
|
||||
# ADR-0025 merged the two validators into one auto-detecting entry point, but
|
||||
# the preflight did NOT merge with them: validate.sh detects the mode first and
|
||||
# then calls kyberforge_skill_preflight or kyberforge_agent_preflight from the
|
||||
# mode library it sources. There are still two preflights, so both are still
|
||||
# probed — once with a skill target and once with an agent target. Collapsing
|
||||
# these to a single probe would leave one mode's preflight unpinned, and a mode
|
||||
# whose preflight is gone reports a vacuous pass on a machine with no PyYAML.
|
||||
REAL_PYTHON="$(command -v python3)"
|
||||
# Absolute path, deliberately. The no-python3 fixture below replaces PATH
|
||||
# wholesale, so a bare `bash` (or `/usr/bin/env bash`) would be resolved against
|
||||
@@ -275,47 +544,48 @@ probe_preflight() {
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "--- a PATH with no python3 is a hard failure in all three scripts, naming python3 ---"
|
||||
echo "--- a PATH with no python3 is a hard failure in both scripts and both modes, naming python3 ---"
|
||||
probe_preflight "scripts/skill-size-check.sh reports missing python3" \
|
||||
nopython "python3 is required" \
|
||||
"$HOOK" "$SUBJECT_SKILL_DIR/SKILL.md"
|
||||
probe_preflight "skill-audit/scripts/validate.sh reports missing python3" \
|
||||
probe_preflight "factory-audit/scripts/validate.sh (skill mode) reports missing python3" \
|
||||
nopython "python3 is required" \
|
||||
"$SKILL_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||
probe_preflight "agent-audit/scripts/validate.sh reports missing python3" \
|
||||
"$FACTORY_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||
probe_preflight "factory-audit/scripts/validate.sh (agent mode) reports missing python3" \
|
||||
nopython "python3 is required" \
|
||||
"$AGENT_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||
"$FACTORY_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||
|
||||
echo ""
|
||||
echo "--- a python3 that cannot import yaml is a hard failure in all three scripts, naming PyYAML ---"
|
||||
echo "--- a python3 that cannot import yaml is a hard failure in both scripts and both modes, naming PyYAML ---"
|
||||
probe_preflight "scripts/skill-size-check.sh reports missing PyYAML" \
|
||||
noyaml "PyYAML is required" \
|
||||
"$HOOK" "$SUBJECT_SKILL_DIR/SKILL.md"
|
||||
probe_preflight "skill-audit/scripts/validate.sh reports missing PyYAML" \
|
||||
probe_preflight "factory-audit/scripts/validate.sh (skill mode) reports missing PyYAML" \
|
||||
noyaml "PyYAML is required" \
|
||||
"$SKILL_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||
probe_preflight "agent-audit/scripts/validate.sh reports missing PyYAML" \
|
||||
"$FACTORY_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||
probe_preflight "factory-audit/scripts/validate.sh (agent mode) reports missing PyYAML" \
|
||||
noyaml "PyYAML is required" \
|
||||
"$AGENT_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||
"$FACTORY_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||
|
||||
# The control. Without it, "fails when the dependency is missing" is satisfied by
|
||||
# a script that fails unconditionally, and the two cases above would be green on
|
||||
# a gate that never runs at all.
|
||||
echo ""
|
||||
echo "--- control: with both dependencies present the same subjects pass ---"
|
||||
for probe in "$HOOK:$SUBJECT_SKILL_DIR/SKILL.md" \
|
||||
"$SKILL_VALIDATE:$SUBJECT_SKILL_DIR" \
|
||||
"$AGENT_VALIDATE:$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"; do
|
||||
script="${probe%%:*}"
|
||||
arg="${probe#*:}"
|
||||
# The label is carried explicitly because the two validate.sh probes now name the
|
||||
# same script and differ only in the mode its target selects.
|
||||
for probe in "scripts/skill-size-check.sh|$HOOK|$SUBJECT_SKILL_DIR/SKILL.md" \
|
||||
"factory-audit/scripts/validate.sh (skill mode)|$FACTORY_VALIDATE|$SUBJECT_SKILL_DIR" \
|
||||
"factory-audit/scripts/validate.sh (agent mode)|$FACTORY_VALIDATE|$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"; do
|
||||
IFS='|' read -r label script arg <<< "$probe"
|
||||
set +e
|
||||
ctl_out="$(bash "$script" "$arg" 2>&1)"
|
||||
ctl_rc=$?
|
||||
set -e
|
||||
if [[ $ctl_rc -eq 0 ]]; then
|
||||
pass "${script#"$REPO_ROOT/"} exits 0 on a clean subject with python3 and PyYAML available"
|
||||
pass "$label exits 0 on a clean subject with python3 and PyYAML available"
|
||||
else
|
||||
fail "${script#"$REPO_ROOT/"} failed a clean subject (exit $ctl_rc): $ctl_out"
|
||||
fail "$label failed a clean subject (exit $ctl_rc): $ctl_out"
|
||||
fi
|
||||
done
|
||||
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# Differential test: scripts/skill-size-check.sh (the pre-commit hook) and
|
||||
# skill-audit/scripts/validate.sh (the in-skill auditor) must reach the SAME
|
||||
# ADR-0020 verdict on the same file.
|
||||
# factory-audit/scripts/validate.sh in SKILL mode (the in-skill auditor) must
|
||||
# reach the SAME ADR-0020 verdict on the same file. ADR-0025 merged skill-audit
|
||||
# and agent-audit behind one auto-detecting entry point; every fixture here is a
|
||||
# skill directory, so every invocation below runs the skill flow. The agent flow
|
||||
# has no counterpart hook to differ from — there is no agent-file size gate in
|
||||
# .pre-commit-hooks.yaml — so it is out of this suite's scope, not dropped from it.
|
||||
#
|
||||
# Why this exists as a separate suite. tests/test-skill-size-check.sh already
|
||||
# asserts the two agree on their CONSTANTS, and that assertion is necessary but
|
||||
@@ -11,7 +15,7 @@
|
||||
# wording, which value gets measured, and which branch runs first are the others,
|
||||
# and none of them is visible to a constant check.
|
||||
#
|
||||
# The consequence of divergence is specific and bad: skill-audit reports a skill
|
||||
# The consequence of divergence is specific and bad: the auditor reports a skill
|
||||
# ready to ship and the commit hook then rejects it, or worse, the reverse. So the
|
||||
# comparison here is over VERDICTS on files, not over source text.
|
||||
#
|
||||
@@ -38,7 +42,7 @@ set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/validate.sh"
|
||||
|
||||
TMPDIR_T="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||
@@ -338,7 +342,7 @@ def compare(label, skill_dir):
|
||||
if only_hook:
|
||||
problems.append('only the hook reported %s' % (only_hook,))
|
||||
if only_audit:
|
||||
problems.append('only skill-audit reported %s' % (only_audit,))
|
||||
problems.append('only the auditor reported %s' % (only_audit,))
|
||||
|
||||
# Exit codes are compared on the ADR-0020 axis only: an ERROR-tier ADR-0020
|
||||
# finding must make BOTH scripts non-zero, and neither may be turned
|
||||
@@ -349,7 +353,7 @@ def compare(label, skill_dir):
|
||||
if hook_err and hook_rc == 0:
|
||||
problems.append('the hook reported an ADR-0020 ERROR but exited 0')
|
||||
if audit_err and audit_rc == 0:
|
||||
problems.append('skill-audit reported an ADR-0020 FAIL but exited 0')
|
||||
problems.append('the auditor reported an ADR-0020 FAIL but exited 0')
|
||||
# No escape hatch here any more. There used to be one — a
|
||||
# `_non_adr_hook_error()` helper that waved through a non-zero hook exit
|
||||
# explained by MAX_LINES / MAX_WORDS, on the grounds that those two were
|
||||
@@ -441,7 +445,7 @@ for name, token, expected in (('spec-lines-u2028', 'SPEC_LINES', '605'),
|
||||
_, a_out = run(['bash', validate, skill_dir])
|
||||
want = ('ERROR', token, expected)
|
||||
missing = [who for who, v in (('the hook', verdict(h_out)),
|
||||
('skill-audit', verdict(a_out)))
|
||||
('the auditor', verdict(a_out)))
|
||||
if want not in v]
|
||||
if missing:
|
||||
bad('%s: %s did not report %s=%s. The two scripts must count with the '
|
||||
|
||||
@@ -44,8 +44,12 @@ set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||
# ADR-0025 merged the two validators into one auto-detecting entry point. The two
|
||||
# names are kept because the two MODES are what this suite probes, and each mode
|
||||
# still needs its own target shape to reach: collapsing to a single invocation
|
||||
# would leave one flow's checks unexercised.
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/validate.sh"
|
||||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/validate.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
@@ -174,9 +178,11 @@ PY
|
||||
# Builds all three subjects for one fixture kind and echoes nothing; the paths
|
||||
# are fixed by convention so the probes below can find them.
|
||||
#
|
||||
# skill-audit takes a DIRECTORY (SKILL.md inside it, name matching the dir);
|
||||
# agent-audit takes a FILE inside an apm package. The hook takes the SKILL.md
|
||||
# directly, so it and skill-audit share one file.
|
||||
# The auditor's skill mode takes a DIRECTORY (SKILL.md inside it, name matching
|
||||
# the dir); its agent mode takes a FILE inside an apm package. The hook takes the
|
||||
# SKILL.md directly, so it and skill mode share one file. The two shapes are also
|
||||
# what selects the mode — validate.sh detects from the target — so the two probes
|
||||
# below are the only way to reach both flows.
|
||||
build_subjects() {
|
||||
local kind="$1" base="$TMPDIR_T/$1"
|
||||
rm -rf "$base"
|
||||
@@ -195,21 +201,21 @@ EOF
|
||||
# assertion per script would let two of them drift apart while the suite stayed
|
||||
# green; the whole point of the shared resolver block is that they cannot.
|
||||
#
|
||||
# A needle written `@skills:<text>` is asserted for the hook and skill-audit but
|
||||
# NOT for agent-audit. There is exactly one such needle in this file — the body
|
||||
# A needle written `@skills:<text>` is asserted for the hook and for skill mode
|
||||
# but NOT for agent mode. There is exactly one such needle in this file — the body
|
||||
# word ceiling — and the exemption is the ADR, not a workaround: ADR-0020 gives
|
||||
# agents the description gates and deliberately NO body word gate, because an
|
||||
# agent body becomes the system prompt of a fresh context rather than competing
|
||||
# with the caller's live conversation. Demanding a body finding from agent-audit
|
||||
# would be demanding the ADR be contradicted.
|
||||
# with the caller's live conversation. Demanding a body finding from the agent
|
||||
# flow would be demanding the ADR be contradicted.
|
||||
probe_all() {
|
||||
local label="$1" kind="$2"
|
||||
shift 2
|
||||
local base="$TMPDIR_T/$kind"
|
||||
local -a targets=(
|
||||
"hook|$HOOK|$base/skill/my-skill/SKILL.md"
|
||||
"skill-audit|$SKILL_VALIDATE|$base/skill/my-skill"
|
||||
"agent-audit|$AGENT_VALIDATE|$base/agent/.apm/agents/my-agent.agent.md"
|
||||
"validate.sh skill mode|$SKILL_VALIDATE|$base/skill/my-skill"
|
||||
"validate.sh agent mode|$AGENT_VALIDATE|$base/agent/.apm/agents/my-agent.agent.md"
|
||||
)
|
||||
local problems=""
|
||||
for target in "${targets[@]}"; do
|
||||
@@ -230,7 +236,7 @@ probe_all() {
|
||||
# short form's exit status is the test's when it is false, and relying on
|
||||
# the &&-list exemption to keep that from aborting the run is a footgun
|
||||
# one edit away from biting.
|
||||
if [[ "$who" == agent-audit ]]; then
|
||||
if [[ "$who" == "validate.sh agent mode" ]]; then
|
||||
continue
|
||||
fi
|
||||
needle="${needle#@skills:}"
|
||||
@@ -397,8 +403,8 @@ for spec in "yaml-malformed|yes" "desc-list|no" "desc-mapping|no" "desc-bool|no"
|
||||
build_subjects "$kind"
|
||||
for target in \
|
||||
"hook|$HOOK|$TMPDIR_T/$kind/skill/my-skill/SKILL.md" \
|
||||
"skill-audit|$SKILL_VALIDATE|$TMPDIR_T/$kind/skill/my-skill" \
|
||||
"agent-audit|$AGENT_VALIDATE|$TMPDIR_T/$kind/agent/.apm/agents/my-agent.agent.md"
|
||||
"validate.sh skill mode|$SKILL_VALIDATE|$TMPDIR_T/$kind/skill/my-skill" \
|
||||
"validate.sh agent mode|$AGENT_VALIDATE|$TMPDIR_T/$kind/agent/.apm/agents/my-agent.agent.md"
|
||||
do
|
||||
who="${target%%|*}"; rest="${target#*|}"
|
||||
script="${rest%%|*}"; arg="${rest#*|}"
|
||||
@@ -430,9 +436,9 @@ SILENT_OUT="$(bash "$AGENT_VALIDATE" "$TMPDIR_T/desc-no-value/agent/.apm/agents/
|
||||
SILENT_RC=$?
|
||||
set -e
|
||||
if [[ $SILENT_RC -ne 0 && -n "$SILENT_OUT" ]]; then
|
||||
pass "agent-audit reports a valueless description rather than exiting 0 with zero output"
|
||||
pass "validate.sh agent mode reports a valueless description rather than exiting 0 with zero output"
|
||||
else
|
||||
fail "agent-audit exited $SILENT_RC with output '${SILENT_OUT:-<empty>}' — the original defect was exit 0 and total silence on a blocking pre-push gate"
|
||||
fail "validate.sh agent mode exited $SILENT_RC with output '${SILENT_OUT:-<empty>}' — the original defect was exit 0 and total silence on a blocking pre-push gate"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Tests for scripts/check-apm-agents-valid.sh — the gate that runs agent-audit's
|
||||
# validate.sh over the repo's REAL plugin-scope agent files.
|
||||
# Tests for scripts/check-apm-agents-valid.sh — the gate that runs factory-audit's
|
||||
# validate.sh over the repo's REAL plugin-scope agent files. Since ADR-0025 merged
|
||||
# skill-audit and agent-audit, that validate.sh is one auto-detecting entry point;
|
||||
# every target this gate hands it is an agent file, so every run below takes the
|
||||
# agent branch and sources lib-checks-agent.sh.
|
||||
#
|
||||
# Case 1 runs against the real repo. Every other case runs against a synthetic
|
||||
# fixture, for the same reason scripts/check-scope-walkup-sync.sh's tests do: the
|
||||
@@ -26,7 +29,7 @@ fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
# closed by design, which is correct behavior but makes every case here assert
|
||||
# the same missing-dependency message instead of what it is meant to assert.
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "SKIP: python3 is not installed — agent-audit's validate.sh cannot run, so these cases would only re-assert the missing-dependency guard"
|
||||
echo "SKIP: python3 is not installed — factory-audit's validate.sh cannot run, so these cases would only re-assert the missing-dependency guard"
|
||||
exit 77
|
||||
fi
|
||||
|
||||
@@ -40,8 +43,19 @@ trap cleanup EXIT
|
||||
RUN_TMP="$(mktemp -d)"
|
||||
FIXTURES+=("$RUN_TMP")
|
||||
|
||||
# Builds a minimal REPO_ROOT: agent-audit's validator and the field inventory it
|
||||
# reads at load time, plus one plugin carrying a valid agent file. The plugin's
|
||||
# Builds a minimal REPO_ROOT: factory-audit's validator and the field inventory it
|
||||
# reads at load time, plus one plugin carrying a valid agent file. The validator
|
||||
# is no longer self-contained — ADR-0025 replaced the embedded resolver with a
|
||||
# sourced lib-boundary-resolver.sh and moved the agent checks into
|
||||
# lib-checks-agent.sh — so the fixture copies the two libraries the agent branch
|
||||
# sources as well. Copying validate.sh alone would make every case below fail on
|
||||
# a missing source file rather than on what it is meant to assert.
|
||||
#
|
||||
# The inventory is `agent-field-inventory.md`: factory-audit prefixes every
|
||||
# flow-specific reference file with skill-/agent-, and validate.sh reads this one
|
||||
# by that name at load time.
|
||||
#
|
||||
# The plugin's
|
||||
# apm.yml needs a top-level `type:` line — that is the marker validate.sh's
|
||||
# walk-up uses to resolve plugin scope, and without it the fixture would resolve
|
||||
# to project scope and fail looking for a .github/agents counterpart.
|
||||
@@ -54,10 +68,14 @@ FIXTURES+=("$RUN_TMP")
|
||||
make_fixture() {
|
||||
local dir
|
||||
dir="$(cd "$(mktemp -d)" && pwd -P)"
|
||||
local aa="$dir/plugins/kyberforge/.apm/skills/agent-audit"
|
||||
mkdir -p "$aa/scripts" "$aa/references" "$dir/plugins/lint/.apm/agents"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh" "$aa/scripts/"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/references/field-inventory.md" "$aa/references/"
|
||||
local fa="$dir/plugins/kyberforge/.apm/skills/factory-audit"
|
||||
local src="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit"
|
||||
mkdir -p "$fa/scripts" "$fa/references" "$dir/plugins/lint/.apm/agents"
|
||||
cp "$src/scripts/validate.sh" \
|
||||
"$src/scripts/lib-boundary-resolver.sh" \
|
||||
"$src/scripts/lib-checks-agent.sh" \
|
||||
"$fa/scripts/"
|
||||
cp "$src/references/agent-field-inventory.md" "$fa/references/"
|
||||
cat > "$dir/plugins/lint/apm.yml" <<'YAML'
|
||||
name: lint
|
||||
version: 0.0.1
|
||||
@@ -248,7 +266,7 @@ echo ""
|
||||
echo "--- a missing validate.sh fails rather than validating nothing ---"
|
||||
FIX7="$(make_fixture)"
|
||||
FIXTURES+=("$FIX7")
|
||||
rm -f "$FIX7/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||
rm -f "$FIX7/plugins/kyberforge/.apm/skills/factory-audit/scripts/validate.sh"
|
||||
if bash "$SCRIPT" "$FIX7" > "$RUN_TMP/novalidator.out" 2>&1; then
|
||||
fail "a missing validate.sh exited 0 — the gate silently validated nothing"
|
||||
sed 's/^/ /' "$RUN_TMP/novalidator.out"
|
||||
|
||||
@@ -24,24 +24,32 @@ FIXTURES+=("$RUN_TMP")
|
||||
# paths) so the mutation cases below don't depend on — or risk mutating — the
|
||||
# real repo tree. Defined up here rather than beside its first mutation case
|
||||
# because case 2b's stale-.apm/ fixture is built from it too.
|
||||
#
|
||||
# Still FOUR ports, still four scripts. ADR-0025 merged two of them into
|
||||
# factory-audit, which is a change of address, not of count: validate.sh and
|
||||
# validate-provenance.sh are now factory-audit's, and the two authors' scaffold
|
||||
# scripts are untouched. factory-audit's whole scripts/ directory is copied
|
||||
# because those two are entry points now — each sources its resolver and its
|
||||
# mode library at run time, and a fixture holding only the entry point would fail
|
||||
# on a missing source file instead of on the scope-walk-up behaviour under test.
|
||||
make_minimal_repo_root() {
|
||||
local dir
|
||||
dir="$(mktemp -d)"
|
||||
local na="$dir/plugins/kyberforge/.apm/skills/agent-author/scripts"
|
||||
local ns="$dir/plugins/kyberforge/.apm/skills/skill-author/scripts"
|
||||
local aa="$dir/plugins/kyberforge/.apm/skills/agent-audit/scripts"
|
||||
mkdir -p "$na" "$ns" "$aa"
|
||||
local fa="$dir/plugins/kyberforge/.apm/skills/factory-audit"
|
||||
mkdir -p "$na" "$ns" "$fa"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-author/scripts/new-agent.sh" "$na/"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-author/scripts/new-skill.sh" "$ns/"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh" "$aa/"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate-provenance.sh" "$aa/"
|
||||
cp -R "$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts" "$fa/"
|
||||
# agent-author's templates are needed by new-agent.sh at runtime.
|
||||
cp -R "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-author/assets" "$dir/plugins/kyberforge/.apm/skills/agent-author/"
|
||||
cp -R "$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-author/assets" "$dir/plugins/kyberforge/.apm/skills/skill-author/"
|
||||
# validate.sh needs field-inventory.md
|
||||
mkdir -p "$dir/plugins/kyberforge/.apm/skills/agent-audit/references"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/references/field-inventory.md" \
|
||||
"$dir/plugins/kyberforge/.apm/skills/agent-audit/references/"
|
||||
# validate.sh's agent mode reads agent-field-inventory.md at load time —
|
||||
# factory-audit prefixes every flow-specific reference file with skill-/agent-.
|
||||
mkdir -p "$fa/references"
|
||||
cp "$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/references/agent-field-inventory.md" \
|
||||
"$fa/references/"
|
||||
echo "$dir"
|
||||
}
|
||||
|
||||
@@ -124,11 +132,16 @@ fi
|
||||
|
||||
# --- 4. Regression guard: reintroducing the $HOME-collapse bug into
|
||||
# validate.sh's detect_scope must make the check fail.
|
||||
#
|
||||
# The mutation target is lib-checks-agent.sh, not validate.sh: ADR-0025 made
|
||||
# validate.sh a mode-detecting entry point and moved the agent check suite —
|
||||
# detect_scope with it — into the library it sources. The gate under test still
|
||||
# runs validate.sh, so the fault injected here still reaches it.
|
||||
echo ""
|
||||
echo "--- exits 1 when validate.sh's detect_scope collapses back to the \$HOME-walk-up bug ---"
|
||||
FIXTURE_BUG="$(make_minimal_repo_root)"
|
||||
FIXTURES+=("$FIXTURE_BUG")
|
||||
python3 - "$FIXTURE_BUG/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh" <<'PYTHON'
|
||||
python3 - "$FIXTURE_BUG/plugins/kyberforge/.apm/skills/factory-audit/scripts/lib-checks-agent.sh" <<'PYTHON'
|
||||
import re, sys
|
||||
path = sys.argv[1]
|
||||
with open(path) as f:
|
||||
@@ -173,7 +186,9 @@ echo ""
|
||||
echo "--- exits 1 when validate-provenance.sh's find_plugin_root loses its \$HOME boundary check ---"
|
||||
FIXTURE_BUG2="$(make_minimal_repo_root)"
|
||||
FIXTURES+=("$FIXTURE_BUG2")
|
||||
python3 - "$FIXTURE_BUG2/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate-provenance.sh" <<'PYTHON'
|
||||
# Same relocation as case 4: the agent provenance suite, find_plugin_root
|
||||
# included, now lives in the library validate-provenance.sh sources.
|
||||
python3 - "$FIXTURE_BUG2/plugins/kyberforge/.apm/skills/factory-audit/scripts/lib-provenance-agent.sh" <<'PYTHON'
|
||||
import re, sys
|
||||
path = sys.argv[1]
|
||||
with open(path) as f:
|
||||
|
||||
@@ -1,797 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="$REPO_ROOT/scripts/check-vale-style-sync.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
|
||||
# Same `exit 77` (automake convention; run-tests.sh renders it as SKIPPED) guard
|
||||
# tests/test-vale-wrap.sh uses for a missing binary. Without it this suite reported 5 genuine failures on a machine with no
|
||||
# vale, none of which were regressions.
|
||||
#
|
||||
# The suite SKIPPING while the script it tests HARD-FAILS is deliberate, not an
|
||||
# inconsistency. The script is a pre-push gate whose exit 0 is a claim that the
|
||||
# repo was verified, and six of its assertions are vale invocations — it must
|
||||
# never make that claim on a machine where they could not run. This suite makes
|
||||
# no claim about the repo; it claims the script behaves correctly, and most of
|
||||
# its cases (every glob-coverage case, 10/10b/11) cannot be exercised at all
|
||||
# without vale. Reporting those as FAIL would say "a regression landed" when the
|
||||
# truth is "this machine is missing a dev dependency" — noise that competes with
|
||||
# real failures. Note also that the vale-absent behavior is still fully covered
|
||||
# here even so: the masking below constructs that condition deliberately on a
|
||||
# machine that HAS vale, which is the only place it can be asserted against a
|
||||
# known-good baseline.
|
||||
if ! command -v vale &>/dev/null; then
|
||||
echo "SKIP: vale is not installed — the glob-coverage cases cannot run (install it: https://vale.sh/docs/vale-cli/installation/)"
|
||||
exit 77
|
||||
fi
|
||||
|
||||
# One trap over a registry, rather than rebuilding the trap line per fixture:
|
||||
# the guard is there because bash 3.2 treats "${arr[@]}" on an empty array as
|
||||
# unbound under `set -u`.
|
||||
FIXTURES=()
|
||||
cleanup() { [[ ${#FIXTURES[@]} -eq 0 ]] || rm -rf "${FIXTURES[@]}"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
# Helper: make a fixture repo with skill-audit/agent-audit's Vale copies, in sync by default.
|
||||
# The wrapper is a stub — the script only diffs it — but the Vale assets and both
|
||||
# pre-commit manifests are the repo's real ones, because the .vale.ini checks ask
|
||||
# vale to apply those globs for real and cross-check them against the shipped
|
||||
# hooks' `files:` regexes. A synthetic style or manifest would prove nothing, and
|
||||
# copying the real ones keeps agent-audit's intentional KyberforgeCopilot
|
||||
# divergence in the fixture instead of a sanitized stand-in for it.
|
||||
make_fixture() {
|
||||
local dir
|
||||
dir="$(mktemp -d)"
|
||||
local skill_audit="$dir/plugins/kyberforge/.apm/skills/skill-audit"
|
||||
local agent_audit="$dir/plugins/kyberforge/.apm/skills/agent-audit"
|
||||
mkdir -p "$skill_audit/scripts" "$agent_audit/scripts"
|
||||
|
||||
echo '#!/usr/bin/env bash' > "$skill_audit/scripts/vale-wrap.sh"
|
||||
echo 'echo wrap' >> "$skill_audit/scripts/vale-wrap.sh"
|
||||
cp "$skill_audit/scripts/vale-wrap.sh" "$agent_audit/scripts/vale-wrap.sh"
|
||||
|
||||
cp -R "$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/assets" "$skill_audit/"
|
||||
cp -R "$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/assets" "$agent_audit/"
|
||||
cp "$REPO_ROOT/.pre-commit-hooks.yaml" "$REPO_ROOT/.pre-commit-config.yaml" "$dir/"
|
||||
|
||||
echo "$dir"
|
||||
}
|
||||
|
||||
# Helper: rewrite a glob section header in one copy's .vale.ini, leaving every
|
||||
# other line — StylesPath, BasedOnStyles — intact. This is the shape of the
|
||||
# typo the check exists to catch: the hook still matches the file via its
|
||||
# `files:` regex, vale lints nothing, and pre-commit reports `Passed`.
|
||||
break_glob() {
|
||||
local ini="$1" old="$2" new="$3"
|
||||
python3 - "$ini" "$old" "$new" <<'PYTHON'
|
||||
import sys
|
||||
path, old, new = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
with open(path, encoding='utf-8') as fh:
|
||||
content = fh.read()
|
||||
assert old in content, f"{old} not found in {path}"
|
||||
with open(path, 'w', encoding='utf-8') as fh:
|
||||
fh.write(content.replace(old, new))
|
||||
PYTHON
|
||||
}
|
||||
|
||||
# Vale masking, hoisted so the text-only cases below can use it. Case 12 keeps
|
||||
# its own independent construction and its own loud failure if masking breaks —
|
||||
# it is what proves this mechanism works, so it is not refactored onto this.
|
||||
#
|
||||
# Why: a script run with vale on PATH performs six `vale` invocations (one per
|
||||
# probe path), and they are the suite's entire wall clock. The cases that assert
|
||||
# a text-level finding — StylesPath, BasedOnStyles, per-rule overrides — reach
|
||||
# their verdict through `grep` alone and gain nothing from paying for the
|
||||
# probes. Masking vale is not merely cheaper for them, it is STRICTER: with vale
|
||||
# present a dropped StylesPath also breaks the probe, so such a case would still
|
||||
# exit 1 with the assertion under test deleted. Without vale, only the assertion
|
||||
# under test can produce the failure.
|
||||
#
|
||||
# run_check falls back to an unmasked run rather than skipping when masking is
|
||||
# not safely available, so a machine where this cannot work loses speed, never
|
||||
# coverage. The utility probe matters as much as the vale probe: PATH_NO_VALE
|
||||
# deletes a whole PATH entry, and if that entry also carried grep/diff/awk/cat
|
||||
# the script would fail for an unrelated reason and every negative case below
|
||||
# would pass vacuously.
|
||||
VALE_DIR="$(dirname "$(command -v vale 2>/dev/null || echo /nonexistent/vale)")"
|
||||
PATH_NO_VALE="$(printf '%s' "$PATH" | tr ':' '\n' | grep -vxF "$VALE_DIR" | paste -sd: -)"
|
||||
VALE_MASKED=false
|
||||
if ! PATH="$PATH_NO_VALE" bash -c 'command -v vale' >/dev/null 2>&1 \
|
||||
&& PATH="$PATH_NO_VALE" bash -c \
|
||||
'command -v grep && command -v diff && command -v awk && command -v cat' >/dev/null 2>&1; then
|
||||
VALE_MASKED=true
|
||||
fi
|
||||
|
||||
# Runs the check with vale masked off PATH when that is safe. For text-only
|
||||
# assertions ONLY — never for a case whose verdict depends on a glob probe
|
||||
# actually running.
|
||||
#
|
||||
# CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 is required now that the script
|
||||
# treats a missing vale as a FAIL rather than a warning: without the opt-out
|
||||
# every masked run exits 1 unconditionally and every negative case below would
|
||||
# pass vacuously — the precise vacuity this whole round is closing. The opt-out
|
||||
# restores what masking is for here: the text assertion under test becomes the
|
||||
# only thing that can produce a non-zero exit. Case 12 asserts the un-opted-out
|
||||
# masked run really does hard-fail, so this env var cannot quietly become the
|
||||
# only path anyone exercises.
|
||||
run_check_no_vale() {
|
||||
if [[ "$VALE_MASKED" == true ]]; then
|
||||
PATH="$PATH_NO_VALE" CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 bash "$SCRIPT" "$@"
|
||||
else
|
||||
bash "$SCRIPT" "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- 1. Exits 0 when the two copies are in sync ---
|
||||
echo ""
|
||||
echo "--- exits 0 when skill-audit and agent-audit copies are in sync ---"
|
||||
FIXTURE="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE")
|
||||
if bash "$SCRIPT" "$FIXTURE" > /dev/null 2>&1; then
|
||||
pass "exits 0 when copies are in sync"
|
||||
else
|
||||
fail "exited non-zero against in-sync copies"
|
||||
bash "$SCRIPT" "$FIXTURE" 2>&1 | sed 's/^/ /' || true
|
||||
fi
|
||||
|
||||
# --- 2. Exits 1 when vale-wrap.sh differs between the two copies ---
|
||||
echo ""
|
||||
echo "--- exits 1 when vale-wrap.sh differs ---"
|
||||
FIXTURE2="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE2")
|
||||
echo 'echo different' >> "$FIXTURE2/plugins/kyberforge/.apm/skills/skill-audit/scripts/vale-wrap.sh"
|
||||
if bash "$SCRIPT" "$FIXTURE2" > /dev/null 2>&1; then
|
||||
fail "exited 0 when vale-wrap.sh copies differ — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when vale-wrap.sh copies differ"
|
||||
fi
|
||||
|
||||
# --- 3. Exits 1 when a style rule differs between the two copies ---
|
||||
echo ""
|
||||
echo "--- exits 1 when a Kyberforge style rule differs ---"
|
||||
FIXTURE3="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE3")
|
||||
echo ' - divergent token' >> "$FIXTURE3/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/styles/Kyberforge/VagueWording.yml"
|
||||
if bash "$SCRIPT" "$FIXTURE3" > /dev/null 2>&1; then
|
||||
fail "exited 0 when a style rule differs — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when a Kyberforge style rule differs between copies"
|
||||
fi
|
||||
|
||||
# --- 4. Exits 1 when a rule file exists in only one copy ---
|
||||
echo ""
|
||||
echo "--- exits 1 when a rule file is missing from one copy ---"
|
||||
FIXTURE4="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE4")
|
||||
cat > "$FIXTURE4/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/styles/Kyberforge/Extra.yml" <<'EOF'
|
||||
extends: existence
|
||||
message: "Extra: '%s'"
|
||||
level: error
|
||||
tokens:
|
||||
- divergent token
|
||||
EOF
|
||||
if bash "$SCRIPT" "$FIXTURE4" > /dev/null 2>&1; then
|
||||
fail "exited 0 when a rule file exists in only one copy — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when a rule file is missing from one copy"
|
||||
fi
|
||||
|
||||
# --- 5. Exits 0 (no-op) ONLY when there is no kyberforge plugin at all ---
|
||||
# The no-op is scoped to a repo that never installed kyberforge. Case 5c below
|
||||
# is its counterpart and the one that matters: `plugins/kyberforge/` present but
|
||||
# the `.apm/` targets under it absent is drift, not absence.
|
||||
echo ""
|
||||
echo "--- exits 0 when there is no plugins/kyberforge at all (no-op) ---"
|
||||
FIXTURE5="$(mktemp -d)"
|
||||
FIXTURES+=("$FIXTURE5")
|
||||
if [[ -e "$FIXTURE5/plugins/kyberforge" ]]; then
|
||||
fail "fixture 5 unexpectedly has a plugins/kyberforge, so it does not exercise the no-kyberforge no-op"
|
||||
elif bash "$SCRIPT" "$FIXTURE5" > /dev/null 2>&1; then
|
||||
pass "exits 0 as a no-op when the repo has no kyberforge plugin"
|
||||
else
|
||||
fail "exited non-zero when the repo simply has no kyberforge plugin"
|
||||
fi
|
||||
|
||||
# --- 5c. Exits 1, saying so, when plugins/kyberforge exists but its .apm/
|
||||
# targets do not ---
|
||||
# This script hardcodes plugins/kyberforge/.apm/skills/{skill-audit,agent-audit}
|
||||
# and had no floor under them: `mv plugins/kyberforge/.apm plugins/kyberforge/.apm2`
|
||||
# made both directories absent, which fell into the no-op above and exited 0 —
|
||||
# indistinguishable from a verified in-sync result, and swallowed by pre-commit
|
||||
# as `Passed`. A path rewrite is exactly the edit that produces this, and it is
|
||||
# what this PR did to these paths.
|
||||
#
|
||||
# Exit code alone proves little here (the script exits 1 for a dozen reasons), so
|
||||
# assert the MESSAGE: deleting the floor leaves exit 0, but a floor that fired
|
||||
# for the wrong reason would still be a bug this case must catch.
|
||||
echo ""
|
||||
echo "--- exits 1 and says so when plugins/kyberforge exists but .apm/ does not ---"
|
||||
FIXTURE5C="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE5C")
|
||||
mv "$FIXTURE5C/plugins/kyberforge/.apm" "$FIXTURE5C/plugins/kyberforge/.apm2"
|
||||
STALE_OUT=""
|
||||
STALE_RC=0
|
||||
STALE_OUT="$(bash "$SCRIPT" "$FIXTURE5C" 2>&1)" || STALE_RC=$?
|
||||
if [[ $STALE_RC -eq 0 ]]; then
|
||||
fail "exited 0 when plugins/kyberforge exists but its .apm/ targets are gone — expected exit 1"
|
||||
elif ! printf '%s\n' "$STALE_OUT" | grep -q "\.apm/ paths have gone stale"; then
|
||||
fail "failed for the wrong reason on a stale .apm/ path: $(printf '%s' "$STALE_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "exits non-zero and reports a stale .apm/ path when plugins/kyberforge exists without it"
|
||||
fi
|
||||
|
||||
# --- 5d/5d2. Exits 1, saying so, when the probe TABLE itself verifies nothing ---
|
||||
# 5d used to relocate `assets/vale/` in both skills, on the belief that doing so
|
||||
# skipped the whole probe table with FAIL still at 0. It does not. Run against
|
||||
# the PRE-guard script that fixture already exited 1 with three errors: the
|
||||
# `.vale.ini` loop errs on both missing files long before the probe loop, and
|
||||
# PROBES_CHECKED can only reach 0 when both files are gone — which necessarily
|
||||
# means FAIL >= 2. So it never exercised the guard as a cause, only checked that
|
||||
# its message showed up beside unrelated failures.
|
||||
#
|
||||
# The guard is still worth having, but its real triggers live in the probe table,
|
||||
# which is part of the script rather than the fixture — so these two cases mutate
|
||||
# a COPY of the script and run that. Both assert `1 error(s)`, which is what makes
|
||||
# them real: with the guard deleted each mutation exits 0, and with it present the
|
||||
# guard is provably the only thing that failed the run.
|
||||
assert_mutated() {
|
||||
if diff -q "$SCRIPT" "$1" >/dev/null 2>&1; then
|
||||
fail "the script mutation changed nothing — the probe table's shape has moved, so this case would pass vacuously"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "--- exits 1 and says so when every probe row names a directory that does not exist ---"
|
||||
FIXTURE5D="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE5D")
|
||||
SCRATCH5D="$(mktemp -d)"
|
||||
FIXTURES+=("$SCRATCH5D")
|
||||
sed 's/^skill-audit|/skill-auditX|/; s/^agent-audit|/agent-auditX|/' "$SCRIPT" > "$SCRATCH5D/drifted.sh"
|
||||
NOPROBE_OUT=""
|
||||
NOPROBE_RC=0
|
||||
if assert_mutated "$SCRATCH5D/drifted.sh"; then
|
||||
NOPROBE_OUT="$(bash "$SCRATCH5D/drifted.sh" "$FIXTURE5D" 2>&1)" || NOPROBE_RC=$?
|
||||
if [[ $NOPROBE_RC -eq 0 ]]; then
|
||||
fail "a probe table naming no existing skill directory exited 0 — the glob-coverage section checked nothing and reported success"
|
||||
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "no probe path was checked"; then
|
||||
fail "did not report that zero probe paths were checked: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
|
||||
elif ! printf '%s\n' "$NOPROBE_OUT" | grep -q "failed: 1 error(s)"; then
|
||||
fail "drifted probe rows failed for reasons beyond the empty probe table, so this guard is not provably what fired: $(printf '%s' "$NOPROBE_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "a probe table whose rows name no existing skill directory fails with that guard as the sole error"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "--- exits 1 and says so when the probe table is empty ---"
|
||||
FIXTURE5D2="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE5D2")
|
||||
SCRATCH5D2="$(mktemp -d)"
|
||||
FIXTURES+=("$SCRATCH5D2")
|
||||
# The other reachable trigger: the heredoc gutted outright by a bad merge or a
|
||||
# truncated edit. `done <<'EOF_PROBE'` with no rows between the delimiters is
|
||||
# valid bash — the loop body simply never runs.
|
||||
awk '
|
||||
/^done <<.EOF_PROBE.$/ { print; inblk = 1; next }
|
||||
inblk && /^EOF_PROBE$/ { print; inblk = 0; next }
|
||||
inblk { next }
|
||||
{ print }
|
||||
' "$SCRIPT" > "$SCRATCH5D2/gutted.sh"
|
||||
EMPTYTBL_OUT=""
|
||||
EMPTYTBL_RC=0
|
||||
if assert_mutated "$SCRATCH5D2/gutted.sh"; then
|
||||
EMPTYTBL_OUT="$(bash "$SCRATCH5D2/gutted.sh" "$FIXTURE5D2" 2>&1)" || EMPTYTBL_RC=$?
|
||||
if [[ $EMPTYTBL_RC -eq 0 ]]; then
|
||||
fail "an empty probe table exited 0 — the glob-coverage section verified nothing and reported success"
|
||||
elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "no probe path was checked"; then
|
||||
fail "did not report that zero probe paths were checked: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')"
|
||||
elif ! printf '%s\n' "$EMPTYTBL_OUT" | grep -q "failed: 1 error(s)"; then
|
||||
fail "an empty probe table failed for reasons beyond the guard: $(printf '%s' "$EMPTYTBL_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "an emptied probe heredoc fails with that guard as the sole error"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- 5e. Positive: the check does real work against THIS repo ---
|
||||
# Every case above runs against a synthetic fixture, so the whole suite could be
|
||||
# green while the script inspected nothing at all in the repo it is wired into at
|
||||
# pre-push. The summary line carries the counts; assert they are non-zero.
|
||||
#
|
||||
# BOTH counts, not just the probe count. The `.vale.ini` half of that line was a
|
||||
# hardcoded `2` in each branch of the summary — true on any clean run, since a
|
||||
# missing or unreadable file errs out before the summary is reached, but a
|
||||
# constant states what the author expected rather than what the run inspected,
|
||||
# and extracting only the probe count left it asserted by nothing. It is computed
|
||||
# now, so the count is worth reading and worth pinning.
|
||||
echo ""
|
||||
echo "--- reports a non-zero number of inspected targets against this repo ---"
|
||||
REAL_OUT=""
|
||||
REAL_RC=0
|
||||
REAL_OUT="$(bash "$SCRIPT" "$REPO_ROOT" 2>&1)" || REAL_RC=$?
|
||||
REAL_PROBES="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*checked, \([0-9][0-9]*\) glob probe(s).*/\1/p')"
|
||||
REAL_INIS="$(printf '%s\n' "$REAL_OUT" | sed -n 's/.*: \([0-9][0-9]*\) \.vale\.ini file(s) checked.*/\1/p')"
|
||||
if [[ $REAL_RC -ne 0 ]]; then
|
||||
fail "exited non-zero against this repo's real Vale copies"
|
||||
printf '%s\n' "$REAL_OUT" | sed 's/^/ /'
|
||||
elif [[ -z "$REAL_PROBES" || -z "$REAL_INIS" ]]; then
|
||||
fail "a clean run against this repo reported no inspected-target counts, so 'it checked something' is unverifiable: $(printf '%s' "$REAL_OUT" | tr '\n' ' ')"
|
||||
elif [[ "$REAL_PROBES" -lt 1 ]]; then
|
||||
fail "a clean run against this repo verified $REAL_PROBES glob probes — a pass that inspected nothing"
|
||||
elif [[ "$REAL_INIS" -lt 2 ]]; then
|
||||
fail "a clean run against this repo reported $REAL_INIS .vale.ini file(s) checked — both copies' configs must be inspected"
|
||||
else
|
||||
pass "inspects $REAL_INIS .vale.ini file(s) and $REAL_PROBES glob probe(s) against this repo, and exits 0"
|
||||
fi
|
||||
|
||||
# --- 5b. Exits 1 when REPO_ROOT does not exist ---
|
||||
# A nonexistent path used to fall through to the "neither copy present" no-op
|
||||
# (test 5 above) and exit 0 — indistinguishable from a real, verified in-sync
|
||||
# result. That guard is for a repo legitimately missing kyberforge, not a
|
||||
# typo'd or stale path.
|
||||
echo ""
|
||||
echo "--- exits 1 when REPO_ROOT does not exist ---"
|
||||
if bash "$SCRIPT" "/nonexistent/path/$(date +%s)-$$" > /dev/null 2>&1; then
|
||||
fail "exited 0 for a nonexistent REPO_ROOT — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero for a nonexistent REPO_ROOT"
|
||||
fi
|
||||
|
||||
# --- 6. Exits 1 when only one of the two copies is present ---
|
||||
# The no-op guard used `||`, so a single missing copy also exited 0 — a deleted
|
||||
# or renamed copy passed the sync check silently.
|
||||
echo ""
|
||||
echo "--- exits 1 when only one of the two copies is present ---"
|
||||
FIXTURE6="$(make_fixture)"
|
||||
FIXTURE7="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE6" "$FIXTURE7")
|
||||
rm -rf "$FIXTURE6/plugins/kyberforge/.apm/skills/skill-audit"
|
||||
rm -rf "$FIXTURE7/plugins/kyberforge/.apm/skills/agent-audit"
|
||||
if bash "$SCRIPT" "$FIXTURE6" > /dev/null 2>&1; then
|
||||
fail "exited 0 when only agent-audit is present — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when skill-audit's copy is missing but agent-audit's is present"
|
||||
fi
|
||||
if bash "$SCRIPT" "$FIXTURE7" > /dev/null 2>&1; then
|
||||
fail "exited 0 when only skill-audit is present — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when agent-audit's canonical copy is missing but skill-audit's is present"
|
||||
fi
|
||||
|
||||
# --- 7. Exits 1 when a .vale.ini is missing entirely ---
|
||||
# Without it vale falls back to an upward config search and lints the file with
|
||||
# whatever config it happens to find, which is not a failure anyone sees.
|
||||
echo ""
|
||||
echo "--- exits 1 when a .vale.ini is missing ---"
|
||||
FIXTURE8="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE8")
|
||||
rm -f "$FIXTURE8/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini"
|
||||
if bash "$SCRIPT" "$FIXTURE8" > /dev/null 2>&1; then
|
||||
fail "exited 0 when skill-audit's .vale.ini is missing — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when a .vale.ini is missing"
|
||||
fi
|
||||
|
||||
# --- 7b. Exits 1, saying so, when a .vale.ini is present but cannot be read ---
|
||||
# Every assertion in that loop is a grep, and grep exits 2 on a read error: the
|
||||
# two `grep -q` checks then misreport a file whose StylesPath and BasedOnStyles
|
||||
# may be perfectly fine, and the override capture swallows the error into an
|
||||
# empty result that reads as "no findings". So the exit code alone proves
|
||||
# nothing here — the check already exits 1 either way, just with the wrong
|
||||
# reason — and this case asserts the MESSAGE. Deleting the readability guard
|
||||
# leaves the exit code at 1 and the diagnosis wrong, which is exactly the
|
||||
# mutation the assertion below kills.
|
||||
#
|
||||
# The unreadable path is a DIRECTORY, not a mode-000 file, and that is the whole
|
||||
# point of the case: `cat` on a directory fails for every uid, while a mode-000
|
||||
# file is readable by root, which is what this repo's dev environment and its
|
||||
# pre-push hooks run as. A permission-based fixture would pass or fail depending
|
||||
# on the invoking uid; this one does not.
|
||||
echo ""
|
||||
echo "--- exits 1 and says so when a .vale.ini exists but cannot be read ---"
|
||||
FIXTURE8B="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE8B")
|
||||
UNREADABLE_INI="$FIXTURE8B/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini"
|
||||
rm -f "$UNREADABLE_INI"
|
||||
mkdir -p "$UNREADABLE_INI"
|
||||
UNREADABLE_OUT=""
|
||||
UNREADABLE_RC=0
|
||||
UNREADABLE_OUT="$(bash "$SCRIPT" "$FIXTURE8B" 2>&1)" || UNREADABLE_RC=$?
|
||||
if [[ -e "$UNREADABLE_INI" ]] && cat "$UNREADABLE_INI" >/dev/null 2>&1; then
|
||||
fail "the fixture's .vale.ini is still readable, so this case proves nothing about the unreadable branch"
|
||||
elif [[ $UNREADABLE_RC -eq 0 ]]; then
|
||||
fail "exited 0 when skill-audit's .vale.ini could not be read — expected exit 1"
|
||||
elif ! printf '%s\n' "$UNREADABLE_OUT" | grep -q "could not be read"; then
|
||||
fail "failed for the wrong reason on an unreadable .vale.ini — the readability guard did not fire, so the greps misdiagnosed it: $(printf '%s' "$UNREADABLE_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "exits non-zero and reports an unreadable .vale.ini as unreadable, not as missing or malformed"
|
||||
fi
|
||||
|
||||
# --- 8. Exits 1 when the shared StylesPath line is dropped from either copy ---
|
||||
# StylesPath resolves relative to the .vale.ini, which is the only reason the
|
||||
# bundled styles are found from a consuming repo's clone prefix.
|
||||
# Run with vale masked: a dropped StylesPath also stops vale finding the styles,
|
||||
# so with vale on PATH the glob probe fails too and this case would still exit 1
|
||||
# with the StylesPath assertion itself deleted. Masking makes the text assertion
|
||||
# the only thing that can produce the verdict.
|
||||
echo ""
|
||||
echo "--- exits 1 when StylesPath is missing from either .vale.ini ---"
|
||||
FIXTURE9="$(make_fixture)"
|
||||
FIXTURE10="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE9" "$FIXTURE10")
|
||||
break_glob "$FIXTURE9/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini" \
|
||||
'StylesPath = styles' 'StylesPath = elsewhere'
|
||||
break_glob "$FIXTURE10/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini" \
|
||||
'StylesPath = styles' 'StylesPath = elsewhere'
|
||||
if run_check_no_vale "$FIXTURE9" > /dev/null 2>&1; then
|
||||
fail "exited 0 when skill-audit's .vale.ini lost StylesPath — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when skill-audit's .vale.ini lost StylesPath"
|
||||
fi
|
||||
if run_check_no_vale "$FIXTURE10" > /dev/null 2>&1; then
|
||||
fail "exited 0 when agent-audit's .vale.ini lost StylesPath — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when agent-audit's .vale.ini lost StylesPath"
|
||||
fi
|
||||
|
||||
# --- 9. Exits 1 when no section's BasedOnStyles names Kyberforge ---
|
||||
# Every rule the prefilter gates on lives in that style, so a section that keeps
|
||||
# its glob but loses the style lints the file and reports nothing.
|
||||
echo ""
|
||||
echo "--- exits 1 when BasedOnStyles no longer names Kyberforge ---"
|
||||
FIXTURE11="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE11")
|
||||
break_glob "$FIXTURE11/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini" \
|
||||
'BasedOnStyles = Kyberforge' 'BasedOnStyles = KyberforgeCopilot'
|
||||
if run_check_no_vale "$FIXTURE11" > /dev/null 2>&1; then
|
||||
fail "exited 0 when agent-audit's .vale.ini stopped naming Kyberforge — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when a .vale.ini no longer names the Kyberforge style"
|
||||
fi
|
||||
|
||||
# --- 9b. Exits 1 when a per-rule override leaves a rule at anything but error ---
|
||||
# The third way to switch a rule off without touching a style file or a glob.
|
||||
# Per ADR-0013, every rule is `level: error` and every alert is a FAIL -- there
|
||||
# is no ignorable tier. Vale's exit code keys on `error`
|
||||
# alerts alone, so any such override leaves the glob intact, the styles
|
||||
# byte-identical, and the run at `0 errors`, exit 0, `Passed`.
|
||||
#
|
||||
# Asserted as an ALLOWLIST because that is vale 3.15.2's own semantic, verified
|
||||
# by enumerating the value space: only the exact tokens `YES` and `error` keep a
|
||||
# rule blocking. `warning`/`suggestion` downgrade it (alert printed, exit 0 --
|
||||
# invisible, since pre-commit swallows a passing hook's output); EVERY other
|
||||
# value silences it outright, including `false`, `0`, `off`, an empty value,
|
||||
# `garbage`, and lowercase `yes`. That last one is why a blocklist of
|
||||
# `NO|warning|suggestion` was not enough: `= yes` reads as "enabled" to a human
|
||||
# and disables the rule. Case 10's glob probe backstops none of this -- it keys
|
||||
# on one Kyberforge.VagueWording alert, so DescriptionOpener, PaddingPhrase,
|
||||
# SentenceOpenerThereIs and ProactivePhrase can each be retired underneath it,
|
||||
# which is why the cases below deliberately target rules that probe never sees.
|
||||
#
|
||||
# Two cases below are about comment forms, and they are NOT symmetric in vale:
|
||||
# `error # note` (spaced) is stripped by vale and stays live, while `error# note`
|
||||
# (no space) is not stripped and silences the rule. The gate demands a bare
|
||||
# token, so it flags both -- deliberately stricter than vale for the spaced form,
|
||||
# and the only way to catch the no-space form without reimplementing vale's
|
||||
# comment parsing. `Kyberforge.Vague2` covers rule names carrying a digit: such a
|
||||
# rule is genuinely silenced by `= NO`, and an alpha-only name class in the gate
|
||||
# would not even see the line.
|
||||
echo ""
|
||||
echo "--- exits 1 when a .vale.ini overrides a Kyberforge rule to anything but YES/error ---"
|
||||
while IFS= read -r override; do
|
||||
[[ -n "$override" ]] || continue
|
||||
# `<EMPTY>` stands in for a bare `Rule =` with no value at all, which the
|
||||
# heredoc cannot carry as a trailing space without a linter eating it.
|
||||
override="${override/<EMPTY>/}"
|
||||
FIXTURE_OV="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE_OV")
|
||||
echo "$override" >> "$FIXTURE_OV/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini"
|
||||
if run_check_no_vale "$FIXTURE_OV" > /dev/null 2>&1; then
|
||||
fail "exited 0 with '$override' in skill-audit's .vale.ini -- expected exit 1"
|
||||
else
|
||||
pass "exits non-zero on '$override'"
|
||||
fi
|
||||
done <<'EOF_OVERRIDES'
|
||||
Kyberforge.SentenceOpenerThereIs = NO
|
||||
Kyberforge.VagueWording = warning
|
||||
Kyberforge.SentenceOpenerThereIs = suggestion
|
||||
Kyberforge.SentenceOpenerThereIs = false
|
||||
Kyberforge.DescriptionOpener = 0
|
||||
Kyberforge.PaddingPhrase = off
|
||||
Kyberforge.SentenceOpenerThereIs = yes
|
||||
Kyberforge.DescriptionOpener = garbage
|
||||
Kyberforge.PaddingPhrase =<EMPTY>
|
||||
Kyberforge.SentenceOpenerThereIs = NO # keep quiet
|
||||
Kyberforge.DescriptionOpener = error# silenced, vale strips no comment without a space
|
||||
Kyberforge.PaddingPhrase = error; silenced too, same no-space rule for ';'
|
||||
Kyberforge.DescriptionOpener = error # stripped by vale, still rejected: bare token required
|
||||
Kyberforge.Vague2 = NO
|
||||
Kyberforge.Vague_2 = NO
|
||||
Kyberforge.Vague-2 = NO
|
||||
EOF_OVERRIDES
|
||||
# Same in agent-audit's copy: the check runs over both .vale.ini files, and a
|
||||
# rule retired in only the canonical copy is the likelier direction. `= false`
|
||||
# on ProactivePhrase is the sharpest shape -- one word off the original defect,
|
||||
# on a KyberforgeCopilot rule no glob probe covers.
|
||||
FIXTURE_OV_AGENT="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE_OV_AGENT")
|
||||
echo "KyberforgeCopilot.ProactivePhrase = false" \
|
||||
>> "$FIXTURE_OV_AGENT/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini"
|
||||
if run_check_no_vale "$FIXTURE_OV_AGENT" > /dev/null 2>&1; then
|
||||
fail "exited 0 with 'KyberforgeCopilot.ProactivePhrase = false' in agent-audit's .vale.ini -- expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when agent-audit's copy retires a KyberforgeCopilot rule"
|
||||
fi
|
||||
# The two allowlisted values must NOT trip the assertion -- otherwise it would
|
||||
# fire on any legitimate explicit enablement. Kept as a positive case so an
|
||||
# over-broad tightening of the regex shows up here rather than in the repo.
|
||||
FIXTURE_OV_OK="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE_OV_OK")
|
||||
{
|
||||
echo "Kyberforge.SentenceOpenerThereIs = YES"
|
||||
echo "Kyberforge.VagueWording = error"
|
||||
} >> "$FIXTURE_OV_OK/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini"
|
||||
if bash "$SCRIPT" "$FIXTURE_OV_OK" > /dev/null 2>&1; then
|
||||
pass "an explicit '= YES' / '= error' override is not flagged"
|
||||
else
|
||||
fail "flagged an explicit '= YES' / '= error' override -- those are the two values that keep a rule blocking"
|
||||
bash "$SCRIPT" "$FIXTURE_OV_OK" 2>&1 | sed 's/^/ /' || true
|
||||
fi
|
||||
|
||||
# --- 9c. Exits 1 when agent-audit ships KyberforgeCopilot but never loads it ---
|
||||
# Case 9 asserts only that Kyberforge is named, because skill-audit's copy
|
||||
# legitimately has no Copilot style. So dropping just `, KyberforgeCopilot` from
|
||||
# agent-audit's [**/*.agent.md] section unloaded the whole style silently: no
|
||||
# glob broke, the styles/ diff stayed clean (the directory is still shipped,
|
||||
# only never loaded), the two .vale.ini files are deliberately unequal so no
|
||||
# equality check applies, and case 10's probe still passed because it keys on a
|
||||
# Kyberforge alert. Verified dead by probing a `.agent.md` carrying
|
||||
# "Use proactively": 0 alerts under the broken config, KyberforgeCopilot.
|
||||
# ProactivePhrase under the shipped one. ADR-0013 scopes the style to
|
||||
# `.agent.md` files only, for the Copilot-only 'Use proactively has no effect'
|
||||
# check, so shipping it unloaded is drift.
|
||||
echo ""
|
||||
echo "--- exits 1 when the shipped KyberforgeCopilot style is named by no BasedOnStyles ---"
|
||||
FIXTURE11C="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE11C")
|
||||
break_glob "$FIXTURE11C/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini" \
|
||||
'BasedOnStyles = Kyberforge, KyberforgeCopilot' 'BasedOnStyles = Kyberforge'
|
||||
if run_check_no_vale "$FIXTURE11C" > /dev/null 2>&1; then
|
||||
fail "exited 0 when KyberforgeCopilot was dropped from BasedOnStyles -- expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when a shipped KyberforgeCopilot style is never loaded"
|
||||
fi
|
||||
# The assertion is conditional on the style being shipped: a copy with no
|
||||
# KyberforgeCopilot directory (skill-audit's, by design) must stay clean --
|
||||
# case 13 below covers the shipped-and-loaded pairing.
|
||||
|
||||
# --- 10. Exits 1 when a glob section stops matching the shape its hook lints ---
|
||||
# One case per glob section, because each covers a file shape the others don't:
|
||||
# agent-audit's [**/*.agent.md] is the only section covering a Copilot agent file
|
||||
# outside an agents/ directory, so breaking it alone is invisible to the others.
|
||||
echo ""
|
||||
echo "--- exits 1 when a .vale.ini glob no longer matches its hook's file shape ---"
|
||||
FIXTURE12="$(make_fixture)"
|
||||
FIXTURE13="$(make_fixture)"
|
||||
FIXTURE14="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE12" "$FIXTURE13" "$FIXTURE14")
|
||||
break_glob "$FIXTURE12/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini" \
|
||||
'[**/SKILL.md]' '[**/NOMATCH.md]'
|
||||
break_glob "$FIXTURE13/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini" \
|
||||
'[**/agents/*.md]' '[**/NOMATCH-agents/*.md]'
|
||||
break_glob "$FIXTURE14/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini" \
|
||||
'[**/*.agent.md]' '[**/*.NOMATCH.md]'
|
||||
if bash "$SCRIPT" "$FIXTURE12" > /dev/null 2>&1; then
|
||||
fail "exited 0 when skill-audit's SKILL.md glob matched nothing — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when skill-audit's SKILL.md glob matches nothing"
|
||||
fi
|
||||
if bash "$SCRIPT" "$FIXTURE13" > /dev/null 2>&1; then
|
||||
fail "exited 0 when agent-audit's agents/*.md glob matched nothing — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when agent-audit's agents/*.md glob matches nothing"
|
||||
fi
|
||||
if bash "$SCRIPT" "$FIXTURE14" > /dev/null 2>&1; then
|
||||
fail "exited 0 when agent-audit's *.agent.md glob matched nothing — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when agent-audit's *.agent.md glob matches nothing"
|
||||
fi
|
||||
|
||||
# --- 10b. Exits 1 when a glob is narrowed to this repo's own plugins/ layout ---
|
||||
# Every probe path used to start with `plugins/`, so a glob narrowed from a
|
||||
# filename shape to a location (`[**/SKILL.md]` -> `[**/.apm/skills/*/SKILL.md]`)
|
||||
# still matched all of them and the check passed -- while a project-scope
|
||||
# `.claude/skills/foo/SKILL.md` started linting as `0 errors ... in 0 files`,
|
||||
# exit 0, hook `Passed`: the exact failure the script's own header comment says
|
||||
# it exists to catch. A `SKILL.md` outside `plugins/` (e.g. project-scope
|
||||
# `.claude/skills/foo/SKILL.md`) still matches `[**/SKILL.md]` and gets linted
|
||||
# normally -- the globs constrain filename shape, not location.
|
||||
# These narrowings are still valid glob syntax and break no `plugins/`-shaped
|
||||
# file, so only a non-`plugins/` probe path catches them.
|
||||
echo ""
|
||||
echo "--- exits 1 when a .vale.ini glob is narrowed from a filename shape to a location ---"
|
||||
FIXTURE14B="$(make_fixture)"
|
||||
FIXTURE14C="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE14B" "$FIXTURE14C")
|
||||
break_glob "$FIXTURE14B/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini" \
|
||||
'[**/SKILL.md]' '[**/.apm/skills/*/SKILL.md]'
|
||||
break_glob "$FIXTURE14C/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini" \
|
||||
'[**/agents/*.md]' '[**/.apm/agents/*.md]'
|
||||
if bash "$SCRIPT" "$FIXTURE14B" > /dev/null 2>&1; then
|
||||
fail "exited 0 when skill-audit's glob stopped covering a SKILL.md outside plugins/ -- expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when skill-audit's glob stops covering a project-scope SKILL.md"
|
||||
fi
|
||||
if bash "$SCRIPT" "$FIXTURE14C" > /dev/null 2>&1; then
|
||||
fail "exited 0 when agent-audit's glob stopped covering an agents/*.md outside plugins/ -- expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when agent-audit's glob stops covering a project-scope agents/*.md"
|
||||
fi
|
||||
|
||||
# --- 11. Exits 1 when a probe path falls out of every hook's `files:` regex ---
|
||||
# The probe paths are hardcoded, so they can silently stop representing anything
|
||||
# the hooks lint. Rescoping the shipped agent hook away from the `.agent.md`
|
||||
# shape has to fail here rather than leave a probe testing a shape no hook
|
||||
# matches any more.
|
||||
echo ""
|
||||
echo "--- exits 1 when a probe path matches no hook's files: regex ---"
|
||||
FIXTURE16="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE16")
|
||||
break_glob "$FIXTURE16/.pre-commit-hooks.yaml" \
|
||||
"files: '(^|/)agents/[^/]+\\.md\$|\\.agent\\.md\$'" "files: '(^|/)agents/[^/]+\\.md\$'"
|
||||
if bash "$SCRIPT" "$FIXTURE16" > /dev/null 2>&1; then
|
||||
fail "exited 0 when the agent hook was rescoped away from .agent.md — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when a probe path is in no hook's scope any more"
|
||||
fi
|
||||
|
||||
# --- 11b. Exits 1 when the local config's files: regex narrows out of sync
|
||||
# with the canonical .pre-commit-hooks.yaml regex ---
|
||||
# hook_file_regexes() used to union the two manifests' `files:` regexes before
|
||||
# checking probe coverage, so a probe that matched only the old, looser
|
||||
# .pre-commit-hooks.yaml pattern still passed as "in scope" even after
|
||||
# .pre-commit-config.yaml's copy of the same hook was narrowed away from it.
|
||||
# That is exactly the shape of rescoping this repo's own agent hook went
|
||||
# through (SKILL/agent `.md` -> `.apm/.../*.agent.md`): the local hook quietly
|
||||
# stopped linting a shape the shipped, external-facing manifest still claims
|
||||
# to cover, and nothing caught it. Reproduce it directly: narrow only the
|
||||
# fixture's local config regex (leave .pre-commit-hooks.yaml as shipped) and
|
||||
# assert the check now flags the disagreement instead of passing silently.
|
||||
echo ""
|
||||
echo "--- exits 1 when .pre-commit-config.yaml's files: regex drifts out of sync with .pre-commit-hooks.yaml's ---"
|
||||
FIXTURE16B="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE16B")
|
||||
break_glob "$FIXTURE16B/.pre-commit-config.yaml" \
|
||||
"files: '^plugins/[^/]+/\\.apm/agents/[^/]+\\.agent\\.md\$'" \
|
||||
"files: '^plugins/kyberforge/\\.apm/agents/[^/]+\\.agent\\.md\$'"
|
||||
if bash "$SCRIPT" "$FIXTURE16B" > /dev/null 2>&1; then
|
||||
fail "exited 0 when the local config regex narrowed out of sync with .pre-commit-hooks.yaml — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero when the local config regex narrows out of sync with the canonical .pre-commit-hooks.yaml regex"
|
||||
fi
|
||||
|
||||
# --- 12. The text-level assertions hold on a machine without vale ---
|
||||
# They are the fallback when the glob probe cannot run. With vale on PATH the
|
||||
# probe fails on these same mutations, so it would mask them: only masking vale
|
||||
# proves a clean run here means the text assertions themselves ran.
|
||||
echo ""
|
||||
echo "--- the StylesPath / BasedOnStyles assertions still gate with vale masked off PATH ---"
|
||||
VALE_DIR="$(dirname "$(command -v vale 2>/dev/null || echo /nonexistent/vale)")"
|
||||
PATH_NO_VALE="$(printf '%s' "$PATH" | tr ':' '\n' | grep -vxF "$VALE_DIR" | paste -sd: -)"
|
||||
if (PATH="$PATH_NO_VALE"; command -v vale >/dev/null 2>&1); then
|
||||
fail "could not mask vale off PATH — the vale-absent fallback was not exercised"
|
||||
else
|
||||
FIXTURE17="$(make_fixture)"
|
||||
FIXTURE18="$(make_fixture)"
|
||||
FIXTURE19="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE17" "$FIXTURE18" "$FIXTURE19")
|
||||
break_glob "$FIXTURE18/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini" \
|
||||
'StylesPath = styles' 'StylesPath = elsewhere'
|
||||
break_glob "$FIXTURE19/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini" \
|
||||
'BasedOnStyles = Kyberforge' 'BasedOnStyles = KyberforgeCopilot'
|
||||
# 12a. Missing vale is a HARD FAILURE, not a warning — even on copies that are
|
||||
# otherwise perfectly in sync. It used to be a warning, and a warning made the
|
||||
# six glob probes self-disable on the machine that most needed them: applying
|
||||
# the one-character typo `[**/SKILL.md]` -> `[**/SKILLS.md]` and running with
|
||||
# vale off PATH exited 0, its sole output a stderr line pre-commit swallows,
|
||||
# so the pre-push hook reported `Passed`. That is the exact defect the
|
||||
# glob-coverage section exists to catch, disabled by the absence of the tool
|
||||
# that catches it. Assert the MESSAGE: exit 1 has a dozen causes here and the
|
||||
# fixture is in sync, so the code alone would not distinguish this from any
|
||||
# other finding.
|
||||
NOVALE_OUT=""
|
||||
NOVALE_RC=0
|
||||
NOVALE_OUT="$(PATH="$PATH_NO_VALE" bash "$SCRIPT" "$FIXTURE17" 2>&1)" || NOVALE_RC=$?
|
||||
if [[ $NOVALE_RC -eq 0 ]]; then
|
||||
fail "exited 0 on in-sync copies with vale unavailable — a run that could not verify glob coverage must not report success"
|
||||
elif ! printf '%s\n' "$NOVALE_OUT" | grep -q "vale is not installed, so none of the .vale.ini glob-coverage probes ran"; then
|
||||
fail "failed without vale for the wrong reason — the missing-binary guard did not fire: $(printf '%s' "$NOVALE_OUT" | tr '\n' ' ')"
|
||||
else
|
||||
pass "hard-fails, saying so, when vale is unavailable and no opt-out is set"
|
||||
fi
|
||||
|
||||
# 12b. The opt-out is the only way to get a clean exit without vale, and it has
|
||||
# to be set deliberately. Absence of the binary must never imply it.
|
||||
if PATH="$PATH_NO_VALE" CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 \
|
||||
bash "$SCRIPT" "$FIXTURE17" > /dev/null 2>&1; then
|
||||
pass "exits 0 on in-sync copies with vale unavailable and the explicit opt-out set"
|
||||
else
|
||||
fail "exited non-zero on in-sync copies with vale unavailable and CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 — the opt-out does not work"
|
||||
fi
|
||||
|
||||
# 12c/12d. The text assertions still gate under the opt-out. This is what the
|
||||
# opt-out has to preserve: masking vale makes the assertion under test the only
|
||||
# thing that can produce the verdict (with vale present, a dropped StylesPath
|
||||
# also breaks the probe, so these cases would still exit 1 with the assertion
|
||||
# itself deleted).
|
||||
if PATH="$PATH_NO_VALE" CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 \
|
||||
bash "$SCRIPT" "$FIXTURE18" > /dev/null 2>&1; then
|
||||
fail "exited 0 on a dropped StylesPath with vale unavailable — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero on a dropped StylesPath with vale unavailable"
|
||||
fi
|
||||
if PATH="$PATH_NO_VALE" CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 \
|
||||
bash "$SCRIPT" "$FIXTURE19" > /dev/null 2>&1; then
|
||||
fail "exited 0 on a BasedOnStyles that dropped Kyberforge with vale unavailable — expected exit 1"
|
||||
else
|
||||
pass "exits non-zero on a BasedOnStyles that dropped Kyberforge with vale unavailable"
|
||||
fi
|
||||
|
||||
# 12e. An opted-out clean run must still say it verified nothing — otherwise
|
||||
# the opt-out just reintroduces the silent vacuous pass under a new name.
|
||||
OPTOUT_OUT="$(PATH="$PATH_NO_VALE" CHECK_VALE_STYLE_SYNC_ALLOW_MISSING_VALE=1 \
|
||||
bash "$SCRIPT" "$FIXTURE17" 2>&1)"
|
||||
if printf '%s\n' "$OPTOUT_OUT" | grep -q "glob coverage was NOT verified" \
|
||||
&& printf '%s\n' "$OPTOUT_OUT" | grep -q "0 glob probe(s) verified"; then
|
||||
pass "an opted-out clean run reports that glob coverage was not verified"
|
||||
else
|
||||
fail "an opted-out clean run did not say it verified no glob coverage — it looks identical to a verified one: $(printf '%s' "$OPTOUT_OUT" | tr '\n' ' ')"
|
||||
fi
|
||||
|
||||
# 12f. The typo the whole section exists to catch must fail with vale absent
|
||||
# and the opt-out set, or not at all — never pass. It cannot be caught without
|
||||
# vale, so the opt-out must not turn it into a green run by accident: with the
|
||||
# opt-out this fixture legitimately passes, which is precisely why the opt-out
|
||||
# is gated on an env var and 12a is the default.
|
||||
FIXTURE19B="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE19B")
|
||||
break_glob "$FIXTURE19B/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini" \
|
||||
'[**/SKILL.md]' '[**/SKILLS.md]'
|
||||
if PATH="$PATH_NO_VALE" bash "$SCRIPT" "$FIXTURE19B" > /dev/null 2>&1; then
|
||||
fail "the one-character glob typo exited 0 with vale off PATH — the probe self-disabled on the exact defect it exists to catch"
|
||||
else
|
||||
pass "the one-character glob typo does not exit 0 with vale off PATH"
|
||||
fi
|
||||
fi
|
||||
|
||||
# --- 13. The intentional agent-audit-only divergence is NOT flagged ---
|
||||
# The two .vale.ini files are deliberately different: agent-audit ships an extra
|
||||
# [**/*.agent.md] section and the KyberforgeCopilot style. A check that diffed
|
||||
# them would fail the repo as it stands, so assert the divergence is really in
|
||||
# the fixture before asserting the check tolerates it — otherwise this case would
|
||||
# still pass if the fixture had quietly stopped carrying it.
|
||||
echo ""
|
||||
echo "--- exits 0 despite agent-audit's KyberforgeCopilot divergence ---"
|
||||
FIXTURE15="$(make_fixture)"
|
||||
FIXTURES+=("$FIXTURE15")
|
||||
AGENT_INI15="$FIXTURE15/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/.vale.ini"
|
||||
SKILL_INI15="$FIXTURE15/plugins/kyberforge/.apm/skills/skill-audit/assets/vale/.vale.ini"
|
||||
if ! grep -q "KyberforgeCopilot" "$AGENT_INI15" \
|
||||
|| grep -q "KyberforgeCopilot" "$SKILL_INI15" \
|
||||
|| [[ ! -d "$FIXTURE15/plugins/kyberforge/.apm/skills/agent-audit/assets/vale/styles/KyberforgeCopilot" ]]; then
|
||||
fail "the fixture no longer carries the agent-audit-only KyberforgeCopilot divergence, so tolerating it proves nothing"
|
||||
elif bash "$SCRIPT" "$FIXTURE15" > /dev/null 2>&1; then
|
||||
pass "exits 0 with agent-audit's extra KyberforgeCopilot section and style present"
|
||||
else
|
||||
fail "flagged the intentional agent-audit-only KyberforgeCopilot divergence — expected exit 0"
|
||||
bash "$SCRIPT" "$FIXTURE15" 2>&1 | sed 's/^/ /' || true
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
@@ -10,15 +10,23 @@
|
||||
# routing targets.
|
||||
#
|
||||
# The constant-agreement block below is the load-bearing part: all three copies
|
||||
# (this hook, skill-audit's validate.sh, agent-audit's validate.sh) are
|
||||
# (this hook, the auditor's skill flow, the auditor's agent flow) are
|
||||
# hand-duplicated because a cache-installed plugin cannot read outside its own
|
||||
# directory, and nothing but these assertions stops them drifting.
|
||||
#
|
||||
# ADR-0025 merged skill-audit and agent-audit into factory-audit, which moved two
|
||||
# of those copies but did not reduce them to one: the constants live in the two
|
||||
# mode libraries validate.sh sources, and the two libraries still declare them
|
||||
# separately. So the comparisons below read lib-checks-skill.sh and
|
||||
# lib-checks-agent.sh directly rather than the entry point, which declares none
|
||||
# of them — grepping validate.sh would find nothing and report every constant as
|
||||
# <unset>, or worse, silently agree that two empty values match.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||
VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/lib-checks-skill.sh"
|
||||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/factory-audit/scripts/lib-checks-agent.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
@@ -81,26 +89,26 @@ fi
|
||||
MAX_WORDS="$(grep -oE '^MAX_WORDS=[0-9]+' "$SCRIPT" | cut -d= -f2)"
|
||||
MAX_LINES="$(grep -oE '^MAX_LINES=[0-9]+' "$SCRIPT" | cut -d= -f2)"
|
||||
|
||||
# The audit (skill-audit/scripts/validate.sh) duplicates both ceilings, because
|
||||
# The audit (factory-audit's lib-checks-skill.sh) duplicates both ceilings, because
|
||||
# a cache-installed plugin's scripts cannot read files outside the plugin
|
||||
# directory. Nothing but this assertion stops the copies drifting, and drift
|
||||
# means a SKILL.md passes its own audit and is then rejected by the commit hook.
|
||||
echo ""
|
||||
echo "--- the hook and skill-audit's validate.sh agree on both ceilings ---"
|
||||
echo "--- the hook and factory-audit's skill checks agree on both ceilings ---"
|
||||
if [[ ! -f "$VALIDATE" ]]; then
|
||||
fail "skill-audit validate.sh not found at $VALIDATE"
|
||||
fail "factory-audit lib-checks-skill.sh not found at $VALIDATE"
|
||||
else
|
||||
V_MAX_WORDS="$(grep -oE '^MAX_WORDS = [0-9]+' "$VALIDATE" | grep -oE '[0-9]+')"
|
||||
V_MAX_LINES="$(grep -oE '^MAX_LINES = [0-9]+' "$VALIDATE" | grep -oE '[0-9]+')"
|
||||
if [[ "$V_MAX_WORDS" == "$MAX_WORDS" ]]; then
|
||||
pass "both enforce MAX_WORDS=$MAX_WORDS"
|
||||
else
|
||||
fail "MAX_WORDS drift: hook says $MAX_WORDS, validate.sh says ${V_MAX_WORDS:-<unset>}"
|
||||
fail "MAX_WORDS drift: hook says $MAX_WORDS, lib-checks-skill.sh says ${V_MAX_WORDS:-<unset>}"
|
||||
fi
|
||||
if [[ "$V_MAX_LINES" == "$MAX_LINES" ]]; then
|
||||
pass "both enforce MAX_LINES=$MAX_LINES"
|
||||
else
|
||||
fail "MAX_LINES drift: hook says $MAX_LINES, validate.sh says ${V_MAX_LINES:-<unset>}"
|
||||
fail "MAX_LINES drift: hook says $MAX_LINES, lib-checks-skill.sh says ${V_MAX_LINES:-<unset>}"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -108,11 +116,11 @@ fi
|
||||
# ADR-0020 constants
|
||||
# ---------------------------------------------------------------------------
|
||||
# Three hand-maintained copies, for the same cache-isolation reason as
|
||||
# MAX_WORDS/MAX_LINES above. skill-audit carries all four; agent-audit carries
|
||||
# MAX_WORDS/MAX_LINES above. The skill flow carries all four; the agent flow carries
|
||||
# only the two description constants, because ADR-0020 deliberately gives
|
||||
# agents NO body word gate (a skill body competes with the caller's live
|
||||
# conversation; an agent body becomes the system prompt of a fresh context).
|
||||
# The absence of BODY_* in agent-audit is asserted below so a well-meaning
|
||||
# The absence of BODY_* in the agent flow is asserted below so a well-meaning
|
||||
# "consistency" edit that adds them fails here rather than contradicting the
|
||||
# ADR silently.
|
||||
DESC_SUGGEST_CHARS="$(grep -oE '^DESC_SUGGEST_CHARS=[0-9]+' "$SCRIPT" | cut -d= -f2)"
|
||||
@@ -132,21 +140,21 @@ for pair in "DESC_SUGGEST_CHARS:$DESC_SUGGEST_CHARS" "DESC_MAX_CHARS:$DESC_MAX_C
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- the hook and skill-audit's validate.sh agree on all four ADR-0020 constants ---"
|
||||
echo "--- the hook and factory-audit's skill checks agree on all four ADR-0020 constants ---"
|
||||
for const in DESC_SUGGEST_CHARS DESC_MAX_CHARS BODY_SUGGEST_WORDS BODY_MAX_WORDS; do
|
||||
hook_value="$(grep -oE "^${const}=[0-9]+" "$SCRIPT" | cut -d= -f2)"
|
||||
audit_value="$(grep -oE "^${const} = [0-9]+" "$VALIDATE" | grep -oE '[0-9]+' || true)"
|
||||
if [[ -n "$hook_value" && "$hook_value" == "$audit_value" ]]; then
|
||||
pass "both enforce $const=$hook_value"
|
||||
else
|
||||
fail "$const drift: hook says ${hook_value:-<unset>}, skill-audit validate.sh says ${audit_value:-<unset>}"
|
||||
fail "$const drift: hook says ${hook_value:-<unset>}, lib-checks-skill.sh says ${audit_value:-<unset>}"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "--- the hook and agent-audit's validate.sh agree on the description constants ---"
|
||||
echo "--- the hook and factory-audit's agent checks agree on the description constants ---"
|
||||
if [[ ! -f "$AGENT_VALIDATE" ]]; then
|
||||
fail "agent-audit validate.sh not found at $AGENT_VALIDATE"
|
||||
fail "factory-audit lib-checks-agent.sh not found at $AGENT_VALIDATE"
|
||||
else
|
||||
for const in DESC_SUGGEST_CHARS DESC_MAX_CHARS; do
|
||||
hook_value="$(grep -oE "^${const}=[0-9]+" "$SCRIPT" | cut -d= -f2)"
|
||||
@@ -154,15 +162,15 @@ else
|
||||
if [[ -n "$hook_value" && "$hook_value" == "$agent_value" ]]; then
|
||||
pass "both enforce $const=$hook_value"
|
||||
else
|
||||
fail "$const drift: hook says ${hook_value:-<unset>}, agent-audit validate.sh says ${agent_value:-<unset>}"
|
||||
fail "$const drift: hook says ${hook_value:-<unset>}, lib-checks-agent.sh says ${agent_value:-<unset>}"
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
echo "--- agent-audit declares NO body word gate (ADR-0020 is explicit about this) ---"
|
||||
echo "--- the agent flow declares NO body word gate (ADR-0020 is explicit about this) ---"
|
||||
if grep -qE '^BODY_(SUGGEST|MAX)_WORDS = ' "$AGENT_VALIDATE"; then
|
||||
fail "agent-audit validate.sh declares a body word gate — ADR-0020 gives agents the description gates and NO body word gate"
|
||||
fail "lib-checks-agent.sh declares a body word gate — ADR-0020 gives agents the description gates and NO body word gate"
|
||||
else
|
||||
pass "agent-audit validate.sh declares no BODY_*_WORDS constant"
|
||||
pass "lib-checks-agent.sh declares no BODY_*_WORDS constant"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -188,7 +196,7 @@ make_line_fixture() {
|
||||
|
||||
# The line ceiling is inclusive of the limit itself, enforced via `>` — so
|
||||
# exactly $MAX_LINES must pass and $((MAX_LINES + 1)) must fail. This matches
|
||||
# skill-audit/scripts/validate.sh's `line_count <= 500` pass condition; the two
|
||||
# factory-audit's lib-checks-skill.sh `line_count <= 500` pass condition; the two
|
||||
# previously disagreed at exactly $MAX_LINES lines, so a SKILL.md could pass its
|
||||
# own audit and still be blocked by the commit hook.
|
||||
echo ""
|
||||
@@ -514,7 +522,7 @@ fi
|
||||
# A skill carrying `disable-model-invocation: true` is removed from the
|
||||
# model-visible listing entirely — it is not preloaded, and the Skill tool
|
||||
# refuses to call it — so its description is never matched against user intent.
|
||||
# ADR-0020, skill-author Step 2 and skill-audit's own Step 0 all give it ONE
|
||||
# ADR-0020, skill-author Step 2 and factory-audit's own Gotchas all give it ONE
|
||||
# plain human-facing sentence: no trigger list, no boundary clause. No validator
|
||||
# knew the field existed, so the boundary-clause SUGGESTION fired on exactly the
|
||||
# shape the contract mandates, and its remedy — "so the router knows where NOT
|
||||
|
||||
@@ -36,12 +36,16 @@ export PRE_COMMIT_HOME="$WORK/pc-home"
|
||||
mkdir -p "$HOOK_REPO/plugins/kyberforge/.apm/skills" "$HOOK_REPO/scripts"
|
||||
cp "$REPO_ROOT/.pre-commit-hooks.yaml" "$HOOK_REPO/"
|
||||
cp "$REPO_ROOT/scripts/skill-size-check.sh" "$HOOK_REPO/scripts/"
|
||||
for skill in skill-audit agent-audit; do
|
||||
mkdir -p "$HOOK_REPO/plugins/kyberforge/.apm/skills/$skill"
|
||||
cp -R "$REPO_ROOT/plugins/kyberforge/.apm/skills/$skill/scripts" \
|
||||
"$REPO_ROOT/plugins/kyberforge/.apm/skills/$skill/assets" \
|
||||
"$HOOK_REPO/plugins/kyberforge/.apm/skills/$skill/"
|
||||
done
|
||||
# One skill since ADR-0025 merged skill-audit and agent-audit into factory-audit,
|
||||
# and one vale-wrap.sh with it. Both Vale hook IDs still ship and both are still
|
||||
# registered by the consumer below — they now point at the same entry and differ
|
||||
# only in their `files:` scope, which is exactly what the per-hook attribution in
|
||||
# case 1 exists to prove is still true.
|
||||
skill=factory-audit
|
||||
mkdir -p "$HOOK_REPO/plugins/kyberforge/.apm/skills/$skill"
|
||||
cp -R "$REPO_ROOT/plugins/kyberforge/.apm/skills/$skill/scripts" \
|
||||
"$REPO_ROOT/plugins/kyberforge/.apm/skills/$skill/assets" \
|
||||
"$HOOK_REPO/plugins/kyberforge/.apm/skills/$skill/"
|
||||
git -C "$HOOK_REPO" init -q
|
||||
git -C "$HOOK_REPO" add -A
|
||||
git -C "$HOOK_REPO" -c user.email=test@example.invalid -c user.name=test commit -qm "hook repo"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user