fix(kyberforge): close the vacuous-pass paths in the ADR-0020 gate scripts
Three ways the gates could report green having measured nothing. All three were
invisible to a passing test suite, because pre-commit prints nothing at all for a
hook that exits 0 — a gate that declines to check and a gate that checked and
passed produce the identical signal.
- A UTF-8 BOM, a leading blank line, a trailing space after a `---` marker or
CRLF line endings defeated the `^---\n` frontmatter matcher. Every ADR-0020
check was then skipped and the file passed: measured at the time, a
550-character description with a 1,000-word body exited 0 behind a BOM.
All four shapes are now tolerated, and frontmatter that genuinely cannot be
parsed is a hard ERROR rather than a silent skip.
- An agent file with a valueless `description:` followed by another key let a
line regex capture the *next* key, which looked non-empty, so the
missing-or-empty branch never fired and every gate below it early-returned on
the empty folded value — zero output, exit 0, on a blocking gate. The one
field this contract is entirely about was the one field a gate could fail to
notice was absent. Presence is now decided on the YAML-folded value and
nowhere else, and a missing or empty description is a hard FAIL in all three
validators.
- The hand-rolled frontmatter fallback disagreed with PyYAML across the FAIL
boundary on folded scalars, so which reader happened to be available decided
the verdict. A fallback that mis-parses a scalar shape reports a vacuous pass,
which is worse than not running, so it is deleted: python3 and PyYAML are hard
requirements that fail loudly with an install pointer.
Boundary-target resolution no longer derives its universe from its own location.
A `${BASH_SOURCE}`-relative repo root leaked this repo's 39-skill universe into
every consumer repo running the hook through pre-commit, so a consumer skill
routing to `skill-audit` resolved against a plugin it had never installed. The
interim form resolved through `.claude/` and `.agents/`, which are gitignored
`apm install` output — the same commit reported 2 dangling targets on a machine
that had run the install and 6 on a fresh clone. Resolution now walks up from the
file being checked to an authoring root (nearest ancestor holding
`plugins/*/.apm/{skills,agents}`, else the nearest `.git`, in two passes so a
nested `.git` cannot outrank a real monorepo root); the universe is every skill
and agent under `<root>/plugins/*/` plus the file's own apm package and that
package's declared `dependencies.apm`. Deployed trees are consulted only when no
authoring root exists at all — the consumer case. One commit now gets one verdict,
which a gate shipping hot with no baseline file has to.
Narrowed in the same pass: a routing target inferred from the prose boundary form
and corroborated by nothing else reports at SUGGESTION instead of blocking. A
blocking check with no escape hatch is the wrong trade when the inference from
prose is the weak part of it.
New deterministic checks, all previously untested or absent: every
`references/<file>.md` a body names must exist (ERROR — a broken pointer is not a
style opinion); a description with no boundary clause at all, a Gotchas section
over five entries, and a Gotchas section over 25% of the body are SUGGESTIONs.
Where no universe can be determined the target check prints `INFO ... DID NOT
RUN` rather than passing quietly. Each prose-scanning check needed its own
false-positive fix — a fenced example of a Gotchas section was being read as the
section itself — and those fixes are pinned rather than assumed.
The resolver is one block copied verbatim into all three scripts between
BEGIN/END markers, because a cache-installed plugin's scripts cannot read outside
their own plugin directory. Nothing asserted the copies were still identical; a
one-line edit to a single copy passed every constant-agreement assertion, since
constants are not what drifts.
Tests land here rather than in a later commit. The existing suites assert the old
behaviour and go red against these scripts, so splitting them would leave a commit
whose own `run-tests` pre-push gate fails in isolation.
Refs: ADR-0020
This commit is contained in:
406
tests/test-adr0020-body-checks.sh
Executable file
406
tests/test-adr0020-body-checks.sh
Executable file
@@ -0,0 +1,406 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for the ADR-0020 body-shape checks and, just as importantly,
|
||||
# for the false-positive fixes each of them needed. Every check here was
|
||||
# completely untested.
|
||||
#
|
||||
# * Gotchas section over 5 entries — SUGGESTION
|
||||
# * Gotchas section over 25% of the body — SUGGESTION
|
||||
# * a references/<file>.md named but absent — ERROR (a broken pointer is not a
|
||||
# style opinion)
|
||||
# * description with no boundary clause — SUGGESTION
|
||||
#
|
||||
# The false-positive half is not optional extra coverage. Each of these checks
|
||||
# scans prose, and the first naive version of each one fired on ordinary writing:
|
||||
# a ```-fenced EXAMPLE of a Gotchas section became the section itself, indented
|
||||
# child bullets were counted as top-level entries, `## Gotcha handling` was read
|
||||
# as the Gotchas section, and a documented-then-removed references/ file became a
|
||||
# hard ERROR. The skills most likely to carry such an example are skill-author and
|
||||
# skill-audit — the two that DOCUMENT these conventions — so a gate that fires on
|
||||
# them is a gate nobody can turn on.
|
||||
#
|
||||
# Every case is a matched pair: the check fires just over its boundary, and stays
|
||||
# silent just under it (or on the shape it must not match). A test asserting only
|
||||
# that a bad file fails proves nothing about a check that fires on everything.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
|
||||
TMPDIR_T="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||
|
||||
# A description with a boundary clause and no routing target — the neutral
|
||||
# default, so a fixture about Gotchas or references does not also trip the
|
||||
# missing-boundary-clause SUGGESTION and stop isolating what it names.
|
||||
CLEAN_DESC="Use when doing the thing. Do not use for anything else."
|
||||
|
||||
# make_skill <name> <desc> — SKILL.md with the body read from stdin. Echoes the
|
||||
# path. Each skill gets its own directory so references/ fixtures are isolated.
|
||||
make_skill() {
|
||||
local name="$1" desc="$2" dir
|
||||
dir="$TMPDIR_T/$name"
|
||||
mkdir -p "$dir"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: $name"
|
||||
echo "description: $desc"
|
||||
echo "---"
|
||||
cat
|
||||
} > "$dir/SKILL.md"
|
||||
echo "$dir/SKILL.md"
|
||||
}
|
||||
|
||||
# expect <label> <file> <expect: silent|suggests|errors> [needle] [absent-needle]
|
||||
#
|
||||
# `silent` is the strict one: exit 0 AND completely empty output. It is what
|
||||
# makes every "must not fire" case below real — a check that fired with some
|
||||
# other wording would still be caught.
|
||||
expect() {
|
||||
local label="$1" file="$2" mode="$3" needle="${4:-}" absent="${5:-}" out status=0
|
||||
set +e
|
||||
out="$(bash "$HOOK" "$file" 2>&1)"
|
||||
status=$?
|
||||
set -e
|
||||
case "$mode" in
|
||||
silent)
|
||||
if [[ $status -eq 0 && -z "$out" ]]; then
|
||||
pass "$label"
|
||||
else
|
||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
suggests)
|
||||
if [[ $status -ne 0 ]]; then
|
||||
fail "$label — a SUGGESTION must never change the exit code (exit $status, output: $out)"
|
||||
elif [[ "$out" != *"SUGGESTION"* || "$out" != *"$needle"* ]]; then
|
||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||
elif [[ -n "$absent" && "$out" == *"$absent"* ]]; then
|
||||
fail "$label — output also contained '$absent', which must not fire here: $out"
|
||||
else
|
||||
pass "$label"
|
||||
fi
|
||||
;;
|
||||
errors)
|
||||
if [[ $status -eq 0 ]]; then
|
||||
fail "$label — expected a hard ERROR, got exit 0 (output: ${out:-<empty>})"
|
||||
elif [[ "$out" != *"$needle"* ]]; then
|
||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||
else
|
||||
pass "$label"
|
||||
fi
|
||||
;;
|
||||
quiet-about)
|
||||
# Exit 0 and the named text absent, but other output permitted. Used where
|
||||
# a second, unrelated finding legitimately fires.
|
||||
if [[ $status -ne 0 ]]; then
|
||||
fail "$label (exit $status, output: $out)"
|
||||
elif [[ "$out" == *"$needle"* ]]; then
|
||||
fail "$label — '$needle' fired when it must not: $out"
|
||||
else
|
||||
pass "$label"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
filler() { python3 -c "print(' '.join(['word'] * $1))"; }
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gotchas: entry count (guideline 5)
|
||||
# ---------------------------------------------------------------------------
|
||||
# The filler after the section keeps the 25% fraction check well clear, so these
|
||||
# two cases isolate the ENTRY count. Without it a six-entry section in a short
|
||||
# body would fire both and the pair would not distinguish them.
|
||||
echo ""
|
||||
echo "--- Gotchas entry count: 5 is fine, 6 is a SUGGESTION ---"
|
||||
F_FIVE="$(make_skill gotchas-five "$CLEAN_DESC" <<EOF
|
||||
|
||||
## Gotchas
|
||||
|
||||
- first trap here
|
||||
- second trap here
|
||||
- third trap here
|
||||
- fourth trap here
|
||||
- fifth trap here
|
||||
|
||||
## Notes
|
||||
|
||||
$(filler 200)
|
||||
EOF
|
||||
)"
|
||||
expect "a Gotchas section with exactly 5 entries is silent" "$F_FIVE" silent
|
||||
|
||||
F_SIX="$(make_skill gotchas-six "$CLEAN_DESC" <<EOF
|
||||
|
||||
## Gotchas
|
||||
|
||||
- first trap here
|
||||
- second trap here
|
||||
- third trap here
|
||||
- fourth trap here
|
||||
- fifth trap here
|
||||
- sixth trap here
|
||||
|
||||
## Notes
|
||||
|
||||
$(filler 200)
|
||||
EOF
|
||||
)"
|
||||
expect "a Gotchas section with 6 entries raises a SUGGESTION and still exits 0" \
|
||||
"$F_SIX" suggests "Gotchas section has 6 entries" "over the 25% guideline"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gotchas: share of the body (guideline 25%)
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exact boundary arithmetic, not an approximation. The section is prose (no list
|
||||
# items) so the entry check cannot fire and confuse the result; body words are
|
||||
# then section + filler + the two two-word headings. At a body of 100 words a
|
||||
# 25-word section is exactly the guideline (inclusive — `>` is the comparison, so
|
||||
# it passes) and a 26-word section is one word past it.
|
||||
echo ""
|
||||
echo "--- Gotchas share of body: exactly 25% is fine, 26% is a SUGGESTION ---"
|
||||
F_AT="$(make_skill gotchas-at-fraction "$CLEAN_DESC" <<EOF
|
||||
|
||||
## Gotchas
|
||||
|
||||
$(filler 25)
|
||||
|
||||
## Notes
|
||||
|
||||
$(filler 71)
|
||||
EOF
|
||||
)"
|
||||
expect "a Gotchas section at exactly 25% of the body is silent" "$F_AT" silent
|
||||
|
||||
F_OVER="$(make_skill gotchas-over-fraction "$CLEAN_DESC" <<EOF
|
||||
|
||||
## Gotchas
|
||||
|
||||
$(filler 26)
|
||||
|
||||
## Notes
|
||||
|
||||
$(filler 70)
|
||||
EOF
|
||||
)"
|
||||
expect "a Gotchas section at 26% of the body raises a SUGGESTION" \
|
||||
"$F_OVER" suggests "Gotchas section is 26 of 100 body words (26%)" "entries"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Gotchas: false positives
|
||||
# ---------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- a ## Gotchas heading inside a fenced block is not the Gotchas section ---"
|
||||
# skill-author and skill-audit both document this convention by showing it. If a
|
||||
# fenced example counted, the two skills that define the rule would be the two
|
||||
# most likely to fail it.
|
||||
F_FENCED_HEADING="$(make_skill gotchas-fenced-heading "$CLEAN_DESC" <<EOF
|
||||
|
||||
## How to write one
|
||||
|
||||
\`\`\`markdown
|
||||
## Gotchas
|
||||
|
||||
- example one
|
||||
- example two
|
||||
- example three
|
||||
- example four
|
||||
- example five
|
||||
- example six
|
||||
- example seven
|
||||
\`\`\`
|
||||
|
||||
$(filler 200)
|
||||
EOF
|
||||
)"
|
||||
expect "a fenced ## Gotchas heading is not read as the section" \
|
||||
"$F_FENCED_HEADING" quiet-about "Gotchas section"
|
||||
|
||||
echo ""
|
||||
echo "--- list items inside a fenced block do not count as Gotchas entries ---"
|
||||
F_FENCED_ENTRIES="$(make_skill gotchas-fenced-entries "$CLEAN_DESC" <<EOF
|
||||
|
||||
## Gotchas
|
||||
|
||||
- a real trap
|
||||
- another real trap
|
||||
|
||||
Shown as an example of what NOT to write:
|
||||
|
||||
\`\`\`markdown
|
||||
- fake one
|
||||
- fake two
|
||||
- fake three
|
||||
- fake four
|
||||
- fake five
|
||||
- fake six
|
||||
- fake seven
|
||||
- fake eight
|
||||
\`\`\`
|
||||
|
||||
## Notes
|
||||
|
||||
$(filler 300)
|
||||
EOF
|
||||
)"
|
||||
expect "eight fenced bullets plus two real ones counts as two entries, not ten" \
|
||||
"$F_FENCED_ENTRIES" quiet-about "Gotchas section has"
|
||||
|
||||
echo ""
|
||||
echo "--- the heading must END in gotcha(s): '## Gotcha handling' is not the section ---"
|
||||
# `## Gotcha handling` and `## Why gotchas matter` are prose sections. Treating
|
||||
# one as the Gotchas section measures a span that was never a gotcha list.
|
||||
F_HANDLING="$(make_skill gotcha-handling "$CLEAN_DESC" <<EOF
|
||||
|
||||
## Gotcha handling
|
||||
|
||||
- item one
|
||||
- item two
|
||||
- item three
|
||||
- item four
|
||||
- item five
|
||||
- item six
|
||||
- item seven
|
||||
|
||||
## Notes
|
||||
|
||||
$(filler 200)
|
||||
EOF
|
||||
)"
|
||||
expect "'## Gotcha handling' is not matched as the Gotchas section" \
|
||||
"$F_HANDLING" quiet-about "Gotchas section"
|
||||
|
||||
# The control for the three cases above. Without it, "no Gotchas finding" could
|
||||
# equally mean the whole check is dead, and all three would still be green.
|
||||
F_CONTROL="$(make_skill gotchas-control "$CLEAN_DESC" <<EOF
|
||||
|
||||
## Common gotchas
|
||||
|
||||
- item one
|
||||
- item two
|
||||
- item three
|
||||
- item four
|
||||
- item five
|
||||
- item six
|
||||
- item seven
|
||||
|
||||
## Notes
|
||||
|
||||
$(filler 200)
|
||||
EOF
|
||||
)"
|
||||
expect "control: a real '## Common gotchas' heading with 7 entries IS matched" \
|
||||
"$F_CONTROL" suggests "Gotchas section has 7 entries"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# references/<file>.md pointers
|
||||
# ---------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- a references/ pointer that is not on disk is a hard ERROR ---"
|
||||
F_REF_MISSING="$(make_skill ref-missing "$CLEAN_DESC" <<EOF
|
||||
|
||||
If the caller needs the long form, read references/nowhere.md first.
|
||||
EOF
|
||||
)"
|
||||
expect "a body pointing at an absent references/nowhere.md ERRORs" \
|
||||
"$F_REF_MISSING" errors "points at references/nowhere.md"
|
||||
|
||||
F_REF_PRESENT="$(make_skill ref-present "$CLEAN_DESC" <<EOF
|
||||
|
||||
If the caller needs the long form, read references/here.md first.
|
||||
EOF
|
||||
)"
|
||||
mkdir -p "$TMPDIR_T/ref-present/references"
|
||||
echo "content" > "$TMPDIR_T/ref-present/references/here.md"
|
||||
expect "the same pointer is silent once the file exists" "$F_REF_PRESENT" silent
|
||||
|
||||
echo ""
|
||||
echo "--- a references/ pointer inside a fenced block is not a dispatch entry ---"
|
||||
F_REF_FENCED="$(make_skill ref-fenced "$CLEAN_DESC" <<EOF
|
||||
|
||||
Dispatch tables look like this:
|
||||
|
||||
\`\`\`markdown
|
||||
If X, read references/example-file.md.
|
||||
\`\`\`
|
||||
EOF
|
||||
)"
|
||||
expect "a fenced references/example-file.md does not ERROR" "$F_REF_FENCED" silent
|
||||
|
||||
echo ""
|
||||
echo "--- a references/ pointer in a same-line removal context is history, not dispatch ---"
|
||||
# Narrow on purpose: a live dispatch table never describes its own target as
|
||||
# removed, so the exemption costs no recall. Each phrasing is checked separately
|
||||
# because they are separate alternatives in one regex, and a typo in any one of
|
||||
# them turns ordinary prose back into a hard ERROR.
|
||||
#
|
||||
# The file names are deliberately NEUTRAL (detail-a.md, not gone-a.md). An
|
||||
# earlier draft of this block named them gone-*.md and every case passed for the
|
||||
# wrong reason: "gone" is itself one of the removal words, so the exemption fired
|
||||
# off the FILENAME and the phrase under test was never exercised. The control
|
||||
# below is what surfaced that — it is the assertion that keeps these five honest.
|
||||
i=0
|
||||
for phrase in \
|
||||
"The old references/detail-a.md was removed in v2." \
|
||||
"references/detail-b.md is no longer part of this skill." \
|
||||
"references/detail-c.md is deprecated and should not be read." \
|
||||
"references/detail-d.md was renamed, so nothing points at it now." \
|
||||
"references/detail-e.md has been superseded by the body itself."; do
|
||||
i=$((i + 1))
|
||||
F_REF_PAST="$(make_skill "ref-past-$i" "$CLEAN_DESC" <<EOF
|
||||
|
||||
$phrase
|
||||
EOF
|
||||
)"
|
||||
expect "removal-context pointer is not an ERROR: \"$phrase\"" "$F_REF_PAST" silent
|
||||
done
|
||||
|
||||
# The control: the SAME sentence shape without a removal word must still ERROR,
|
||||
# or the exemption above has swallowed the check rather than narrowed it.
|
||||
F_REF_LIVE="$(make_skill ref-live "$CLEAN_DESC" <<EOF
|
||||
|
||||
The details live in references/detail-a.md, which the agent should read first.
|
||||
EOF
|
||||
)"
|
||||
expect "control: the same pointer with no removal word still ERRORs" \
|
||||
"$F_REF_LIVE" errors "points at references/detail-a.md"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Missing boundary clause
|
||||
# ---------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- a description with no boundary clause raises a SUGGESTION ---"
|
||||
F_NO_BOUNDARY="$(make_skill no-boundary "Use when the user wants the thing done." <<EOF
|
||||
|
||||
Do the thing.
|
||||
EOF
|
||||
)"
|
||||
expect "a description with no boundary clause raises a SUGGESTION and still exits 0" \
|
||||
"$F_NO_BOUNDARY" suggests "description has no boundary clause"
|
||||
|
||||
# Both accepted shapes, asserted separately: the prose markers and ADR-0020's
|
||||
# compressed arrow form. Dropping either from the detector would leave the other
|
||||
# green.
|
||||
F_PROSE_BOUNDARY="$(make_skill prose-boundary "Use when the user wants the thing done. Do not use for anything else." <<EOF
|
||||
|
||||
Do the thing.
|
||||
EOF
|
||||
)"
|
||||
expect "the prose boundary form satisfies the check" "$F_PROSE_BOUNDARY" silent
|
||||
|
||||
F_ARROW_BOUNDARY="$(make_skill arrow-boundary "Use when the user wants the thing done. Not the other thing -> sibling-skill." <<EOF
|
||||
|
||||
Do the thing.
|
||||
EOF
|
||||
)"
|
||||
expect "ADR-0020's compressed 'Not X -> y' form satisfies the check" \
|
||||
"$F_ARROW_BOUNDARY" quiet-about "no boundary clause"
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
331
tests/test-adr0020-contract.sh
Executable file
331
tests/test-adr0020-contract.sh
Executable file
@@ -0,0 +1,331 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for the three STRUCTURAL claims the ADR-0020 gate family makes
|
||||
# about itself. None of them was pinned anywhere before this file, and each one
|
||||
# fails silently — which is the whole reason they need a test rather than a
|
||||
# comment:
|
||||
#
|
||||
# 1. "ONE resolver, embedded VERBATIM in three scripts." The block between the
|
||||
# BEGIN/END markers is copied, not imported, because a cache-installed
|
||||
# plugin's scripts cannot read files outside their own plugin directory.
|
||||
# Nothing but this file asserts the three copies are still identical, and a
|
||||
# one-line edit to a single copy is invisible: every constant-agreement
|
||||
# assertion in tests/test-skill-size-check.sh still passes, because the
|
||||
# CONSTANTS are not what drifted.
|
||||
# 2. Both interpreter preflights, in all three scripts. python3 and PyYAML are
|
||||
# declared HARD dependencies precisely so a missing one cannot turn into a
|
||||
# vacuous pass, and the two are checked separately so the message names the
|
||||
# thing to install rather than the wrong one.
|
||||
# 3. `verbose: true` on the skill-size-check hook. It is the ENTIRE delivery
|
||||
# mechanism for the SUGGESTION tier: pre-commit prints nothing at all for a
|
||||
# passing hook, and a SUGGESTION deliberately does not fail, so dropping
|
||||
# one word from the config silences the tier ADR-0020 depends on while
|
||||
# every test and every hook still reports green.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
|
||||
TMPDIR_T="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||
|
||||
BEGIN_MARKER='# ===== BEGIN ADR-0020 SHARED BOUNDARY RESOLVER ====='
|
||||
END_MARKER='# ===== END ADR-0020 SHARED BOUNDARY RESOLVER ====='
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The shared resolver block is byte-identical in all three scripts
|
||||
# ---------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- the ADR-0020 shared resolver block is byte-identical in all three scripts ---"
|
||||
|
||||
# Marker discipline first. An unbalanced or duplicated marker pair makes the
|
||||
# extraction below silently measure the wrong span — a sed range that never
|
||||
# closes swallows the rest of the file, and one that opens twice concatenates
|
||||
# two spans. Both would still compare "equal" if all three were mangled the
|
||||
# same way, so the shape is asserted before the contents.
|
||||
MARKERS_OK=true
|
||||
for f in "$HOOK" "$SKILL_VALIDATE" "$AGENT_VALIDATE"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
fail "script not found: $f"
|
||||
MARKERS_OK=false
|
||||
continue
|
||||
fi
|
||||
b="$(grep -cFx "$BEGIN_MARKER" "$f" || true)"
|
||||
e="$(grep -cFx "$END_MARKER" "$f" || true)"
|
||||
if [[ "$b" == "1" && "$e" == "1" ]]; then
|
||||
pass "${f#"$REPO_ROOT/"} carries exactly one BEGIN and one END marker"
|
||||
else
|
||||
fail "${f#"$REPO_ROOT/"} has $b BEGIN and $e END markers, expected 1 and 1"
|
||||
MARKERS_OK=false
|
||||
fi
|
||||
done
|
||||
|
||||
if ! $MARKERS_OK; then
|
||||
fail "skipping the byte-identity comparison — the marker pairs are not well-formed, so any extraction would measure the wrong span"
|
||||
else
|
||||
HASHES=()
|
||||
LINECOUNTS=()
|
||||
for f in "$HOOK" "$SKILL_VALIDATE" "$AGENT_VALIDATE"; do
|
||||
out="$TMPDIR_T/block-$(echo "$f" | md5sum | cut -c1-8).txt"
|
||||
sed -n "/^${BEGIN_MARKER}\$/,/^${END_MARKER}\$/p" "$f" > "$out"
|
||||
HASHES+=("$(md5sum < "$out" | cut -d' ' -f1)")
|
||||
LINECOUNTS+=("$(wc -l < "$out" | tr -d ' ')")
|
||||
done
|
||||
if [[ "${HASHES[0]}" == "${HASHES[1]}" && "${HASHES[1]}" == "${HASHES[2]}" ]]; then
|
||||
pass "all three copies hash to ${HASHES[0]} (${LINECOUNTS[0]} lines) — agreement by construction, not by coincidence"
|
||||
else
|
||||
fail "the shared resolver has DRIFTED: skill-size-check=${HASHES[0]} (${LINECOUNTS[0]} lines), skill-audit=${HASHES[1]} (${LINECOUNTS[1]} lines), agent-audit=${HASHES[2]} (${LINECOUNTS[2]} lines). Edit one copy, then paste it over the other two."
|
||||
fi
|
||||
# A block that has been emptied out would hash equal in all three and pass the
|
||||
# comparison above while enforcing nothing. The resolver is ~570 lines; 100 is
|
||||
# a floor low enough never to need maintenance and high enough that a gutted
|
||||
# block cannot sneak past.
|
||||
if [[ "${LINECOUNTS[0]}" -gt 100 ]]; then
|
||||
pass "the extracted block is ${LINECOUNTS[0]} lines — the comparison is over real content, not an empty span"
|
||||
else
|
||||
fail "the extracted shared block is only ${LINECOUNTS[0]} lines — three identical empty spans would compare equal and assert nothing"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Both interpreter preflights, in all three scripts
|
||||
# ---------------------------------------------------------------------------
|
||||
# The two are checked separately on purpose: `python3 -c 'import yaml'` fails
|
||||
# identically whether python3 is missing or PyYAML is, and naming the wrong one
|
||||
# sends the reader to install the wrong thing.
|
||||
REAL_PYTHON="$(command -v python3)"
|
||||
# Absolute path, deliberately. The no-python3 fixture below replaces PATH
|
||||
# wholesale, so a bare `bash` (or `/usr/bin/env bash`) would be resolved against
|
||||
# that stripped PATH and die with "No such file or directory" before the script
|
||||
# under test ever starts -- a 127 that looks like the preflight firing.
|
||||
BASH_BIN="$(command -v bash)"
|
||||
|
||||
# A PATH that genuinely has no python3 on it. Built by symlinking the handful of
|
||||
# binaries the three scripts touch before their own preflight rather than by
|
||||
# hiding python3 from a full PATH, because there is no portable way to subtract
|
||||
# one entry from a directory. `bash` is invoked by absolute path below so the
|
||||
# interpreter itself does not have to be on this PATH.
|
||||
NOPY_BIN="$TMPDIR_T/nopython-bin"
|
||||
mkdir -p "$NOPY_BIN"
|
||||
for b in awk cat cut dirname basename grep sed pwd rm mkdir tr; do
|
||||
src="$(command -v "$b" 2>/dev/null || true)"
|
||||
[[ -n "$src" ]] && ln -sf "$src" "$NOPY_BIN/$b"
|
||||
done
|
||||
|
||||
# A python3 that runs but cannot import yaml. A shim on PATH re-execs the real
|
||||
# interpreter with a PYTHONPATH entry holding a `yaml` module that raises on
|
||||
# import; PYTHONPATH precedes site-packages on sys.path, so it shadows a real
|
||||
# PyYAML install without touching it.
|
||||
SHADOW="$TMPDIR_T/shadow"
|
||||
mkdir -p "$SHADOW"
|
||||
printf 'raise ImportError("PyYAML deliberately unavailable in this fixture")\n' \
|
||||
> "$SHADOW/yaml.py"
|
||||
NOYAML_BIN="$TMPDIR_T/noyaml-bin"
|
||||
mkdir -p "$NOYAML_BIN"
|
||||
cat > "$NOYAML_BIN/python3" <<EOF
|
||||
#!/bin/sh
|
||||
PYTHONPATH="$SHADOW\${PYTHONPATH:+:\$PYTHONPATH}" exec "$REAL_PYTHON" "\$@"
|
||||
EOF
|
||||
chmod +x "$NOYAML_BIN/python3"
|
||||
|
||||
# Sanity-check the two fixtures themselves before trusting any verdict they
|
||||
# produce. A shim that silently still imports yaml would make every PyYAML
|
||||
# assertion below pass for the wrong reason.
|
||||
if PATH="$NOYAML_BIN:$PATH" python3 -c 'import yaml' 2>/dev/null; then
|
||||
fail "the no-PyYAML shim does not actually shadow PyYAML — every PyYAML assertion below would be vacuous"
|
||||
else
|
||||
pass "fixture check: the no-PyYAML shim makes 'import yaml' fail while python3 still runs"
|
||||
fi
|
||||
if PATH="$NOPY_BIN" command -v python3 > /dev/null 2>&1; then
|
||||
fail "the no-python3 PATH still resolves python3 — every python3 assertion below would be vacuous"
|
||||
else
|
||||
pass "fixture check: the no-python3 PATH resolves no python3"
|
||||
fi
|
||||
|
||||
# A minimal, entirely clean subject for each script. The preflight must fire
|
||||
# before any measurement, so the subject's own content is irrelevant — which is
|
||||
# exactly what makes a clean one the right choice: nothing else can produce the
|
||||
# non-zero exit these cases assert.
|
||||
SUBJECT_SKILL_DIR="$TMPDIR_T/subject/my-skill"
|
||||
mkdir -p "$SUBJECT_SKILL_DIR"
|
||||
cat > "$SUBJECT_SKILL_DIR/SKILL.md" <<'EOF'
|
||||
---
|
||||
name: my-skill
|
||||
description: A short valid description. Do not use for anything else.
|
||||
---
|
||||
|
||||
Do the thing.
|
||||
EOF
|
||||
SUBJECT_AGENT_ROOT="$TMPDIR_T/subject-agent"
|
||||
mkdir -p "$SUBJECT_AGENT_ROOT/.apm/agents"
|
||||
cat > "$SUBJECT_AGENT_ROOT/apm.yml" <<'EOF'
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md" <<'EOF'
|
||||
---
|
||||
name: my-agent
|
||||
description: A short valid description. Do not use for anything else.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
|
||||
# probe_preflight <label> <env-kind: nopython|noyaml> <expect-needle> <cmd...>
|
||||
probe_preflight() {
|
||||
local label="$1" kind="$2" needle="$3"
|
||||
shift 3
|
||||
local out status=0
|
||||
set +e
|
||||
if [[ "$kind" == nopython ]]; then
|
||||
out="$(env -i PATH="$NOPY_BIN" HOME="$HOME" "$BASH_BIN" "$@" 2>&1)"
|
||||
else
|
||||
out="$(env PATH="$NOYAML_BIN:$PATH" "$BASH_BIN" "$@" 2>&1)"
|
||||
fi
|
||||
status=$?
|
||||
set -e
|
||||
if [[ $status -eq 0 ]]; then
|
||||
fail "$label exited 0 — a missing hard dependency became a vacuous pass (output: ${out:-<empty>})"
|
||||
elif [[ "$out" != *"$needle"* ]]; then
|
||||
# The needle is the DIAGNOSTIC ("python3 is required"), not the bare word.
|
||||
# Deleting the preflight entirely would still produce a non-zero exit and a
|
||||
# message mentioning python3 -- bash's own "python3: command not found" --
|
||||
# so a bare-word needle would go green on a script with no preflight at all.
|
||||
fail "$label exited $status but never produced the '$needle' diagnostic (output: ${out:-<empty>})"
|
||||
elif [[ "$kind" == nopython && "$out" == *PyYAML* ]]; then
|
||||
fail "$label reported PyYAML when python3 itself is missing — that sends the reader to install the wrong thing (output: $out)"
|
||||
else
|
||||
pass "$label"
|
||||
fi
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "--- a PATH with no python3 is a hard failure in all three scripts, naming python3 ---"
|
||||
probe_preflight "scripts/skill-size-check.sh reports missing python3" \
|
||||
nopython "python3 is required" \
|
||||
"$HOOK" "$SUBJECT_SKILL_DIR/SKILL.md"
|
||||
probe_preflight "skill-audit/scripts/validate.sh reports missing python3" \
|
||||
nopython "python3 is required" \
|
||||
"$SKILL_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||
probe_preflight "agent-audit/scripts/validate.sh reports missing python3" \
|
||||
nopython "python3 is required" \
|
||||
"$AGENT_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||
|
||||
echo ""
|
||||
echo "--- a python3 that cannot import yaml is a hard failure in all three scripts, naming PyYAML ---"
|
||||
probe_preflight "scripts/skill-size-check.sh reports missing PyYAML" \
|
||||
noyaml "PyYAML is required" \
|
||||
"$HOOK" "$SUBJECT_SKILL_DIR/SKILL.md"
|
||||
probe_preflight "skill-audit/scripts/validate.sh reports missing PyYAML" \
|
||||
noyaml "PyYAML is required" \
|
||||
"$SKILL_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||
probe_preflight "agent-audit/scripts/validate.sh reports missing PyYAML" \
|
||||
noyaml "PyYAML is required" \
|
||||
"$AGENT_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||
|
||||
# The control. Without it, "fails when the dependency is missing" is satisfied by
|
||||
# a script that fails unconditionally, and the two cases above would be green on
|
||||
# a gate that never runs at all.
|
||||
echo ""
|
||||
echo "--- control: with both dependencies present the same subjects pass ---"
|
||||
for probe in "$HOOK:$SUBJECT_SKILL_DIR/SKILL.md" \
|
||||
"$SKILL_VALIDATE:$SUBJECT_SKILL_DIR" \
|
||||
"$AGENT_VALIDATE:$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"; do
|
||||
script="${probe%%:*}"
|
||||
arg="${probe#*:}"
|
||||
set +e
|
||||
ctl_out="$(bash "$script" "$arg" 2>&1)"
|
||||
ctl_rc=$?
|
||||
set -e
|
||||
if [[ $ctl_rc -eq 0 ]]; then
|
||||
pass "${script#"$REPO_ROOT/"} exits 0 on a clean subject with python3 and PyYAML available"
|
||||
else
|
||||
fail "${script#"$REPO_ROOT/"} failed a clean subject (exit $ctl_rc): $ctl_out"
|
||||
fi
|
||||
done
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. verbose: true on the skill-size-check hook, in BOTH manifests
|
||||
# ---------------------------------------------------------------------------
|
||||
# .pre-commit-config.yaml governs this repo; .pre-commit-hooks.yaml is what a
|
||||
# CONSUMER repo gets when it points at this one. Dropping the flag from either
|
||||
# silences the SUGGESTION tier for that audience alone, which is the hardest
|
||||
# version of the defect to notice.
|
||||
echo ""
|
||||
echo "--- the skill-size-check hook declares verbose: true in both manifests ---"
|
||||
VERBOSE_REPORT="$(python3 - "$REPO_ROOT" <<'PY'
|
||||
import os
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
root = sys.argv[1]
|
||||
|
||||
|
||||
def emit(status, msg):
|
||||
print("%s\t%s" % (status, msg))
|
||||
|
||||
|
||||
# Repo config: nested repos[].hooks[].
|
||||
path = os.path.join(root, '.pre-commit-config.yaml')
|
||||
try:
|
||||
with open(path, encoding='utf-8') as fh:
|
||||
cfg = yaml.safe_load(fh) or {}
|
||||
except Exception as exc:
|
||||
emit('FAIL', '.pre-commit-config.yaml did not parse: %s' % exc)
|
||||
cfg = {}
|
||||
found = None
|
||||
for repo in cfg.get('repos') or []:
|
||||
for hook in (repo.get('hooks') or []):
|
||||
if hook.get('id') == 'skill-size-check':
|
||||
found = hook
|
||||
if found is None:
|
||||
emit('FAIL', '.pre-commit-config.yaml declares no hook with id skill-size-check')
|
||||
elif found.get('verbose') is True:
|
||||
emit('PASS', '.pre-commit-config.yaml: skill-size-check is verbose: true')
|
||||
else:
|
||||
emit('FAIL', '.pre-commit-config.yaml: skill-size-check has verbose=%r — '
|
||||
'pre-commit prints nothing for a passing hook, so every '
|
||||
'ADR-0020 SUGGESTION is swallowed' % (found.get('verbose'),))
|
||||
|
||||
# Consumer manifest: a flat list of hooks.
|
||||
path = os.path.join(root, '.pre-commit-hooks.yaml')
|
||||
try:
|
||||
with open(path, encoding='utf-8') as fh:
|
||||
hooks = yaml.safe_load(fh) or []
|
||||
except Exception as exc:
|
||||
emit('FAIL', '.pre-commit-hooks.yaml did not parse: %s' % exc)
|
||||
hooks = []
|
||||
found = None
|
||||
for hook in hooks:
|
||||
if isinstance(hook, dict) and hook.get('id') == 'kyberforge-skill-size-check':
|
||||
found = hook
|
||||
if found is None:
|
||||
emit('FAIL', '.pre-commit-hooks.yaml declares no hook with id kyberforge-skill-size-check')
|
||||
elif found.get('verbose') is True:
|
||||
emit('PASS', '.pre-commit-hooks.yaml: kyberforge-skill-size-check is verbose: true')
|
||||
else:
|
||||
emit('FAIL', '.pre-commit-hooks.yaml: kyberforge-skill-size-check has verbose=%r — '
|
||||
'a consumer repo would never see the SUGGESTION tier'
|
||||
% (found.get('verbose'),))
|
||||
PY
|
||||
)"
|
||||
while IFS=$'\t' read -r status msg; do
|
||||
[[ -n "$status" ]] || continue
|
||||
if [[ "$status" == PASS ]]; then
|
||||
pass "$msg"
|
||||
else
|
||||
fail "$msg"
|
||||
fi
|
||||
done <<< "$VERBOSE_REPORT"
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
365
tests/test-adr0020-differential.sh
Executable file
365
tests/test-adr0020-differential.sh
Executable file
@@ -0,0 +1,365 @@
|
||||
#!/usr/bin/env bash
|
||||
# Differential test: scripts/skill-size-check.sh (the pre-commit hook) and
|
||||
# skill-audit/scripts/validate.sh (the in-skill auditor) must reach the SAME
|
||||
# ADR-0020 verdict on the same file.
|
||||
#
|
||||
# Why this exists as a separate suite. tests/test-skill-size-check.sh already
|
||||
# asserts the two agree on their CONSTANTS, and that assertion is necessary but
|
||||
# demonstrably not sufficient: a previous review found the two scripts disagreeing
|
||||
# on real files while every constant matched perfectly. Constants are one of the
|
||||
# ways two hand-duplicated implementations diverge; comparison operators, message
|
||||
# wording, which value gets measured, and which branch runs first are the others,
|
||||
# and none of them is visible to a constant check.
|
||||
#
|
||||
# The consequence of divergence is specific and bad: skill-audit reports a skill
|
||||
# ready to ship and the commit hook then rejects it, or worse, the reverse. So the
|
||||
# comparison here is over VERDICTS on files, not over source text.
|
||||
#
|
||||
# Scope: the ADR-0020 axes the two scripts share — description length and tier,
|
||||
# body word count and tier, dangling routing targets, missing references/
|
||||
# pointers, the two Gotchas suggestions, the missing-boundary-clause suggestion,
|
||||
# a declined resolution, and an empty description. The two scripts legitimately
|
||||
# differ elsewhere (validate.sh also checks name/directory agreement, script
|
||||
# executability and the 1024-char spec backstop; the hook checks whole-file lines
|
||||
# and words), and those lines are ignored rather than being forced into a shared
|
||||
# shape they were never meant to have.
|
||||
#
|
||||
# Run over the real 39-skill corpus AND over purpose-built fixtures that sit ON
|
||||
# each boundary. The corpus alone is not enough — it happens not to contain a
|
||||
# file at exactly 900 body words, which is precisely where an inclusive/exclusive
|
||||
# comparison mismatch would hide.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||
|
||||
TMPDIR_T="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures: one per ADR-0020 axis, placed ON the boundary wherever there is one.
|
||||
# ---------------------------------------------------------------------------
|
||||
# Built inside a synthetic plugin monorepo so boundary-target resolution actually
|
||||
# runs for both scripts (in a bare temp dir both would decline, and "both
|
||||
# declined" is agreement about nothing).
|
||||
FIXTURE_ROOT="$TMPDIR_T/fixtures"
|
||||
FX="$FIXTURE_ROOT/plugins/fixture-plugin/.apm/skills"
|
||||
mkdir -p "$FX/sibling-skill" "$FIXTURE_ROOT/plugins/fixture-plugin/.apm/agents"
|
||||
|
||||
make_fx() {
|
||||
local name="$1" desc="$2" body_words="$3"
|
||||
mkdir -p "$FX/$name"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: $name"
|
||||
echo "description: $desc"
|
||||
echo "---"
|
||||
echo ""
|
||||
python3 -c "print(' '.join(['word'] * $body_words))"
|
||||
} > "$FX/$name/SKILL.md"
|
||||
}
|
||||
desc_of_length() {
|
||||
python3 - "$1" <<'PY'
|
||||
import sys
|
||||
n = int(sys.argv[1])
|
||||
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||
print(prefix + 'x' * (n - len(prefix)))
|
||||
PY
|
||||
}
|
||||
CLEAN="Use when doing the thing. Do not use for anything else."
|
||||
|
||||
# Description tier boundaries, both sides of both thresholds.
|
||||
make_fx desc-249 "$(desc_of_length 249)" 10
|
||||
make_fx desc-250 "$(desc_of_length 250)" 10
|
||||
make_fx desc-251 "$(desc_of_length 251)" 10
|
||||
make_fx desc-400 "$(desc_of_length 400)" 10
|
||||
make_fx desc-401 "$(desc_of_length 401)" 10
|
||||
# Body tier boundaries, both sides of both thresholds.
|
||||
make_fx body-599 "$CLEAN" 599
|
||||
make_fx body-600 "$CLEAN" 600
|
||||
make_fx body-601 "$CLEAN" 601
|
||||
make_fx body-900 "$CLEAN" 900
|
||||
make_fx body-901 "$CLEAN" 901
|
||||
# Folding: the value has to be measured after YAML folding in both scripts.
|
||||
mkdir -p "$FX/folded-desc"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: folded-desc"
|
||||
echo "description: >"
|
||||
python3 -c "print('\n'.join([' ' + 'x' * 40] * 11))"
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "Do the thing."
|
||||
} > "$FX/folded-desc/SKILL.md"
|
||||
# Routing targets, one per tier the resolver can produce: resolves, dangles with
|
||||
# in-sentence corroboration (ERROR/FAIL), dangles alone (SUGGESTION on both
|
||||
# sides), route notation (ERROR/FAIL without corroboration), attributive
|
||||
# (silent). Each tier is here because the two scripts have to agree on the TIER,
|
||||
# not merely on the finding — a copy that promoted or demoted one of them would
|
||||
# otherwise pass this comparison.
|
||||
make_fx target-resolves "Use when doing the thing. Do not use for the other thing — use sibling-skill instead." 10
|
||||
make_fx target-dangles "Use when doing the thing. Do not use for the other thing — use sibling-skill or no-such-skill instead." 10
|
||||
make_fx target-dangles-lone "Use when doing the thing. Do not use for the other thing — use no-such-lone-skill instead." 10
|
||||
make_fx target-dangles-notation "Use when doing the thing. Do not use for the other thing — use /no-such-notation-skill instead." 10
|
||||
make_fx target-attributive "Use when doing the thing. Use pre-commit hooks instead of ad-hoc scripts." 10
|
||||
# No boundary clause at all.
|
||||
make_fx no-boundary "Use when the user wants the thing done." 10
|
||||
# A missing references/ pointer, and a present one.
|
||||
make_fx ref-missing "$CLEAN" 10
|
||||
printf '\nIf the caller needs detail, read references/absent.md first.\n' >> "$FX/ref-missing/SKILL.md"
|
||||
make_fx ref-present "$CLEAN" 10
|
||||
printf '\nIf the caller needs detail, read references/there.md first.\n' >> "$FX/ref-present/SKILL.md"
|
||||
mkdir -p "$FX/ref-present/references"
|
||||
echo "detail" > "$FX/ref-present/references/there.md"
|
||||
# Gotchas, over each guideline.
|
||||
make_fx gotchas-many "$CLEAN" 0
|
||||
cat >> "$FX/gotchas-many/SKILL.md" <<'EOF'
|
||||
|
||||
## Gotchas
|
||||
|
||||
- one trap here
|
||||
- two trap here
|
||||
- three trap here
|
||||
- four trap here
|
||||
- five trap here
|
||||
- six trap here
|
||||
- seven trap here
|
||||
|
||||
## Notes
|
||||
|
||||
EOF
|
||||
python3 -c "print(' '.join(['word'] * 200))" >> "$FX/gotchas-many/SKILL.md"
|
||||
# Gotchas over the 25% body-fraction guideline. Prose, not list items, so the
|
||||
# entry guideline cannot fire and the two suggestions stay separable: 26 section
|
||||
# words in a 100-word body is one word past the threshold.
|
||||
make_fx gotchas-fraction "$CLEAN" 0
|
||||
{
|
||||
echo ""
|
||||
echo "## Gotchas"
|
||||
echo ""
|
||||
python3 -c "print(' '.join(['word'] * 26))"
|
||||
echo ""
|
||||
echo "## Notes"
|
||||
echo ""
|
||||
python3 -c "print(' '.join(['word'] * 70))"
|
||||
} >> "$FX/gotchas-fraction/SKILL.md"
|
||||
|
||||
# Empty description — the shape that used to exit 0 in silence.
|
||||
mkdir -p "$FX/empty-desc"
|
||||
printf -- '---\nname: empty-desc\ndescription:\nmodel: sonnet\n---\n\nDo the thing.\n' \
|
||||
> "$FX/empty-desc/SKILL.md"
|
||||
# Every boundary shape at once, so a divergence that only appears when several
|
||||
# findings fire together is not missed.
|
||||
make_fx combined "$(desc_of_length 401)" 901
|
||||
printf '\nIf the caller needs detail, read references/absent.md first.\n' >> "$FX/combined/SKILL.md"
|
||||
|
||||
# A skill with routing targets and NO authoring root above it — deliberately
|
||||
# OUTSIDE the fixture plugin tree. Both scripts must decline out loud, and both
|
||||
# must decline identically; "both declined" is only meaningful as agreement if
|
||||
# the declining path is exercised on purpose somewhere.
|
||||
ORPHAN_ROOT="$TMPDIR_T/orphan"
|
||||
mkdir -p "$ORPHAN_ROOT/no-universe"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: no-universe"
|
||||
echo "description: Use when doing the thing. Do not use for the other thing — use some-other-skill instead."
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "Do the thing."
|
||||
} > "$ORPHAN_ROOT/no-universe/SKILL.md"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The comparison
|
||||
# ---------------------------------------------------------------------------
|
||||
python3 - "$HOOK" "$SKILL_VALIDATE" "$REPO_ROOT" "$FX" "$ORPHAN_ROOT" <<'PYTHON'
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
hook, validate, repo_root, fixture_dir, orphan_dir = sys.argv[1:6]
|
||||
|
||||
passes = 0
|
||||
failures = 0
|
||||
|
||||
|
||||
def ok(msg):
|
||||
global passes
|
||||
passes += 1
|
||||
print(" PASS: %s" % msg)
|
||||
|
||||
|
||||
def bad(msg):
|
||||
global failures
|
||||
failures += 1
|
||||
print(" FAIL: %s" % msg)
|
||||
|
||||
|
||||
# Tier prefixes. The hook writes `ERROR: ` / `SUGGESTION: ` / `INFO: `; the
|
||||
# auditor writes `FAIL ` / `SUGGESTION ` / `INFO ` and additionally `PASS `
|
||||
# lines, which carry no finding and are dropped.
|
||||
TIERS = (
|
||||
('ERROR', ('ERROR:', 'FAIL ')),
|
||||
('SUGGESTION', ('SUGGESTION:', 'SUGGESTION ')),
|
||||
('INFO', ('INFO:', 'INFO ')),
|
||||
)
|
||||
|
||||
# Each rule turns a finding line into a canonical token. Wording differs between
|
||||
# the two scripts by design (one addresses a committer, the other an auditor), so
|
||||
# the tokens deliberately capture the MEASUREMENT and not the sentence.
|
||||
RULES = (
|
||||
('DESC_CHARS', re.compile(r'description is (\d+) char')),
|
||||
('BODY_WORDS', re.compile(r'body is (\d+) words')),
|
||||
('ROUTE', re.compile(r"routes to '([^']+)'")),
|
||||
('MISSING_REF', re.compile(r'points at (references/[^\s,]+)')),
|
||||
('GOTCHA_ENTRIES', re.compile(r'Gotchas section has (\d+) entries')),
|
||||
('GOTCHA_FRACTION', re.compile(r'Gotchas section is (\d+) of (\d+) body words')),
|
||||
('NO_BOUNDARY_CLAUSE', re.compile(r'(description has no boundary clause)')),
|
||||
('RESOLUTION_DECLINED', re.compile(r'(boundary-target resolution DID NOT RUN)')),
|
||||
('DESC_EMPTY', re.compile(r'(description field is missing or empty)')),
|
||||
)
|
||||
|
||||
|
||||
def verdict(output):
|
||||
"""The set of ADR-0020 findings in a script's output, tier included.
|
||||
|
||||
Lines that match no rule are dropped rather than compared: the two scripts
|
||||
legitimately check different things outside ADR-0020 (name/directory
|
||||
agreement, script executability, the 1024-char spec backstop, whole-file
|
||||
line and word ceilings), and forcing those into the comparison would report
|
||||
a difference that is not a disagreement.
|
||||
"""
|
||||
found = set()
|
||||
for raw in output.splitlines():
|
||||
line = raw.strip()
|
||||
tier = None
|
||||
for name, prefixes in TIERS:
|
||||
if any(line.startswith(p) for p in prefixes):
|
||||
tier = name
|
||||
break
|
||||
if tier is None:
|
||||
continue
|
||||
for token, pattern in RULES:
|
||||
match = pattern.search(line)
|
||||
if match:
|
||||
found.add((tier, token) + tuple(match.groups()))
|
||||
return found
|
||||
|
||||
|
||||
def run(cmd):
|
||||
proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||
return proc.returncode, proc.stdout.decode('utf-8', 'replace')
|
||||
|
||||
|
||||
def compare(label, skill_dir):
|
||||
skill_md = os.path.join(skill_dir, 'SKILL.md')
|
||||
hook_rc, hook_out = run(['bash', hook, skill_md])
|
||||
audit_rc, audit_out = run(['bash', validate, skill_dir])
|
||||
hook_v = verdict(hook_out)
|
||||
audit_v = verdict(audit_out)
|
||||
|
||||
problems = []
|
||||
only_hook = sorted(hook_v - audit_v)
|
||||
only_audit = sorted(audit_v - hook_v)
|
||||
if only_hook:
|
||||
problems.append('only the hook reported %s' % (only_hook,))
|
||||
if only_audit:
|
||||
problems.append('only skill-audit reported %s' % (only_audit,))
|
||||
|
||||
# Exit codes are compared on the ADR-0020 axis only: an ERROR-tier ADR-0020
|
||||
# finding must make BOTH scripts non-zero, and neither may be turned
|
||||
# non-zero by a SUGGESTION. The raw codes cannot be compared directly —
|
||||
# validate.sh also fails on checks the hook does not run at all.
|
||||
hook_err = any(t == 'ERROR' for t, *_ in hook_v)
|
||||
audit_err = any(t == 'ERROR' for t, *_ in audit_v)
|
||||
if hook_err and hook_rc == 0:
|
||||
problems.append('the hook reported an ADR-0020 ERROR but exited 0')
|
||||
if audit_err and audit_rc == 0:
|
||||
problems.append('skill-audit reported an ADR-0020 FAIL but exited 0')
|
||||
if not hook_err and hook_rc != 0 and not _non_adr_hook_error(hook_out):
|
||||
problems.append('the hook exited %d with no ADR-0020 ERROR and no spec-ceiling ERROR'
|
||||
% hook_rc)
|
||||
|
||||
if problems:
|
||||
bad('%s: %s' % (label, '; '.join(problems)))
|
||||
else:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _non_adr_hook_error(output):
|
||||
"""True if the hook failed on a spec ceiling rather than an ADR-0020 gate.
|
||||
|
||||
MAX_LINES / MAX_WORDS are the hook's other ERROR sources and are outside
|
||||
this comparison, so a non-zero exit explained by one of them is not a
|
||||
disagreement.
|
||||
"""
|
||||
return bool(re.search(r'ERROR: .*(-line ceiling|-word ceiling \(~5,000 tokens)', output))
|
||||
|
||||
|
||||
# --- The real corpus -------------------------------------------------------
|
||||
corpus = sorted(glob.glob(os.path.join(repo_root, 'plugins', '*', '.apm', 'skills', '*')))
|
||||
corpus = [d for d in corpus if os.path.isfile(os.path.join(d, 'SKILL.md'))]
|
||||
print("")
|
||||
print("--- the two scripts agree on every skill in the live corpus (%d files) ---" % len(corpus))
|
||||
if len(corpus) < 30:
|
||||
bad('only %d corpus skills were discovered — the glob is wrong, so this leg '
|
||||
'proves nothing' % len(corpus))
|
||||
else:
|
||||
ok('discovered %d corpus skills to compare' % len(corpus))
|
||||
agreed = 0
|
||||
for skill_dir in corpus:
|
||||
rel = os.path.relpath(skill_dir, repo_root)
|
||||
if compare(rel, skill_dir):
|
||||
agreed += 1
|
||||
if agreed == len(corpus):
|
||||
ok('all %d corpus skills produce identical ADR-0020 verdicts from both scripts' % agreed)
|
||||
|
||||
# --- Boundary fixtures -----------------------------------------------------
|
||||
fixtures = sorted(d for d in (glob.glob(os.path.join(fixture_dir, '*'))
|
||||
+ glob.glob(os.path.join(orphan_dir, '*')))
|
||||
if os.path.isfile(os.path.join(d, 'SKILL.md')))
|
||||
print("")
|
||||
print("--- the two scripts agree on every boundary fixture (%d files) ---" % len(fixtures))
|
||||
if len(fixtures) < 15:
|
||||
bad('only %d fixtures were built — the fixture set is incomplete, so the '
|
||||
'boundaries the corpus does not cover are untested' % len(fixtures))
|
||||
else:
|
||||
ok('built %d boundary fixtures to compare' % len(fixtures))
|
||||
fx_agreed = 0
|
||||
for skill_dir in fixtures:
|
||||
if compare(os.path.basename(skill_dir), skill_dir):
|
||||
fx_agreed += 1
|
||||
if fx_agreed == len(fixtures):
|
||||
ok('all %d boundary fixtures produce identical ADR-0020 verdicts from both scripts' % fx_agreed)
|
||||
|
||||
# --- The comparison must not be vacuous ------------------------------------
|
||||
# Everything above would also pass if verdict() extracted nothing at all. So the
|
||||
# fixtures are required to have produced findings across every axis this suite
|
||||
# claims to compare — if a rule stops matching (a reworded message, say), that is
|
||||
# a silent loss of coverage and it fails here instead.
|
||||
print("")
|
||||
print("--- the comparison actually extracted findings on every axis it claims to cover ---")
|
||||
seen_tokens = set()
|
||||
for skill_dir in fixtures:
|
||||
_, out = run(['bash', hook, os.path.join(skill_dir, 'SKILL.md')])
|
||||
for entry in verdict(out):
|
||||
seen_tokens.add(entry[1])
|
||||
_, out = run(['bash', validate, skill_dir])
|
||||
for entry in verdict(out):
|
||||
seen_tokens.add(entry[1])
|
||||
expected_tokens = {t for t, _ in RULES}
|
||||
missing = sorted(expected_tokens - seen_tokens)
|
||||
if missing:
|
||||
bad('no fixture produced a finding for %s — verdict() may no longer match '
|
||||
'those messages, and any disagreement on them would go unseen' % missing)
|
||||
else:
|
||||
ok('every one of the %d compared axes was exercised by at least one fixture'
|
||||
% len(expected_tokens))
|
||||
|
||||
print("")
|
||||
print("Results: %d passed, %d failed" % (passes, failures))
|
||||
sys.exit(1 if failures else 0)
|
||||
PYTHON
|
||||
286
tests/test-adr0020-frontmatter.sh
Executable file
286
tests/test-adr0020-frontmatter.sh
Executable file
@@ -0,0 +1,286 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for the two ways an ADR-0020 gate can be made to check NOTHING
|
||||
# while still exiting 0. Both were live defects, both were silent, and both sit
|
||||
# in the shared resolver block that all three scripts embed verbatim — so every
|
||||
# case below runs against all three.
|
||||
#
|
||||
# 1. THE FRONTMATTER BLOCKER. The frontmatter matcher used to be `^---\n`. A
|
||||
# UTF-8 BOM, a leading blank line, a trailing space after either marker, or
|
||||
# CRLF line endings all defeated it, and the miss was not reported: every
|
||||
# ADR-0020 check was skipped and the file passed. Measured at the time: a
|
||||
# 550-character description with a 1,000-word body exited 0 behind a BOM.
|
||||
# So this file asserts two complementary things — that each of those four
|
||||
# shapes is now TOLERATED (the findings actually fire), and that
|
||||
# frontmatter which genuinely cannot be parsed is a hard ERROR rather than
|
||||
# a quiet skip. A file that cannot be measured must never report green.
|
||||
#
|
||||
# 2. THE VALUELESS DESCRIPTION. `description:` with no value, followed by
|
||||
# another key, let a line regex's `\s*` cross the newline and capture the
|
||||
# NEXT key. The value then looked present (so "missing or empty" never
|
||||
# fired) and was empty once folded (so every ADR-0020 gate early-returned).
|
||||
# An agent file with one exited 0 with zero output through a BLOCKING
|
||||
# pre-push gate. All five spellings of "no value" are pinned here.
|
||||
#
|
||||
# Both fixtures carry an over-ceiling description AND an over-ceiling body on
|
||||
# purpose: asserting a non-zero exit alone would be satisfied by the "cannot
|
||||
# parse" error itself, so the tolerated shapes are asserted on the CONTENT of
|
||||
# the findings, not on the exit code.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
|
||||
TMPDIR_T="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||
|
||||
DESC_CHARS=450
|
||||
BODY_WORDS=1000
|
||||
|
||||
# write_fixture <kind> <path> <name> — one generator for both file shapes.
|
||||
#
|
||||
# Byte-level control is the point: BOM placement, line endings and trailing
|
||||
# whitespace are exactly what is under test, so the file is emitted in binary
|
||||
# mode rather than through a shell heredoc that would normalise them.
|
||||
write_fixture() {
|
||||
python3 - "$1" "$2" "$3" "$DESC_CHARS" "$BODY_WORDS" <<'PY'
|
||||
import sys
|
||||
|
||||
kind, path, name, desc_chars, body_words = sys.argv[1:6]
|
||||
desc = 'x' * int(desc_chars)
|
||||
body = ' '.join(['word'] * int(body_words))
|
||||
|
||||
# The default, well-formed shape. Variants below mutate it.
|
||||
open_marker = '---'
|
||||
close_marker = '---'
|
||||
prefix = ''
|
||||
newline = '\n'
|
||||
fm_lines = ['name: ' + name, 'description: ' + desc]
|
||||
|
||||
if kind == 'plain':
|
||||
pass
|
||||
elif kind == 'bom':
|
||||
prefix = ''
|
||||
elif kind == 'leading-blanks':
|
||||
prefix = '\n\n \n'
|
||||
elif kind == 'trailing-ws':
|
||||
open_marker = '--- '
|
||||
close_marker = '---\t '
|
||||
elif kind == 'crlf':
|
||||
newline = '\r\n'
|
||||
elif kind == 'no-close':
|
||||
close_marker = None
|
||||
elif kind == 'yaml-list':
|
||||
fm_lines = ['- one', '- two']
|
||||
elif kind == 'yaml-string':
|
||||
fm_lines = ['just a bare scalar, not a mapping']
|
||||
elif kind == 'yaml-none':
|
||||
fm_lines = []
|
||||
elif kind == 'yaml-malformed':
|
||||
fm_lines = ['name: ' + name, 'description: "unterminated', 'tabs:\t- a']
|
||||
elif kind == 'desc-no-value':
|
||||
# The exact shape that exited 0 with zero output: a line regex's `\s*`
|
||||
# crosses the newline and captures `model: sonnet` as the description.
|
||||
fm_lines = ['name: ' + name, 'description:', 'model: sonnet']
|
||||
elif kind == 'desc-null':
|
||||
fm_lines = ['name: ' + name, 'description: null']
|
||||
elif kind == 'desc-single-quoted-empty':
|
||||
fm_lines = ['name: ' + name, "description: ''"]
|
||||
elif kind == 'desc-double-quoted-empty':
|
||||
fm_lines = ['name: ' + name, 'description: ""']
|
||||
elif kind == 'desc-empty-fold':
|
||||
fm_lines = ['name: ' + name, 'description: >']
|
||||
else:
|
||||
raise SystemExit('unknown fixture kind: %s' % kind)
|
||||
|
||||
parts = [prefix, open_marker, newline]
|
||||
for line in fm_lines:
|
||||
parts.append(line)
|
||||
parts.append(newline)
|
||||
if close_marker is not None:
|
||||
parts.append(close_marker)
|
||||
parts.append(newline)
|
||||
parts.append(newline)
|
||||
parts.append(body)
|
||||
parts.append(newline)
|
||||
|
||||
with open(path, 'wb') as fh:
|
||||
fh.write(''.join(parts).encode('utf-8'))
|
||||
PY
|
||||
}
|
||||
|
||||
# Builds all three subjects for one fixture kind and echoes nothing; the paths
|
||||
# are fixed by convention so the probes below can find them.
|
||||
#
|
||||
# skill-audit takes a DIRECTORY (SKILL.md inside it, name matching the dir);
|
||||
# agent-audit takes a FILE inside an apm package. The hook takes the SKILL.md
|
||||
# directly, so it and skill-audit share one file.
|
||||
build_subjects() {
|
||||
local kind="$1" base="$TMPDIR_T/$1"
|
||||
rm -rf "$base"
|
||||
mkdir -p "$base/skill/my-skill" "$base/agent/.apm/agents"
|
||||
cat > "$base/agent/apm.yml" <<'EOF'
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
write_fixture "$kind" "$base/skill/my-skill/SKILL.md" my-skill
|
||||
write_fixture "$kind" "$base/agent/.apm/agents/my-agent.agent.md" my-agent
|
||||
}
|
||||
|
||||
# probe_all <label> <kind> <needle>... — runs all three scripts over the fixture
|
||||
# and requires every one of them to exit non-zero AND report every needle. One
|
||||
# assertion per script would let two of them drift apart while the suite stayed
|
||||
# green; the whole point of the shared resolver block is that they cannot.
|
||||
#
|
||||
# A needle written `@skills:<text>` is asserted for the hook and skill-audit but
|
||||
# NOT for agent-audit. There is exactly one such needle in this file — the body
|
||||
# word ceiling — and the exemption is the ADR, not a workaround: ADR-0020 gives
|
||||
# agents the description gates and deliberately NO body word gate, because an
|
||||
# agent body becomes the system prompt of a fresh context rather than competing
|
||||
# with the caller's live conversation. Demanding a body finding from agent-audit
|
||||
# would be demanding the ADR be contradicted.
|
||||
probe_all() {
|
||||
local label="$1" kind="$2"
|
||||
shift 2
|
||||
local base="$TMPDIR_T/$kind"
|
||||
local -a targets=(
|
||||
"hook|$HOOK|$base/skill/my-skill/SKILL.md"
|
||||
"skill-audit|$SKILL_VALIDATE|$base/skill/my-skill"
|
||||
"agent-audit|$AGENT_VALIDATE|$base/agent/.apm/agents/my-agent.agent.md"
|
||||
)
|
||||
local problems=""
|
||||
for target in "${targets[@]}"; do
|
||||
local who="${target%%|*}" rest="${target#*|}"
|
||||
local script="${rest%%|*}" arg="${rest#*|}"
|
||||
local out status=0
|
||||
set +e
|
||||
out="$(bash "$script" "$arg" 2>&1)"
|
||||
status=$?
|
||||
set -e
|
||||
if [[ $status -eq 0 ]]; then
|
||||
problems="$problems [$who exited 0: ${out:-<no output>}]"
|
||||
continue
|
||||
fi
|
||||
for needle in "$@"; do
|
||||
if [[ "$needle" == @skills:* ]]; then
|
||||
# Spelled as a full `if`, not `[[ ... ]] && continue`. Under `set -e` the
|
||||
# short form's exit status is the test's when it is false, and relying on
|
||||
# the &&-list exemption to keep that from aborting the run is a footgun
|
||||
# one edit away from biting.
|
||||
if [[ "$who" == agent-audit ]]; then
|
||||
continue
|
||||
fi
|
||||
needle="${needle#@skills:}"
|
||||
fi
|
||||
if [[ "$out" != *"$needle"* ]]; then
|
||||
problems="$problems [$who never said '$needle': $out]"
|
||||
fi
|
||||
done
|
||||
done
|
||||
if [[ -z "$problems" ]]; then
|
||||
pass "$label"
|
||||
else
|
||||
fail "$label —$problems"
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1a. Tolerated frontmatter shapes — the gates must RUN, not merely not-pass
|
||||
# ---------------------------------------------------------------------------
|
||||
# The needles are the FINDINGS, not the exit code. A script that rejected the BOM
|
||||
# outright would exit non-zero too, and would still be skipping every ADR-0020
|
||||
# measurement — which is the defect, one error message later.
|
||||
echo ""
|
||||
echo "--- a BOM, leading blanks, trailing marker whitespace and CRLF are all tolerated, and the gates still fire ---"
|
||||
for kind in plain bom leading-blanks trailing-ws crlf; do
|
||||
build_subjects "$kind"
|
||||
done
|
||||
probe_all "control: a well-formed over-ceiling file fails on BOTH the description and the body" \
|
||||
plain "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||
probe_all "a UTF-8 BOM does not hide an over-ceiling description or body" \
|
||||
bom "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||
probe_all "leading blank lines before the opening --- do not hide the findings" \
|
||||
leading-blanks "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||
probe_all "trailing whitespace after either --- marker does not hide the findings" \
|
||||
trailing-ws "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||
# Note on what this last one can and cannot detect. read_text() opens the file in
|
||||
# TEXT mode, so Python's universal-newline translation turns \r\n into \n before
|
||||
# the frontmatter matcher ever sees it — verified by mutation: reverting
|
||||
# FRONTMATTER_RE to the old `^---\n(.*?)\n---` breaks the leading-blanks and
|
||||
# trailing-whitespace cases above but NOT this one. So this case pins the
|
||||
# end-to-end behaviour (a CRLF file is measured, not skipped) rather than the
|
||||
# `\r?\n` alternations in the regex, and it would catch a future switch to binary
|
||||
# reads or to a newline='' open. Kept for that reason, and labelled so nobody
|
||||
# reads it as covering more than it does.
|
||||
probe_all "CRLF line endings do not hide the findings" \
|
||||
crlf "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1b. Unparseable frontmatter is a hard ERROR, never a quiet skip
|
||||
# ---------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "--- genuinely unparseable frontmatter exits non-zero with a message, rather than passing quietly ---"
|
||||
for kind in no-close yaml-list yaml-string yaml-none yaml-malformed; do
|
||||
build_subjects "$kind"
|
||||
done
|
||||
probe_all "frontmatter with no closing --- is reported, not skipped" \
|
||||
no-close "frontmatter"
|
||||
probe_all "frontmatter that parses to a LIST is reported, not skipped" \
|
||||
yaml-list "frontmatter"
|
||||
probe_all "frontmatter that parses to a STRING is reported, not skipped" \
|
||||
yaml-string "frontmatter"
|
||||
probe_all "frontmatter that parses to None (empty block) is reported, not skipped" \
|
||||
yaml-none "frontmatter"
|
||||
probe_all "malformed YAML in the frontmatter is reported, not skipped" \
|
||||
yaml-malformed "frontmatter"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. A valueless description is a hard FAIL in all three scripts
|
||||
# ---------------------------------------------------------------------------
|
||||
# All five spellings mean the same thing to a YAML parser — an empty value — and
|
||||
# all five have to be decided on the FOLDED value rather than on a line regex.
|
||||
# `description:` followed by `model: sonnet` is the one that shipped: it made the
|
||||
# value look present, skipped the "missing or empty" failure, and then
|
||||
# early-returned out of every ADR-0020 gate on the genuinely empty folded value.
|
||||
echo ""
|
||||
echo "--- every spelling of a valueless description hard-FAILs in all three scripts ---"
|
||||
for kind in desc-no-value desc-null desc-single-quoted-empty desc-double-quoted-empty desc-empty-fold; do
|
||||
build_subjects "$kind"
|
||||
done
|
||||
probe_all "'description:' with no value (next key not captured as the value) FAILs" \
|
||||
desc-no-value "description field is missing or empty"
|
||||
probe_all "'description: null' FAILs" \
|
||||
desc-null "description field is missing or empty"
|
||||
probe_all "\"description: ''\" FAILs" \
|
||||
desc-single-quoted-empty "description field is missing or empty"
|
||||
probe_all "'description: \"\"' FAILs" \
|
||||
desc-double-quoted-empty "description field is missing or empty"
|
||||
probe_all "'description: >' with nothing folded under it FAILs" \
|
||||
desc-empty-fold "description field is missing or empty"
|
||||
|
||||
# The specific regression, spelled out: the valueless-description agent file must
|
||||
# not merely fail — it must not be SILENT. Zero output on a blocking gate is what
|
||||
# made this un-diagnosable, so the output is asserted non-empty independently.
|
||||
echo ""
|
||||
echo "--- the valueless-description agent file produces output, not silence ---"
|
||||
build_subjects desc-no-value
|
||||
set +e
|
||||
SILENT_OUT="$(bash "$AGENT_VALIDATE" "$TMPDIR_T/desc-no-value/agent/.apm/agents/my-agent.agent.md" 2>&1)"
|
||||
SILENT_RC=$?
|
||||
set -e
|
||||
if [[ $SILENT_RC -ne 0 && -n "$SILENT_OUT" ]]; then
|
||||
pass "agent-audit reports a valueless description rather than exiting 0 with zero output"
|
||||
else
|
||||
fail "agent-audit exited $SILENT_RC with output '${SILENT_OUT:-<empty>}' — the original defect was exit 0 and total silence on a blocking pre-push gate"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
409
tests/test-adr0020-targets.sh
Executable file
409
tests/test-adr0020-targets.sh
Executable file
@@ -0,0 +1,409 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression test for the two properties of ADR-0020 boundary-target resolution
|
||||
# that decide whether the gate can be trusted at all.
|
||||
#
|
||||
# 1. MACHINE INDEPENDENCE. The resolution universe is derived by walking up
|
||||
# FROM THE TARGET FILE to an authoring root, and when one is found the
|
||||
# deployed .claude/ and .agents/ trees are deliberately NOT consulted. Those
|
||||
# trees are `apm install` output — gitignored, and present only on a machine
|
||||
# that has run it. Four cross-plugin targets in this repo resolved through
|
||||
# .claude/skills/ alone, so the same commit measured 2 dangling targets on a
|
||||
# developer machine and 6 on a fresh clone. A gate shipping hot with no
|
||||
# baseline cannot give two answers, so this file asserts the verdict is
|
||||
# identical with and without a deployed tree — on a synthetic fixture AND on
|
||||
# the real 39-skill corpus.
|
||||
#
|
||||
# 2. THE BARE-TARGET GRAMMAR RULE. A hyphenated token used as a compound
|
||||
# MODIFIER ("pre-commit hooks", "pull-request template") is prose, not a
|
||||
# route; a terminal one is a real target. Getting this wrong in either
|
||||
# direction is fatal: firing on prose makes the gate untrustworthy and it
|
||||
# gets turned off, while suppressing too much deletes the only two true
|
||||
# positives the corpus has. Both live true positives are BARE, which is why
|
||||
# the rule keys on the FOLLOWER TOKEN rather than on marking, and why they
|
||||
# are pinned by name below — a future false-positive fix must not be able to
|
||||
# quietly take them with it.
|
||||
#
|
||||
# 3. IN-SENTENCE CORROBORATION. Terminal position alone is not evidence of a
|
||||
# route: "run `pre-commit` instead", "see `commit-msg`", "use the clean-up
|
||||
# instead" and "run unit-tests" are all terminal, all prose, and all were
|
||||
# hard FAILs with no suppression mechanism anywhere in the gate. A
|
||||
# prose-form target therefore blocks only when its own sentence names
|
||||
# another target that RESOLVES; otherwise it is reported at SUGGESTION tier
|
||||
# and the commit proceeds. Route NOTATION (`/name`, `-> name`) is exempt
|
||||
# and always blocks. Both halves are asserted below: the prose class must
|
||||
# report-not-block, and the notation and corroborated forms must still
|
||||
# ERROR, or the fix would have eaten the gate rather than narrowed it.
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||
PASS=0
|
||||
FAIL=0
|
||||
|
||||
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
|
||||
TMPDIR_T="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||
|
||||
# write_skill <skill-dir> <name> <desc>
|
||||
write_skill() {
|
||||
mkdir -p "$1"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: $2"
|
||||
echo "description: $3"
|
||||
echo "---"
|
||||
echo ""
|
||||
echo "Do the thing."
|
||||
} > "$1/SKILL.md"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Machine independence — synthetic fixture
|
||||
# ---------------------------------------------------------------------------
|
||||
# Two trees, identical except that one also carries a deployed .claude/ tree
|
||||
# holding a skill and an agent that exist NOWHERE in plugins/. The subject routes
|
||||
# to one name that lives in a sibling plugin (must resolve in both) and one that
|
||||
# lives only in .claude/ (must DANGLE in both — an authoring root exists, so the
|
||||
# deployed tree is not part of the universe).
|
||||
#
|
||||
# If the deployed tree were consulted, the second target would resolve on the
|
||||
# machine that has run `apm install` and dangle on a fresh clone. That is the
|
||||
# 2-vs-6 defect exactly, at fixture scale.
|
||||
echo ""
|
||||
echo "--- the same file gets the same verdict with and without a deployed .claude/ tree ---"
|
||||
build_tree() {
|
||||
local root="$1"
|
||||
write_skill "$root/plugins/other-plugin/.apm/skills/cross-plugin-skill" cross-plugin-skill \
|
||||
"Use when doing the other thing. Do not use for anything else."
|
||||
write_skill "$root/plugins/subject-plugin/.apm/skills/my-skill" my-skill \
|
||||
"Use when doing the thing. Do not use for the other thing — use cross-plugin-skill or deployed-only-skill instead."
|
||||
}
|
||||
build_tree "$TMPDIR_T/no-claude"
|
||||
build_tree "$TMPDIR_T/with-claude"
|
||||
# The deployed tree, present only in the second root. Both a skill and an agent,
|
||||
# because both are valid routing targets and both would leak.
|
||||
mkdir -p "$TMPDIR_T/with-claude/.claude/skills/deployed-only-skill" \
|
||||
"$TMPDIR_T/with-claude/.claude/agents"
|
||||
: > "$TMPDIR_T/with-claude/.claude/agents/deployed-only-agent.md"
|
||||
|
||||
run_subject() {
|
||||
local root="$1" out
|
||||
set +e
|
||||
out="$(bash "$HOOK" "$root/plugins/subject-plugin/.apm/skills/my-skill/SKILL.md" 2>&1)"
|
||||
set -e
|
||||
# Normalise the tree root out of the paths so the two runs are comparable.
|
||||
printf '%s\n' "$out" | sed "s#$root#<ROOT>#g"
|
||||
}
|
||||
NO_CLAUDE_OUT="$(run_subject "$TMPDIR_T/no-claude")"
|
||||
WITH_CLAUDE_OUT="$(run_subject "$TMPDIR_T/with-claude")"
|
||||
|
||||
if [[ "$NO_CLAUDE_OUT" == "$WITH_CLAUDE_OUT" ]]; then
|
||||
pass "identical output with and without a deployed .claude/ tree"
|
||||
else
|
||||
fail "the deployed .claude/ tree changed the verdict — without: [$NO_CLAUDE_OUT] with: [$WITH_CLAUDE_OUT]"
|
||||
fi
|
||||
# Identical-but-wrong is still possible (both could resolve everything, or
|
||||
# neither could resolve anything), so the CONTENT is asserted too: the
|
||||
# sibling-plugin name must resolve and the deployed-only name must not.
|
||||
if [[ "$WITH_CLAUDE_OUT" == *"routes to 'deployed-only-skill'"* ]]; then
|
||||
pass "a name that exists only in .claude/ still dangles when an authoring root is present"
|
||||
else
|
||||
fail "the deployed-only target did not dangle — the deployed tree is being read into the universe: $WITH_CLAUDE_OUT"
|
||||
fi
|
||||
if [[ "$WITH_CLAUDE_OUT" != *"routes to 'cross-plugin-skill'"* ]]; then
|
||||
pass "a name in a SIBLING PLUGIN resolves, so the comparison above is not 'nothing resolves'"
|
||||
else
|
||||
fail "the sibling-plugin target dangled — the monorepo universe is not being built: $WITH_CLAUDE_OUT"
|
||||
fi
|
||||
|
||||
# The other half of the rule: with NO authoring root, deployed trees ARE the
|
||||
# universe. That is the consumer case, and without this the rule above could be
|
||||
# implemented as "never read .claude/", which would leave consumers with no
|
||||
# resolution at all.
|
||||
echo ""
|
||||
echo "--- with no authoring root, a deployed .claude/ tree IS the universe ---"
|
||||
CONSUMER="$TMPDIR_T/consumer"
|
||||
mkdir -p "$CONSUMER/.claude/skills/deployed-only-skill"
|
||||
write_skill "$CONSUMER/.claude/skills/my-skill" my-skill \
|
||||
"Use when doing the thing. Do not use for the other thing — use deployed-only-skill instead."
|
||||
set +e
|
||||
CONSUMER_OUT="$(bash "$HOOK" "$CONSUMER/.claude/skills/my-skill/SKILL.md" 2>&1)"
|
||||
CONSUMER_RC=$?
|
||||
set -e
|
||||
if [[ $CONSUMER_RC -eq 0 && "$CONSUMER_OUT" != *"routes to"* && "$CONSUMER_OUT" != *"DID NOT RUN"* ]]; then
|
||||
pass "a sibling in a deployed .claude/skills/ tree resolves when there is no authoring root"
|
||||
else
|
||||
fail "the consumer path did not resolve through the deployed tree (exit $CONSUMER_RC): ${CONSUMER_OUT:-<empty>}"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1b. Machine independence — the real corpus
|
||||
# ---------------------------------------------------------------------------
|
||||
# The fixture above proves the rule; this proves it at the scale where it broke.
|
||||
#
|
||||
# The A/B is built rather than borrowed. plugins/ is copied TWICE: once bare (a
|
||||
# fresh clone), and once with a synthetic .claude/skills/ tree deployed beside it
|
||||
# holding a directory for every name the corpus currently reports as dangling. If
|
||||
# deployed trees leaked back into the universe, the second copy would resolve
|
||||
# those names and report an empty dangling set while the first reported two —
|
||||
# 2-vs-6, reproduced deterministically.
|
||||
#
|
||||
# Deliberately NOT keyed on whether THIS machine has run `apm install`. Doing that
|
||||
# would make the suite fail on a fresh clone (where there is no .claude/ to
|
||||
# contrast against) — a test of machine independence that is itself
|
||||
# machine-dependent. The live tree is still compared, as a third data point, but
|
||||
# nothing here requires it to be in either state.
|
||||
echo ""
|
||||
echo "--- the real corpus reports the same dangling targets with and without a deployed tree ---"
|
||||
dangling_set() {
|
||||
local -a files=()
|
||||
local f
|
||||
# Collected with a `while read` loop, not `mapfile`: macOS ships bash 3.2,
|
||||
# which has no `mapfile`, and tests/test-vale-wrap.sh scans tests/*.sh for
|
||||
# exactly that hazard. `find` rather than a glob so both roots walk identically.
|
||||
while IFS= read -r f; do
|
||||
files+=("$f")
|
||||
done < <(find "$1" -path '*/.apm/skills/*/SKILL.md' | sort)
|
||||
if [[ ${#files[@]} -eq 0 ]]; then
|
||||
echo "NO-FILES-FOUND"
|
||||
return
|
||||
fi
|
||||
set +e
|
||||
bash "$HOOK" ${files[@]+"${files[@]}"} 2>&1 \
|
||||
| grep -oE "routes to '[^']+'" \
|
||||
| sed "s/routes to '//; s/'//" \
|
||||
| sort -u
|
||||
set -e
|
||||
}
|
||||
LIVE_DANGLING="$(dangling_set "$REPO_ROOT/plugins")"
|
||||
|
||||
FRESH_ROOT="$TMPDIR_T/fresh-clone"
|
||||
mkdir -p "$FRESH_ROOT"
|
||||
cp -R "$REPO_ROOT/plugins" "$FRESH_ROOT/plugins"
|
||||
[[ -f "$REPO_ROOT/apm.yml" ]] && cp "$REPO_ROOT/apm.yml" "$FRESH_ROOT/apm.yml"
|
||||
FRESH_DANGLING="$(dangling_set "$FRESH_ROOT/plugins")"
|
||||
|
||||
DEPLOYED_ROOT="$TMPDIR_T/deployed-clone"
|
||||
mkdir -p "$DEPLOYED_ROOT/.claude/skills" "$DEPLOYED_ROOT/.claude/agents"
|
||||
cp -R "$REPO_ROOT/plugins" "$DEPLOYED_ROOT/plugins"
|
||||
[[ -f "$REPO_ROOT/apm.yml" ]] && cp "$REPO_ROOT/apm.yml" "$DEPLOYED_ROOT/apm.yml"
|
||||
# Deploy exactly the names that currently dangle. That is the strongest possible
|
||||
# bait: if the deployed tree were consulted, every one of them would resolve and
|
||||
# the dangling set would collapse to empty.
|
||||
DEPLOY_COUNT=0
|
||||
while IFS= read -r name; do
|
||||
[[ -n "$name" ]] || continue
|
||||
mkdir -p "$DEPLOYED_ROOT/.claude/skills/$name"
|
||||
DEPLOY_COUNT=$((DEPLOY_COUNT + 1))
|
||||
done <<< "$FRESH_DANGLING"
|
||||
DEPLOYED_DANGLING="$(dangling_set "$DEPLOYED_ROOT/plugins")"
|
||||
|
||||
if [[ "$DEPLOY_COUNT" -gt 0 ]]; then
|
||||
pass "precondition: $DEPLOY_COUNT dangling name(s) deployed into the contrast tree's .claude/skills/, so the A/B has something to distinguish"
|
||||
else
|
||||
fail "no dangling names to deploy — the corpus reports none, so this A/B distinguishes nothing. Deploy a known-absent name explicitly instead of deriving one."
|
||||
fi
|
||||
if [[ ! -d "$FRESH_ROOT/.claude" && ! -d "$FRESH_ROOT/.agents" ]]; then
|
||||
pass "precondition: the fresh-clone copy has no deployed tree of its own"
|
||||
else
|
||||
fail "the fresh-clone copy picked up a deployed tree — it is not a fresh-clone fixture"
|
||||
fi
|
||||
if [[ "$FRESH_DANGLING" == "$DEPLOYED_DANGLING" ]]; then
|
||||
pass "deploying every dangling name into .claude/skills/ changes nothing: $(echo "$FRESH_DANGLING" | tr '\n' ' ')"
|
||||
else
|
||||
fail "the corpus verdict depends on whether apm install has been run — fresh clone: [$(echo "$FRESH_DANGLING" | tr '\n' ' ')] with a deployed tree: [$(echo "$DEPLOYED_DANGLING" | tr '\n' ' ')]"
|
||||
fi
|
||||
# Third data point: whatever state THIS machine happens to be in, the live tree
|
||||
# must agree with a bare copy of the same plugins/. No precondition on that state
|
||||
# — see the section header.
|
||||
if [[ "$LIVE_DANGLING" == "$FRESH_DANGLING" ]]; then
|
||||
pass "the live tree agrees with a bare copy (this machine $( [[ -d "$REPO_ROOT/.claude/skills" ]] && echo "HAS" || echo "has no" ) deployed .claude/skills/ tree)"
|
||||
else
|
||||
fail "the live tree disagrees with a bare copy of the same plugins/ — live: [$(echo "$LIVE_DANGLING" | tr '\n' ' ')] fresh clone: [$(echo "$FRESH_DANGLING" | tr '\n' ' ')]"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1c. The two live true positives, pinned by name
|
||||
# ---------------------------------------------------------------------------
|
||||
# ADR-0020 records these as real broken routing targets and splits fixing them
|
||||
# into Gitea issue #100. Until that lands they are the ONLY evidence the dangling
|
||||
# check finds anything at all in real prose, so they are asserted as an exact set
|
||||
# rather than a "contains" — a false-positive fix that suppressed one of them
|
||||
# would otherwise land green.
|
||||
#
|
||||
# `gitea-labels` is the subtler of the two and is worth keeping: it is not
|
||||
# written anywhere as `gitea-labels`. gitea-issues' description says "Composes
|
||||
# `gitea-labels-\n milestones`" in a `>`-folded scalar, and the fold joins the
|
||||
# lines into "gitea-labels- milestones" — the trailing hyphen is what keeps the
|
||||
# token terminal and therefore danglable.
|
||||
#
|
||||
# WHEN ISSUE #100 IS FIXED: update EXPECTED_DANGLING to match. Do not delete the
|
||||
# assertion — an empty expected set is fine and still pins that no NEW dangling
|
||||
# target appeared.
|
||||
echo ""
|
||||
echo "--- the two live dangling targets in the corpus are exactly the two ADR-0020 records ---"
|
||||
EXPECTED_DANGLING="$(printf '%s\n' gitea-labels neuledge-context)"
|
||||
if [[ "$LIVE_DANGLING" == "$EXPECTED_DANGLING" ]]; then
|
||||
pass "the corpus dangling set is exactly {gitea-labels, neuledge-context}"
|
||||
else
|
||||
fail "the corpus dangling set changed — expected [$(echo "$EXPECTED_DANGLING" | tr '\n' ' ')], got [$(echo "$LIVE_DANGLING" | tr '\n' ' ')]. If a retrofit fixed one, update EXPECTED_DANGLING; if a false-positive fix silently deleted one, that is the regression this asserts."
|
||||
fi
|
||||
for probe in \
|
||||
"plugins/bin/.apm/skills/research/SKILL.md:neuledge-context" \
|
||||
"plugins/gitea/.apm/skills/gitea-issues/SKILL.md:gitea-labels"; do
|
||||
probe_file="$REPO_ROOT/${probe%%:*}"
|
||||
probe_name="${probe##*:}"
|
||||
if [[ ! -f "$probe_file" ]]; then
|
||||
fail "the true-positive fixture ${probe%%:*} no longer exists — this pin has become vacuous"
|
||||
continue
|
||||
fi
|
||||
set +e
|
||||
probe_out="$(bash "$HOOK" "$probe_file" 2>&1)"
|
||||
set -e
|
||||
if [[ "$probe_out" == *"routes to '$probe_name'"* ]]; then
|
||||
pass "detects the dangling '$probe_name' target in ${probe%%:*}"
|
||||
else
|
||||
fail "did not detect the dangling '$probe_name' target in ${probe%%:*} — a false-positive fix has taken a true positive with it: $probe_out"
|
||||
fi
|
||||
done
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The bare-target grammar rule
|
||||
# ---------------------------------------------------------------------------
|
||||
# Every fixture is built inside a real plugin tree. In a bare temp directory the
|
||||
# resolver would decline ("DID NOT RUN") and every must-not-error case would pass
|
||||
# vacuously, proving nothing about extraction.
|
||||
echo ""
|
||||
echo "--- attributive compound modifiers are prose, not routing targets ---"
|
||||
GRAMMAR_ROOT="$TMPDIR_T/grammar"
|
||||
write_skill "$GRAMMAR_ROOT/plugins/p/.apm/skills/sibling-skill" sibling-skill \
|
||||
"Use when doing the other thing. Do not use for anything else."
|
||||
|
||||
# grammar_case <slug> <expect: silent|errors> <needle> <description>
|
||||
grammar_case() {
|
||||
local slug="$1" mode="$2" needle="$3" desc="$4" out status=0
|
||||
write_skill "$GRAMMAR_ROOT/plugins/p/.apm/skills/$slug" "$slug" "$desc"
|
||||
set +e
|
||||
out="$(bash "$HOOK" "$GRAMMAR_ROOT/plugins/p/.apm/skills/$slug/SKILL.md" 2>&1)"
|
||||
status=$?
|
||||
set -e
|
||||
if [[ "$out" == *"DID NOT RUN"* ]]; then
|
||||
fail "\"$desc\" — the resolver declined, so this case asserts nothing about extraction: $out"
|
||||
return
|
||||
fi
|
||||
case "$mode" in
|
||||
silent)
|
||||
if [[ $status -eq 0 && -z "$out" ]]; then
|
||||
pass "not a dangling target: \"$desc\""
|
||||
else
|
||||
fail "\"$desc\" (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
errors)
|
||||
if [[ $status -ne 0 && "$out" == *"$needle"* ]]; then
|
||||
pass "still a dangling target: \"$desc\""
|
||||
else
|
||||
fail "\"$desc\" should have ERRORed with $needle (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
suggests)
|
||||
# Reported, not blocking. Both halves matter: an ERROR here would be the
|
||||
# unsuppressable false positive this tier exists to remove, and silence
|
||||
# would mean the gate stopped noticing the target at all.
|
||||
if [[ $status -eq 0 && "$out" == *"SUGGESTION"*"$needle"* && "$out" != *"ERROR"* ]]; then
|
||||
pass "reported but not blocking: \"$desc\""
|
||||
else
|
||||
fail "\"$desc\" should have exited 0 with a SUGGESTION naming $needle (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# The four phrasings that were hard dangling FAILs with no suppression. All four
|
||||
# are lifted from real descriptions in this corpus.
|
||||
grammar_case fp-precommit-hooks silent "" \
|
||||
"Use when running the linter. Use pre-commit hooks instead of ad-hoc scripts."
|
||||
grammar_case fp-pull-request silent "" \
|
||||
"Use when opening changes. Invoke the pull-request template instead of writing one by hand."
|
||||
grammar_case fp-conventional silent "" \
|
||||
"Use when writing history. Use conventional-commits formatting rather than free-form messages."
|
||||
grammar_case fp-prepush-backticked silent "" \
|
||||
"Use when checking a branch. Do not use for local edits — run the \`pre-push\` hooks instead."
|
||||
|
||||
echo ""
|
||||
echo "--- a lone unresolvable token in terminal position is REPORTED, not blocking ---"
|
||||
# The class this tier was added for. Every one of these is grammatically
|
||||
# identical to a real broken route — "route verb + name + terminal" is also how
|
||||
# prose cites a hook, a linter, a file format or an English compound — and every
|
||||
# one of them was a hard FAIL with no suppression mechanism anywhere in the gate.
|
||||
# The skills most exposed are exactly the ones the ADR-0020 retrofit sends
|
||||
# authors back to rewrite first: pc-run, pc-author, vale-run, vale-config and the
|
||||
# apm-* family are all ABOUT hyphenated tools.
|
||||
grammar_case fp-precommit-terminal suggests "routes to 'pre-commit'" \
|
||||
"Use when running the linter. Do not use for running hooks — run \`pre-commit\` instead."
|
||||
grammar_case fp-commit-msg suggests "routes to 'commit-msg'" \
|
||||
"Use when writing history. Do not use for the commit message — see \`commit-msg\`."
|
||||
grammar_case fp-type-check suggests "routes to 'type-check'" \
|
||||
"Use when compiling. Do not use for type errors — run \`type-check\` first."
|
||||
grammar_case fp-semantic-release suggests "routes to 'semantic-release'" \
|
||||
"Use when tagging a version. Instead, use \`semantic-release\`."
|
||||
# Single-word tool names are the same defect: `eslint` in terminal position hit
|
||||
# the marked-target path and hard-FAILed just as `pre-commit` did.
|
||||
grammar_case fp-eslint suggests "routes to 'eslint'" \
|
||||
"Use when linting JS. Do not use for style — run \`eslint\` instead."
|
||||
# Bare English compounds, which the backtick path never sees at all.
|
||||
grammar_case fp-clean-up suggests "routes to 'clean-up'" \
|
||||
"Use when doing the thing. Do not use for the old flow — use the clean-up instead."
|
||||
grammar_case fp-built-in suggests "routes to 'built-in'" \
|
||||
"Use when doing the thing. Do not use for the custom path — Instead, prefer the built-in."
|
||||
grammar_case fp-write-up suggests "routes to 'write-up'" \
|
||||
"Use when doing the thing. Do not use for the summary — see the write-up."
|
||||
grammar_case fp-front-end suggests "routes to 'front-end'" \
|
||||
"Use when doing the thing. Do not use for the API layer — use the front-end."
|
||||
grammar_case fp-unit-tests suggests "routes to 'unit-tests'" \
|
||||
"Use when testing. Do not run end-to-end, run unit-tests."
|
||||
|
||||
echo ""
|
||||
echo "--- terminal targets that resolve to nothing still ERROR ---"
|
||||
# The controls. Without them the cases above are satisfied by a check that never
|
||||
# fires, and the narrowing would have eaten the gate rather than sharpened it.
|
||||
# Three forms, three code paths:
|
||||
# * route NOTATION — `/name` and `-> name` — is exempt from corroboration and
|
||||
# blocks on its own. Nobody writes `/pre-commit` or `-> pre-commit` to mean
|
||||
# the hook, so there is no ambiguity to resolve, and an author who wants a
|
||||
# route checked unconditionally has two ways to say so.
|
||||
# * a PROSE-form target — backticked or bare — blocks when its own sentence
|
||||
# names another target that resolves. `sibling-skill` is that corroborator
|
||||
# here; it is the same shape as both live true positives, which sit beside
|
||||
# `write-docs` and `gitea-labels-milestones` respectively.
|
||||
grammar_case tp-arrow errors "routes to 'no-such-arrow-target'" \
|
||||
"Use when doing the thing. Not the other thing → no-such-arrow-target."
|
||||
grammar_case tp-slash errors "routes to 'no-such-slash-skill'" \
|
||||
"Use when doing the thing. Do not use for improvements — use /no-such-slash-skill instead."
|
||||
grammar_case tp-backticked errors "routes to 'no-such-backticked-skill'" \
|
||||
"Use when doing the thing. Do not use for improvements — use \`sibling-skill\` or \`no-such-backticked-skill\` instead."
|
||||
grammar_case tp-bare-terminal errors "routes to 'no-such-bare-skill'" \
|
||||
"Use when doing the thing. Do not use for improvements — use sibling-skill or no-such-bare-skill instead."
|
||||
|
||||
# And the confirming half of the grammar rule: a compound-modifier target is
|
||||
# CONFIRM-ONLY, not ignored. When the name does exist it still counts as a route
|
||||
# — the rule suppresses the ERROR, it does not delete the target.
|
||||
echo ""
|
||||
echo "--- an attributive target that DOES resolve is still a route, not a discarded token ---"
|
||||
write_skill "$GRAMMAR_ROOT/plugins/p/.apm/skills/attributive-subject" attributive-subject \
|
||||
"Use when doing the thing. Do not use for the other thing — use the sibling-skill helper instead."
|
||||
set +e
|
||||
ATTR_OUT="$(bash "$HOOK" "$GRAMMAR_ROOT/plugins/p/.apm/skills/attributive-subject/SKILL.md" 2>&1)"
|
||||
ATTR_RC=$?
|
||||
set -e
|
||||
if [[ $ATTR_RC -eq 0 && -z "$ATTR_OUT" ]]; then
|
||||
pass "a resolving attributive target neither errors nor is reported"
|
||||
else
|
||||
fail "an attributive target naming a REAL skill produced output (exit $ATTR_RC): $ATTR_OUT"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed"
|
||||
[[ $FAIL -eq 0 ]]
|
||||
@@ -294,6 +294,61 @@ make_budget_fixture() {
|
||||
echo "$file"
|
||||
}
|
||||
|
||||
# desc_of_length <n> — a description of EXACTLY n characters that carries a
|
||||
# boundary clause and names no routing target.
|
||||
#
|
||||
# ADR-0020's missing-boundary-clause SUGGESTION fires on every description
|
||||
# without one, so a fixture that omits it is never "otherwise clean": a test
|
||||
# asserting silence would be asserting the boundary check's ABSENCE rather than
|
||||
# the length boundary it names. The clause is paid for out of the same budget
|
||||
# being measured (padding arithmetic, not a fixed suffix) so the character count
|
||||
# stays exact. "anything else" is unhyphenated, so no routing target rides along.
|
||||
desc_of_length() {
|
||||
python3 - "$1" <<'PY'
|
||||
import sys
|
||||
n = int(sys.argv[1])
|
||||
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
||||
print(prefix + 'x' * (n - len(prefix)))
|
||||
PY
|
||||
}
|
||||
|
||||
# make_tree_fixture <label> <desc> <body_words> — a SKILL.md inside a synthetic
|
||||
# apm plugin monorepo, so the boundary-target resolver has a universe.
|
||||
#
|
||||
# Resolution walks up FROM THE TARGET FILE to an authoring root (the nearest
|
||||
# ancestor holding plugins/*/.apm/{skills,agents}, falling back to .git); it is
|
||||
# never derived from the checker's own location, because deriving it from
|
||||
# ${BASH_SOURCE} leaked this repo's 39-skill universe into every consumer repo
|
||||
# running the hook. A fixture in a bare mktemp -d therefore has NO universe and
|
||||
# correctly reports "DID NOT RUN" — that is not a bug to paper over with a
|
||||
# looser assertion, it is why the fixture has to be a real tree:
|
||||
#
|
||||
# <root>/plugins/subject-plugin/.apm/skills/<label>/SKILL.md <- the subject
|
||||
# <root>/plugins/subject-plugin/.apm/skills/sibling-skill/ <- same package
|
||||
# <root>/plugins/subject-plugin/.apm/agents/sibling-agent.agent.md
|
||||
# <root>/plugins/other-plugin/.apm/skills/cross-plugin-skill/ <- sibling plugin
|
||||
#
|
||||
# The sibling plugin is what makes "every plugin in the monorepo contributes its
|
||||
# names" testable; without it a cross-plugin target and a typo are the same.
|
||||
make_tree_fixture() {
|
||||
local label="$1" desc="$2" body_words="$3" root apm
|
||||
root="$TMPDIR/tree-$label"
|
||||
apm="$root/plugins/subject-plugin/.apm"
|
||||
mkdir -p "$apm/skills/$label" "$apm/skills/sibling-skill" "$apm/agents" \
|
||||
"$root/plugins/other-plugin/.apm/skills/cross-plugin-skill"
|
||||
: > "$apm/agents/sibling-agent.agent.md"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: $label"
|
||||
echo "description: $desc"
|
||||
echo "---"
|
||||
echo ""
|
||||
python3 -c "print(' '.join(['word'] * $body_words))"
|
||||
} > "$apm/skills/$label/SKILL.md"
|
||||
echo "$apm/skills/$label/SKILL.md"
|
||||
}
|
||||
|
||||
# expect_gate <label> <expected: pass|suggest|fail> <file> [needle]
|
||||
expect_gate() {
|
||||
local label="$1" expected="$2" file="$3" needle="${4:-}" out status
|
||||
@@ -323,15 +378,26 @@ expect_gate() {
|
||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
# A check that DECLINED to run must say so and must not fail the file. The
|
||||
# ERROR guard is the point: a declined check that also errored would satisfy
|
||||
# a bare "output contains INFO" assertion.
|
||||
info)
|
||||
if [[ $status -eq 0 && "$out" == *"INFO"* && "$out" == *"$needle"* \
|
||||
&& "$out" != *"ERROR"* ]]; then
|
||||
pass "$label"
|
||||
else
|
||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "--- description budget: $DESC_SUGGEST_CHARS SUGGESTION / $DESC_MAX_CHARS FAIL, both inclusive ---"
|
||||
D_AT_SUGGEST="$(python3 -c "print('x' * $DESC_SUGGEST_CHARS)")"
|
||||
D_OVER_SUGGEST="$(python3 -c "print('x' * $((DESC_SUGGEST_CHARS + 1)))")"
|
||||
D_AT_MAX="$(python3 -c "print('x' * $DESC_MAX_CHARS)")"
|
||||
D_OVER_MAX="$(python3 -c "print('x' * $((DESC_MAX_CHARS + 1)))")"
|
||||
D_AT_SUGGEST="$(desc_of_length "$DESC_SUGGEST_CHARS")"
|
||||
D_OVER_SUGGEST="$(desc_of_length "$((DESC_SUGGEST_CHARS + 1))")"
|
||||
D_AT_MAX="$(desc_of_length "$DESC_MAX_CHARS")"
|
||||
D_OVER_MAX="$(desc_of_length "$((DESC_MAX_CHARS + 1))")"
|
||||
expect_gate "description at exactly $DESC_SUGGEST_CHARS chars is silent" \
|
||||
pass "$(make_budget_fixture desc-at-suggest "$D_AT_SUGGEST" 10)"
|
||||
expect_gate "description at $((DESC_SUGGEST_CHARS + 1)) chars suggests and exits 0" \
|
||||
@@ -359,18 +425,23 @@ FOLDED="$TMPDIR/folded.md"
|
||||
expect_gate "a >-folded 450-char description fails (raw first line would read as 1 char)" \
|
||||
fail "$FOLDED" "description is 450 characters"
|
||||
|
||||
# Every body fixture below carries a boundary clause for the same reason
|
||||
# desc_of_length() does: without one the missing-boundary-clause SUGGESTION
|
||||
# fires and a body-budget test that asserts silence stops isolating the body
|
||||
# budget. It is short, so the description gate stays quiet too.
|
||||
CLEAN_DESC="Short valid description. Do not use for anything else."
|
||||
echo ""
|
||||
echo "--- body budget: $BODY_SUGGEST_WORDS SUGGESTION / $BODY_MAX_WORDS FAIL, body only, both inclusive ---"
|
||||
expect_gate "body at exactly $BODY_SUGGEST_WORDS words is silent" \
|
||||
pass "$(make_budget_fixture body-at-suggest "Short valid description." "$BODY_SUGGEST_WORDS")"
|
||||
pass "$(make_budget_fixture body-at-suggest "$CLEAN_DESC" "$BODY_SUGGEST_WORDS")"
|
||||
expect_gate "body at $((BODY_SUGGEST_WORDS + 1)) words suggests and exits 0" \
|
||||
suggest "$(make_budget_fixture body-over-suggest "Short valid description." "$((BODY_SUGGEST_WORDS + 1))")" \
|
||||
suggest "$(make_budget_fixture body-over-suggest "$CLEAN_DESC" "$((BODY_SUGGEST_WORDS + 1))")" \
|
||||
"body is $((BODY_SUGGEST_WORDS + 1)) words"
|
||||
expect_gate "body at exactly $BODY_MAX_WORDS words suggests, does not fail" \
|
||||
suggest "$(make_budget_fixture body-at-max "Short valid description." "$BODY_MAX_WORDS")" \
|
||||
suggest "$(make_budget_fixture body-at-max "$CLEAN_DESC" "$BODY_MAX_WORDS")" \
|
||||
"body is $BODY_MAX_WORDS words"
|
||||
expect_gate "body at $((BODY_MAX_WORDS + 1)) words fails" \
|
||||
fail "$(make_budget_fixture body-over-max "Short valid description." "$((BODY_MAX_WORDS + 1))")" \
|
||||
fail "$(make_budget_fixture body-over-max "$CLEAN_DESC" "$((BODY_MAX_WORDS + 1))")" \
|
||||
"$BODY_MAX_WORDS-word ceiling"
|
||||
|
||||
# The two word gates measure different things and must stay separable: a file
|
||||
@@ -382,7 +453,7 @@ BODY_ONLY_DESC="$(python3 -c "print(' '.join(['w'] * 100))")"
|
||||
expect_gate "frontmatter words do not count toward the $BODY_MAX_WORDS-word body ceiling" \
|
||||
suggest "$(make_budget_fixture body-independent "$BODY_ONLY_DESC" "$((BODY_MAX_WORDS - 5))")" \
|
||||
"words"
|
||||
BIG_BODY="$(make_budget_fixture body-over-not-whole-file "Short valid description." "$((BODY_MAX_WORDS + 1))")"
|
||||
BIG_BODY="$(make_budget_fixture body-over-not-whole-file "$CLEAN_DESC" "$((BODY_MAX_WORDS + 1))")"
|
||||
BIG_BODY_WORDS="$(wc -w < "$BIG_BODY")"
|
||||
if [[ "$BIG_BODY_WORDS" -le "$MAX_WORDS" ]]; then
|
||||
pass "the body-gate fixture is $BIG_BODY_WORDS whole-file words, well under MAX_WORDS=$MAX_WORDS — it fails on the body gate alone"
|
||||
@@ -393,21 +464,37 @@ fi
|
||||
echo ""
|
||||
echo "--- resolvable boundary targets ---"
|
||||
# Resolution is against the AUTHORING SOURCE (plugins/*/.apm/skills/ and
|
||||
# plugins/*/.apm/agents/), found here via the script's own repo root — these
|
||||
# fixtures live in a temp dir with no plugin tree of their own, so a resolving
|
||||
# target proves the repo-root path works.
|
||||
expect_gate "a boundary target naming a real skill resolves" \
|
||||
pass "$(make_budget_fixture target-ok \
|
||||
"Use when doing the thing. Do not use for commits — use git-commits instead." 10)"
|
||||
expect_gate "a boundary target naming a real AGENT resolves (agents are valid targets)" \
|
||||
pass "$(make_budget_fixture target-agent-ok \
|
||||
"Use when doing the thing. Do not use when the caller is an agent — invoke git-orchestrate instead." 10)"
|
||||
expect_gate "a boundary target that resolves to nothing fails" \
|
||||
fail "$(make_budget_fixture target-missing \
|
||||
"Use when doing the thing. Do not use for improvements — use no-such-skill-anywhere instead." 10)" \
|
||||
# plugins/*/.apm/agents/), reached by walking up FROM THE SKILL FILE. These
|
||||
# fixtures therefore build their own synthetic monorepo (make_tree_fixture) and
|
||||
# name only fixture-local targets: they must not depend on this repo's live
|
||||
# skills, or renaming git-commits would break a test about extraction grammar.
|
||||
expect_gate "a boundary target naming a sibling skill in the same package resolves" \
|
||||
pass "$(make_tree_fixture target-ok \
|
||||
"Use when doing the thing. Do not use for commits — use sibling-skill instead." 10)"
|
||||
expect_gate "a boundary target naming a skill in a SIBLING PLUGIN resolves (that is what a monorepo means)" \
|
||||
pass "$(make_tree_fixture target-cross-plugin \
|
||||
"Use when doing the thing. Do not use for the other thing — use cross-plugin-skill instead." 10)"
|
||||
expect_gate "a boundary target naming an AGENT resolves (agents are valid targets)" \
|
||||
pass "$(make_tree_fixture target-agent-ok \
|
||||
"Use when doing the thing. Do not use when the caller is an agent — invoke sibling-agent instead." 10)"
|
||||
# CORROBORATED: `sibling-skill` resolves in the same sentence, which is what
|
||||
# promotes a prose-form target from "reported" to "blocking". A lone prose-form
|
||||
# target is deliberately not fatal — see the case below and the shared resolver's
|
||||
# CORROBORATION note.
|
||||
expect_gate "a boundary target that resolves to nothing fails when its sentence names one that does" \
|
||||
fail "$(make_tree_fixture target-missing \
|
||||
"Use when doing the thing. Do not use for improvements — use sibling-skill or no-such-skill-anywhere instead." 10)" \
|
||||
"routes to 'no-such-skill-anywhere'"
|
||||
# UNCORROBORATED: identical grammar to the case above, and identical grammar to
|
||||
# "run `pre-commit` instead". Reported at SUGGESTION tier, exit 0 — a gate that
|
||||
# ships hot with no baseline and no suppression mechanism must not block a commit
|
||||
# on a token it cannot tell from a tool name.
|
||||
expect_gate "a lone boundary target that resolves to nothing is reported, not fatal" \
|
||||
suggest "$(make_tree_fixture target-missing-lone \
|
||||
"Use when doing the thing. Do not use for improvements — use no-such-lone-skill instead." 10)" \
|
||||
"routes to 'no-such-lone-skill'"
|
||||
expect_gate "a /slash-command boundary target that resolves to nothing fails" \
|
||||
fail "$(make_budget_fixture target-missing-slash \
|
||||
fail "$(make_tree_fixture target-missing-slash \
|
||||
"Use when doing the thing. Do not use for improvements — use /no-such-slash-skill instead." 10)" \
|
||||
"routes to 'no-such-slash-skill'"
|
||||
# False-positive guards. These phrasings are lifted from real descriptions:
|
||||
@@ -415,16 +502,38 @@ expect_gate "a /slash-command boundary target that resolves to nothing fails" \
|
||||
# gitea-files says "(use Read/Write/Edit)", gitea-labels-milestones says
|
||||
# "through `issue_write`/`pull_request_write`". None of them is a routing
|
||||
# target, and reading any of them as one makes the gate untrustworthy.
|
||||
#
|
||||
# Each carries a boundary clause in a SEPARATE sentence. That is not decoration:
|
||||
# target extraction is decided per sentence, so the clause satisfies the
|
||||
# missing-boundary-clause SUGGESTION (keeping the expected output empty) while
|
||||
# leaving the sentence under test outside a boundary context, which is the exact
|
||||
# condition each of these is about. They are built as trees so a universe exists
|
||||
# — in a bare temp dir the resolver would decline and the guard would pass
|
||||
# vacuously, proving nothing about extraction.
|
||||
expect_gate "'run pre-commit hooks' outside a boundary sentence is not a routing target" \
|
||||
pass "$(make_budget_fixture fp-precommit \
|
||||
"Use when the user wants to run pre-commit hooks or install git hooks." 10)"
|
||||
pass "$(make_tree_fixture fp-precommit \
|
||||
"Use when the user wants to run pre-commit hooks or install git hooks. Do not use for anything else." 10)"
|
||||
expect_gate "an arrow chain outside a boundary clause is not a routing target" \
|
||||
pass "$(make_budget_fixture fp-arrow \
|
||||
"Reproduce → minimise → instrument → fix → regression-test. Use when a bug is reported." 10)"
|
||||
pass "$(make_tree_fixture fp-arrow \
|
||||
"Reproduce → minimise → instrument → fix → regression-test. Use when a bug is reported. Do not use for anything else." 10)"
|
||||
expect_gate "tool names and MCP tool names are not routing targets" \
|
||||
pass "$(make_budget_fixture fp-tools \
|
||||
pass "$(make_tree_fixture fp-tools \
|
||||
"Use when writing issues. Do not use for local files (use Read/Write/Edit) — that write goes through \`issue_write\`/\`pull_request_write\` instead." 10)"
|
||||
|
||||
echo ""
|
||||
echo "--- with NO authoring root the resolver declines OUT LOUD and does not fail the file ---"
|
||||
# The consumer/draft case, and a real one: a SKILL.md in a bare directory with no
|
||||
# plugins/*/.apm/ above it and no .git has no universe to resolve against. The
|
||||
# required behaviour is neither a false FAIL nor silence — silence is how a whole
|
||||
# gate family goes missing unnoticed — so the INFO and the named unchecked target
|
||||
# are both asserted, along with exit 0. This is the same path make_tree_fixture
|
||||
# exists to escape, kept pinned so a future "just use the repo root" shortcut
|
||||
# (the ${BASH_SOURCE} universe leak ADR-0020 removed) fails here.
|
||||
expect_gate "a fixture with no authoring root reports DID NOT RUN and exits 0" \
|
||||
info "$(make_budget_fixture no-universe \
|
||||
"Use when doing the thing. Do not use for improvements — use some-other-skill instead." 10)" \
|
||||
"Unchecked target(s): some-other-skill"
|
||||
|
||||
echo ""
|
||||
echo "--- the three live dangling routing targets are caught (issue #100) ---"
|
||||
# ADR-0020 records four broken routing targets and splits fixing them into its
|
||||
|
||||
Reference in New Issue
Block a user