fix(kyberforge): close the vacuous-pass paths in the ADR-0020 gate scripts
Three ways the gates could report green having measured nothing. All three were
invisible to a passing test suite, because pre-commit prints nothing at all for a
hook that exits 0 — a gate that declines to check and a gate that checked and
passed produce the identical signal.
- A UTF-8 BOM, a leading blank line, a trailing space after a `---` marker or
CRLF line endings defeated the `^---\n` frontmatter matcher. Every ADR-0020
check was then skipped and the file passed: measured at the time, a
550-character description with a 1,000-word body exited 0 behind a BOM.
All four shapes are now tolerated, and frontmatter that genuinely cannot be
parsed is a hard ERROR rather than a silent skip.
- An agent file with a valueless `description:` followed by another key let a
line regex capture the *next* key, which looked non-empty, so the
missing-or-empty branch never fired and every gate below it early-returned on
the empty folded value — zero output, exit 0, on a blocking gate. The one
field this contract is entirely about was the one field a gate could fail to
notice was absent. Presence is now decided on the YAML-folded value and
nowhere else, and a missing or empty description is a hard FAIL in all three
validators.
- The hand-rolled frontmatter fallback disagreed with PyYAML across the FAIL
boundary on folded scalars, so which reader happened to be available decided
the verdict. A fallback that mis-parses a scalar shape reports a vacuous pass,
which is worse than not running, so it is deleted: python3 and PyYAML are hard
requirements that fail loudly with an install pointer.
Boundary-target resolution no longer derives its universe from its own location.
A `${BASH_SOURCE}`-relative repo root leaked this repo's 39-skill universe into
every consumer repo running the hook through pre-commit, so a consumer skill
routing to `skill-audit` resolved against a plugin it had never installed. The
interim form resolved through `.claude/` and `.agents/`, which are gitignored
`apm install` output — the same commit reported 2 dangling targets on a machine
that had run the install and 6 on a fresh clone. Resolution now walks up from the
file being checked to an authoring root (nearest ancestor holding
`plugins/*/.apm/{skills,agents}`, else the nearest `.git`, in two passes so a
nested `.git` cannot outrank a real monorepo root); the universe is every skill
and agent under `<root>/plugins/*/` plus the file's own apm package and that
package's declared `dependencies.apm`. Deployed trees are consulted only when no
authoring root exists at all — the consumer case. One commit now gets one verdict,
which a gate shipping hot with no baseline file has to.
Narrowed in the same pass: a routing target inferred from the prose boundary form
and corroborated by nothing else reports at SUGGESTION instead of blocking. A
blocking check with no escape hatch is the wrong trade when the inference from
prose is the weak part of it.
New deterministic checks, all previously untested or absent: every
`references/<file>.md` a body names must exist (ERROR — a broken pointer is not a
style opinion); a description with no boundary clause at all, a Gotchas section
over five entries, and a Gotchas section over 25% of the body are SUGGESTIONs.
Where no universe can be determined the target check prints `INFO ... DID NOT
RUN` rather than passing quietly. Each prose-scanning check needed its own
false-positive fix — a fenced example of a Gotchas section was being read as the
section itself — and those fixes are pinned rather than assumed.
The resolver is one block copied verbatim into all three scripts between
BEGIN/END markers, because a cache-installed plugin's scripts cannot read outside
their own plugin directory. Nothing asserted the copies were still identical; a
one-line edit to a single copy passed every constant-agreement assertion, since
constants are not what drifts.
Tests land here rather than in a later commit. The existing suites assert the old
behaviour and go red against these scripts, so splitting them would leave a commit
whose own `run-tests` pre-push gate fails in isolation.
Refs: ADR-0020
This commit is contained in:
@@ -294,6 +294,61 @@ make_budget_fixture() {
|
||||
echo "$file"
|
||||
}
|
||||
|
||||
# desc_of_length <n> — a description of EXACTLY n characters that carries a
|
||||
# boundary clause and names no routing target.
|
||||
#
|
||||
# ADR-0020's missing-boundary-clause SUGGESTION fires on every description
|
||||
# without one, so a fixture that omits it is never "otherwise clean": a test
|
||||
# asserting silence would be asserting the boundary check's ABSENCE rather than
|
||||
# the length boundary it names. The clause is paid for out of the same budget
|
||||
# being measured (padding arithmetic, not a fixed suffix) so the character count
|
||||
# stays exact. "anything else" is unhyphenated, so no routing target rides along.
|
||||
desc_of_length() {
|
||||
python3 - "$1" <<'PY'
|
||||
import sys
|
||||
n = int(sys.argv[1])
|
||||
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
||||
print(prefix + 'x' * (n - len(prefix)))
|
||||
PY
|
||||
}
|
||||
|
||||
# make_tree_fixture <label> <desc> <body_words> — a SKILL.md inside a synthetic
|
||||
# apm plugin monorepo, so the boundary-target resolver has a universe.
|
||||
#
|
||||
# Resolution walks up FROM THE TARGET FILE to an authoring root (the nearest
|
||||
# ancestor holding plugins/*/.apm/{skills,agents}, falling back to .git); it is
|
||||
# never derived from the checker's own location, because deriving it from
|
||||
# ${BASH_SOURCE} leaked this repo's 39-skill universe into every consumer repo
|
||||
# running the hook. A fixture in a bare mktemp -d therefore has NO universe and
|
||||
# correctly reports "DID NOT RUN" — that is not a bug to paper over with a
|
||||
# looser assertion, it is why the fixture has to be a real tree:
|
||||
#
|
||||
# <root>/plugins/subject-plugin/.apm/skills/<label>/SKILL.md <- the subject
|
||||
# <root>/plugins/subject-plugin/.apm/skills/sibling-skill/ <- same package
|
||||
# <root>/plugins/subject-plugin/.apm/agents/sibling-agent.agent.md
|
||||
# <root>/plugins/other-plugin/.apm/skills/cross-plugin-skill/ <- sibling plugin
|
||||
#
|
||||
# The sibling plugin is what makes "every plugin in the monorepo contributes its
|
||||
# names" testable; without it a cross-plugin target and a typo are the same.
|
||||
make_tree_fixture() {
|
||||
local label="$1" desc="$2" body_words="$3" root apm
|
||||
root="$TMPDIR/tree-$label"
|
||||
apm="$root/plugins/subject-plugin/.apm"
|
||||
mkdir -p "$apm/skills/$label" "$apm/skills/sibling-skill" "$apm/agents" \
|
||||
"$root/plugins/other-plugin/.apm/skills/cross-plugin-skill"
|
||||
: > "$apm/agents/sibling-agent.agent.md"
|
||||
{
|
||||
echo "---"
|
||||
echo "name: $label"
|
||||
echo "description: $desc"
|
||||
echo "---"
|
||||
echo ""
|
||||
python3 -c "print(' '.join(['word'] * $body_words))"
|
||||
} > "$apm/skills/$label/SKILL.md"
|
||||
echo "$apm/skills/$label/SKILL.md"
|
||||
}
|
||||
|
||||
# expect_gate <label> <expected: pass|suggest|fail> <file> [needle]
|
||||
expect_gate() {
|
||||
local label="$1" expected="$2" file="$3" needle="${4:-}" out status
|
||||
@@ -323,15 +378,26 @@ expect_gate() {
|
||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
# A check that DECLINED to run must say so and must not fail the file. The
|
||||
# ERROR guard is the point: a declined check that also errored would satisfy
|
||||
# a bare "output contains INFO" assertion.
|
||||
info)
|
||||
if [[ $status -eq 0 && "$out" == *"INFO"* && "$out" == *"$needle"* \
|
||||
&& "$out" != *"ERROR"* ]]; then
|
||||
pass "$label"
|
||||
else
|
||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "--- description budget: $DESC_SUGGEST_CHARS SUGGESTION / $DESC_MAX_CHARS FAIL, both inclusive ---"
|
||||
D_AT_SUGGEST="$(python3 -c "print('x' * $DESC_SUGGEST_CHARS)")"
|
||||
D_OVER_SUGGEST="$(python3 -c "print('x' * $((DESC_SUGGEST_CHARS + 1)))")"
|
||||
D_AT_MAX="$(python3 -c "print('x' * $DESC_MAX_CHARS)")"
|
||||
D_OVER_MAX="$(python3 -c "print('x' * $((DESC_MAX_CHARS + 1)))")"
|
||||
D_AT_SUGGEST="$(desc_of_length "$DESC_SUGGEST_CHARS")"
|
||||
D_OVER_SUGGEST="$(desc_of_length "$((DESC_SUGGEST_CHARS + 1))")"
|
||||
D_AT_MAX="$(desc_of_length "$DESC_MAX_CHARS")"
|
||||
D_OVER_MAX="$(desc_of_length "$((DESC_MAX_CHARS + 1))")"
|
||||
expect_gate "description at exactly $DESC_SUGGEST_CHARS chars is silent" \
|
||||
pass "$(make_budget_fixture desc-at-suggest "$D_AT_SUGGEST" 10)"
|
||||
expect_gate "description at $((DESC_SUGGEST_CHARS + 1)) chars suggests and exits 0" \
|
||||
@@ -359,18 +425,23 @@ FOLDED="$TMPDIR/folded.md"
|
||||
expect_gate "a >-folded 450-char description fails (raw first line would read as 1 char)" \
|
||||
fail "$FOLDED" "description is 450 characters"
|
||||
|
||||
# Every body fixture below carries a boundary clause for the same reason
|
||||
# desc_of_length() does: without one the missing-boundary-clause SUGGESTION
|
||||
# fires and a body-budget test that asserts silence stops isolating the body
|
||||
# budget. It is short, so the description gate stays quiet too.
|
||||
CLEAN_DESC="Short valid description. Do not use for anything else."
|
||||
echo ""
|
||||
echo "--- body budget: $BODY_SUGGEST_WORDS SUGGESTION / $BODY_MAX_WORDS FAIL, body only, both inclusive ---"
|
||||
expect_gate "body at exactly $BODY_SUGGEST_WORDS words is silent" \
|
||||
pass "$(make_budget_fixture body-at-suggest "Short valid description." "$BODY_SUGGEST_WORDS")"
|
||||
pass "$(make_budget_fixture body-at-suggest "$CLEAN_DESC" "$BODY_SUGGEST_WORDS")"
|
||||
expect_gate "body at $((BODY_SUGGEST_WORDS + 1)) words suggests and exits 0" \
|
||||
suggest "$(make_budget_fixture body-over-suggest "Short valid description." "$((BODY_SUGGEST_WORDS + 1))")" \
|
||||
suggest "$(make_budget_fixture body-over-suggest "$CLEAN_DESC" "$((BODY_SUGGEST_WORDS + 1))")" \
|
||||
"body is $((BODY_SUGGEST_WORDS + 1)) words"
|
||||
expect_gate "body at exactly $BODY_MAX_WORDS words suggests, does not fail" \
|
||||
suggest "$(make_budget_fixture body-at-max "Short valid description." "$BODY_MAX_WORDS")" \
|
||||
suggest "$(make_budget_fixture body-at-max "$CLEAN_DESC" "$BODY_MAX_WORDS")" \
|
||||
"body is $BODY_MAX_WORDS words"
|
||||
expect_gate "body at $((BODY_MAX_WORDS + 1)) words fails" \
|
||||
fail "$(make_budget_fixture body-over-max "Short valid description." "$((BODY_MAX_WORDS + 1))")" \
|
||||
fail "$(make_budget_fixture body-over-max "$CLEAN_DESC" "$((BODY_MAX_WORDS + 1))")" \
|
||||
"$BODY_MAX_WORDS-word ceiling"
|
||||
|
||||
# The two word gates measure different things and must stay separable: a file
|
||||
@@ -382,7 +453,7 @@ BODY_ONLY_DESC="$(python3 -c "print(' '.join(['w'] * 100))")"
|
||||
expect_gate "frontmatter words do not count toward the $BODY_MAX_WORDS-word body ceiling" \
|
||||
suggest "$(make_budget_fixture body-independent "$BODY_ONLY_DESC" "$((BODY_MAX_WORDS - 5))")" \
|
||||
"words"
|
||||
BIG_BODY="$(make_budget_fixture body-over-not-whole-file "Short valid description." "$((BODY_MAX_WORDS + 1))")"
|
||||
BIG_BODY="$(make_budget_fixture body-over-not-whole-file "$CLEAN_DESC" "$((BODY_MAX_WORDS + 1))")"
|
||||
BIG_BODY_WORDS="$(wc -w < "$BIG_BODY")"
|
||||
if [[ "$BIG_BODY_WORDS" -le "$MAX_WORDS" ]]; then
|
||||
pass "the body-gate fixture is $BIG_BODY_WORDS whole-file words, well under MAX_WORDS=$MAX_WORDS — it fails on the body gate alone"
|
||||
@@ -393,21 +464,37 @@ fi
|
||||
echo ""
|
||||
echo "--- resolvable boundary targets ---"
|
||||
# Resolution is against the AUTHORING SOURCE (plugins/*/.apm/skills/ and
|
||||
# plugins/*/.apm/agents/), found here via the script's own repo root — these
|
||||
# fixtures live in a temp dir with no plugin tree of their own, so a resolving
|
||||
# target proves the repo-root path works.
|
||||
expect_gate "a boundary target naming a real skill resolves" \
|
||||
pass "$(make_budget_fixture target-ok \
|
||||
"Use when doing the thing. Do not use for commits — use git-commits instead." 10)"
|
||||
expect_gate "a boundary target naming a real AGENT resolves (agents are valid targets)" \
|
||||
pass "$(make_budget_fixture target-agent-ok \
|
||||
"Use when doing the thing. Do not use when the caller is an agent — invoke git-orchestrate instead." 10)"
|
||||
expect_gate "a boundary target that resolves to nothing fails" \
|
||||
fail "$(make_budget_fixture target-missing \
|
||||
"Use when doing the thing. Do not use for improvements — use no-such-skill-anywhere instead." 10)" \
|
||||
# plugins/*/.apm/agents/), reached by walking up FROM THE SKILL FILE. These
|
||||
# fixtures therefore build their own synthetic monorepo (make_tree_fixture) and
|
||||
# name only fixture-local targets: they must not depend on this repo's live
|
||||
# skills, or renaming git-commits would break a test about extraction grammar.
|
||||
expect_gate "a boundary target naming a sibling skill in the same package resolves" \
|
||||
pass "$(make_tree_fixture target-ok \
|
||||
"Use when doing the thing. Do not use for commits — use sibling-skill instead." 10)"
|
||||
expect_gate "a boundary target naming a skill in a SIBLING PLUGIN resolves (that is what a monorepo means)" \
|
||||
pass "$(make_tree_fixture target-cross-plugin \
|
||||
"Use when doing the thing. Do not use for the other thing — use cross-plugin-skill instead." 10)"
|
||||
expect_gate "a boundary target naming an AGENT resolves (agents are valid targets)" \
|
||||
pass "$(make_tree_fixture target-agent-ok \
|
||||
"Use when doing the thing. Do not use when the caller is an agent — invoke sibling-agent instead." 10)"
|
||||
# CORROBORATED: `sibling-skill` resolves in the same sentence, which is what
|
||||
# promotes a prose-form target from "reported" to "blocking". A lone prose-form
|
||||
# target is deliberately not fatal — see the case below and the shared resolver's
|
||||
# CORROBORATION note.
|
||||
expect_gate "a boundary target that resolves to nothing fails when its sentence names one that does" \
|
||||
fail "$(make_tree_fixture target-missing \
|
||||
"Use when doing the thing. Do not use for improvements — use sibling-skill or no-such-skill-anywhere instead." 10)" \
|
||||
"routes to 'no-such-skill-anywhere'"
|
||||
# UNCORROBORATED: identical grammar to the case above, and identical grammar to
|
||||
# "run `pre-commit` instead". Reported at SUGGESTION tier, exit 0 — a gate that
|
||||
# ships hot with no baseline and no suppression mechanism must not block a commit
|
||||
# on a token it cannot tell from a tool name.
|
||||
expect_gate "a lone boundary target that resolves to nothing is reported, not fatal" \
|
||||
suggest "$(make_tree_fixture target-missing-lone \
|
||||
"Use when doing the thing. Do not use for improvements — use no-such-lone-skill instead." 10)" \
|
||||
"routes to 'no-such-lone-skill'"
|
||||
expect_gate "a /slash-command boundary target that resolves to nothing fails" \
|
||||
fail "$(make_budget_fixture target-missing-slash \
|
||||
fail "$(make_tree_fixture target-missing-slash \
|
||||
"Use when doing the thing. Do not use for improvements — use /no-such-slash-skill instead." 10)" \
|
||||
"routes to 'no-such-slash-skill'"
|
||||
# False-positive guards. These phrasings are lifted from real descriptions:
|
||||
@@ -415,16 +502,38 @@ expect_gate "a /slash-command boundary target that resolves to nothing fails" \
|
||||
# gitea-files says "(use Read/Write/Edit)", gitea-labels-milestones says
|
||||
# "through `issue_write`/`pull_request_write`". None of them is a routing
|
||||
# target, and reading any of them as one makes the gate untrustworthy.
|
||||
#
|
||||
# Each carries a boundary clause in a SEPARATE sentence. That is not decoration:
|
||||
# target extraction is decided per sentence, so the clause satisfies the
|
||||
# missing-boundary-clause SUGGESTION (keeping the expected output empty) while
|
||||
# leaving the sentence under test outside a boundary context, which is the exact
|
||||
# condition each of these is about. They are built as trees so a universe exists
|
||||
# — in a bare temp dir the resolver would decline and the guard would pass
|
||||
# vacuously, proving nothing about extraction.
|
||||
expect_gate "'run pre-commit hooks' outside a boundary sentence is not a routing target" \
|
||||
pass "$(make_budget_fixture fp-precommit \
|
||||
"Use when the user wants to run pre-commit hooks or install git hooks." 10)"
|
||||
pass "$(make_tree_fixture fp-precommit \
|
||||
"Use when the user wants to run pre-commit hooks or install git hooks. Do not use for anything else." 10)"
|
||||
expect_gate "an arrow chain outside a boundary clause is not a routing target" \
|
||||
pass "$(make_budget_fixture fp-arrow \
|
||||
"Reproduce → minimise → instrument → fix → regression-test. Use when a bug is reported." 10)"
|
||||
pass "$(make_tree_fixture fp-arrow \
|
||||
"Reproduce → minimise → instrument → fix → regression-test. Use when a bug is reported. Do not use for anything else." 10)"
|
||||
expect_gate "tool names and MCP tool names are not routing targets" \
|
||||
pass "$(make_budget_fixture fp-tools \
|
||||
pass "$(make_tree_fixture fp-tools \
|
||||
"Use when writing issues. Do not use for local files (use Read/Write/Edit) — that write goes through \`issue_write\`/\`pull_request_write\` instead." 10)"
|
||||
|
||||
echo ""
|
||||
echo "--- with NO authoring root the resolver declines OUT LOUD and does not fail the file ---"
|
||||
# The consumer/draft case, and a real one: a SKILL.md in a bare directory with no
|
||||
# plugins/*/.apm/ above it and no .git has no universe to resolve against. The
|
||||
# required behaviour is neither a false FAIL nor silence — silence is how a whole
|
||||
# gate family goes missing unnoticed — so the INFO and the named unchecked target
|
||||
# are both asserted, along with exit 0. This is the same path make_tree_fixture
|
||||
# exists to escape, kept pinned so a future "just use the repo root" shortcut
|
||||
# (the ${BASH_SOURCE} universe leak ADR-0020 removed) fails here.
|
||||
expect_gate "a fixture with no authoring root reports DID NOT RUN and exits 0" \
|
||||
info "$(make_budget_fixture no-universe \
|
||||
"Use when doing the thing. Do not use for improvements — use some-other-skill instead." 10)" \
|
||||
"Unchecked target(s): some-other-skill"
|
||||
|
||||
echo ""
|
||||
echo "--- the three live dangling routing targets are caught (issue #100) ---"
|
||||
# ADR-0020 records four broken routing targets and splits fixing them into its
|
||||
|
||||
Reference in New Issue
Block a user