fix(kyberforge): close the vacuous-pass paths in the ADR-0020 gate scripts
Three ways the gates could report green having measured nothing. All three were
invisible to a passing test suite, because pre-commit prints nothing at all for a
hook that exits 0 — a gate that declines to check and a gate that checked and
passed produce the identical signal.
- A UTF-8 BOM, a leading blank line, a trailing space after a `---` marker or
CRLF line endings defeated the `^---\n` frontmatter matcher. Every ADR-0020
check was then skipped and the file passed: measured at the time, a
550-character description with a 1,000-word body exited 0 behind a BOM.
All four shapes are now tolerated, and frontmatter that genuinely cannot be
parsed is a hard ERROR rather than a silent skip.
- An agent file with a valueless `description:` followed by another key let a
line regex capture the *next* key, which looked non-empty, so the
missing-or-empty branch never fired and every gate below it early-returned on
the empty folded value — zero output, exit 0, on a blocking gate. The one
field this contract is entirely about was the one field a gate could fail to
notice was absent. Presence is now decided on the YAML-folded value and
nowhere else, and a missing or empty description is a hard FAIL in all three
validators.
- The hand-rolled frontmatter fallback disagreed with PyYAML across the FAIL
boundary on folded scalars, so which reader happened to be available decided
the verdict. A fallback that mis-parses a scalar shape reports a vacuous pass,
which is worse than not running, so it is deleted: python3 and PyYAML are hard
requirements that fail loudly with an install pointer.
Boundary-target resolution no longer derives its universe from its own location.
A `${BASH_SOURCE}`-relative repo root leaked this repo's 39-skill universe into
every consumer repo running the hook through pre-commit, so a consumer skill
routing to `skill-audit` resolved against a plugin it had never installed. The
interim form resolved through `.claude/` and `.agents/`, which are gitignored
`apm install` output — the same commit reported 2 dangling targets on a machine
that had run the install and 6 on a fresh clone. Resolution now walks up from the
file being checked to an authoring root (nearest ancestor holding
`plugins/*/.apm/{skills,agents}`, else the nearest `.git`, in two passes so a
nested `.git` cannot outrank a real monorepo root); the universe is every skill
and agent under `<root>/plugins/*/` plus the file's own apm package and that
package's declared `dependencies.apm`. Deployed trees are consulted only when no
authoring root exists at all — the consumer case. One commit now gets one verdict,
which a gate shipping hot with no baseline file has to.
Narrowed in the same pass: a routing target inferred from the prose boundary form
and corroborated by nothing else reports at SUGGESTION instead of blocking. A
blocking check with no escape hatch is the wrong trade when the inference from
prose is the weak part of it.
New deterministic checks, all previously untested or absent: every
`references/<file>.md` a body names must exist (ERROR — a broken pointer is not a
style opinion); a description with no boundary clause at all, a Gotchas section
over five entries, and a Gotchas section over 25% of the body are SUGGESTIONs.
Where no universe can be determined the target check prints `INFO ... DID NOT
RUN` rather than passing quietly. Each prose-scanning check needed its own
false-positive fix — a fenced example of a Gotchas section was being read as the
section itself — and those fixes are pinned rather than assumed.
The resolver is one block copied verbatim into all three scripts between
BEGIN/END markers, because a cache-installed plugin's scripts cannot read outside
their own plugin directory. Nothing asserted the copies were still identical; a
one-line edit to a single copy passed every constant-agreement assertion, since
constants are not what drifts.
Tests land here rather than in a later commit. The existing suites assert the old
behaviour and go red against these scripts, so splitting them would leave a commit
whose own `run-tests` pre-push gate fails in isolation.
Refs: ADR-0020
This commit is contained in:
@@ -8,7 +8,14 @@ setup() {
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
|
||||
# Helper: create a minimal valid skill directory
|
||||
# Helper: create a minimal valid skill directory.
|
||||
#
|
||||
# The description carries a boundary clause deliberately. ADR-0020's
|
||||
# missing-boundary-clause SUGGESTION fires on any description without one, so
|
||||
# a fixture that omits it is never "otherwise clean" — every test asserting
|
||||
# SUGGESTION-freedom would be asserting the boundary check's absence instead
|
||||
# of the thing it names. "anything else" is not hyphenated, so the clause adds
|
||||
# a boundary marker without adding a routing target to resolve.
|
||||
make_valid_skill() {
|
||||
local dir="$1"
|
||||
local name
|
||||
@@ -17,7 +24,7 @@ setup() {
|
||||
cat > "$dir/SKILL.md" <<EOF
|
||||
---
|
||||
name: $name
|
||||
description: A valid skill description that is well within the limit.
|
||||
description: A valid skill description that is well within the limit. Do not use for anything else.
|
||||
---
|
||||
|
||||
## Step 1
|
||||
@@ -26,6 +33,20 @@ Do the thing.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: a description of EXACTLY <n> characters that carries a boundary
|
||||
# clause and names no routing target. The tests below measure the description
|
||||
# LENGTH, so the clause has to be paid for out of the same budget rather than
|
||||
# appended to it — hence the padding arithmetic instead of a fixed suffix.
|
||||
desc_of_length() {
|
||||
python3 - "$1" <<'PY'
|
||||
import sys
|
||||
n = int(sys.argv[1])
|
||||
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
||||
print(prefix + 'x' * (n - len(prefix)))
|
||||
PY
|
||||
}
|
||||
|
||||
# Helper: create a skill directory with an exact description length and an
|
||||
# exact body word count. <desc> is used verbatim; <body_words> "word"
|
||||
# tokens follow the frontmatter. Used by the ADR-0020 boundary tests.
|
||||
@@ -300,7 +321,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: description of exactly 250 chars raises no suggestion" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 250)")" 10
|
||||
make_sized_skill "$skill" "$(desc_of_length 250)" 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
refute_output --partial "SUGGESTION"
|
||||
@@ -308,7 +329,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: description of 251 chars raises a SUGGESTION and still exits 0" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 251)")" 10
|
||||
make_sized_skill "$skill" "$(desc_of_length 251)" 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
assert_output --partial "SUGGESTION"
|
||||
@@ -318,7 +339,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 400)")" 10
|
||||
make_sized_skill "$skill" "$(desc_of_length 400)" 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
assert_output --partial "SUGGESTION"
|
||||
@@ -326,7 +347,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: description of 401 chars FAILs and exits non-zero" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 401)")" 10
|
||||
make_sized_skill "$skill" "$(desc_of_length 401)" 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_failure
|
||||
assert_output --partial "description is 401 chars"
|
||||
@@ -364,7 +385,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: body of exactly 600 words raises no suggestion" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "A short valid description." 600
|
||||
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 600
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
refute_output --partial "SUGGESTION"
|
||||
@@ -372,7 +393,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: body of 601 words raises a SUGGESTION and still exits 0" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "A short valid description." 601
|
||||
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 601
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
assert_output --partial "body is 601 words"
|
||||
@@ -381,7 +402,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: body of exactly 900 words is a SUGGESTION, not a FAIL" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "A short valid description." 900
|
||||
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 900
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
assert_output --partial "body is 900 words"
|
||||
@@ -389,7 +410,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: body of 901 words FAILs and exits non-zero" {
|
||||
local skill="$TMPDIR/my-skill"
|
||||
make_sized_skill "$skill" "A short valid description." 901
|
||||
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 901
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_failure
|
||||
assert_output --partial "body is 901 words"
|
||||
@@ -425,15 +446,33 @@ EOF
|
||||
assert_output --partial "boundary target(s) resolve"
|
||||
}
|
||||
|
||||
@test "ADR-0020: a boundary target naming a non-existent skill FAILs" {
|
||||
@test "ADR-0020: a boundary target naming a non-existent skill FAILs when its sentence names one that resolves" {
|
||||
local skill
|
||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-missing-skill instead." 10
|
||||
# `fixture-sibling-skill` is the corroborator: a prose-form target only earns
|
||||
# a FAIL when its own sentence proves it is a routing sentence. See the
|
||||
# shared resolver's CORROBORATION note, and the uncorroborated case below.
|
||||
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-sibling-skill or fixture-missing-skill instead." 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_failure
|
||||
assert_output --partial "routes to 'fixture-missing-skill'"
|
||||
}
|
||||
|
||||
@test "ADR-0020: a LONE boundary target naming a non-existent skill is a SUGGESTION, not a FAIL" {
|
||||
local skill
|
||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||
# Same grammar as the case above and as "run \`pre-commit\` instead" — a
|
||||
# route verb, a hyphenated name, terminal position. Nothing local separates a
|
||||
# broken route from a tool name, so the target is named on every run but does
|
||||
# not block: this gate ships with no baseline and no suppression mechanism.
|
||||
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-missing-skill instead." 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
assert_output --partial "SUGGESTION"
|
||||
assert_output --partial "routes to 'fixture-missing-skill'"
|
||||
refute_output --partial "FAIL description routes to"
|
||||
}
|
||||
|
||||
@test "ADR-0020: a boundary target naming an AGENT file resolves (agents are valid routing targets)" {
|
||||
local skill
|
||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||
@@ -452,15 +491,27 @@ EOF
|
||||
assert_output --partial "routes to 'fixture-missing-improve'"
|
||||
}
|
||||
|
||||
@test "ADR-0020: a backticked name that does not resolve FAILs" {
|
||||
@test "ADR-0020: a backticked name that does not resolve FAILs when its sentence names one that resolves" {
|
||||
local skill
|
||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||
make_sized_skill "$skill" "Use when doing the thing. Composes \`fixture-missing-helper\` for the shared part." 10
|
||||
make_sized_skill "$skill" "Use when doing the thing. Composes \`fixture-sibling-skill\` and \`fixture-missing-helper\` for the shared part." 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_failure
|
||||
assert_output --partial "routes to 'fixture-missing-helper'"
|
||||
}
|
||||
|
||||
@test "ADR-0020: a /slash-command target is route NOTATION and FAILs on its own, uncorroborated" {
|
||||
local skill
|
||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||
# The escape hatch from the SUGGESTION tier: `/name` and `-> name` are never
|
||||
# how prose cites a tool, so they are exempt from corroboration. An author
|
||||
# who wants a route checked unconditionally writes one of those two forms.
|
||||
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use /fixture-missing-notation instead." 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_failure
|
||||
assert_output --partial "routes to 'fixture-missing-notation'"
|
||||
}
|
||||
|
||||
@test "ADR-0020: a bare hyphenated word outside a boundary sentence is not read as a routing target" {
|
||||
local skill
|
||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||
@@ -502,9 +553,18 @@ EOF
|
||||
}
|
||||
|
||||
@test "ADR-0020: the boundary check declines rather than false-FAILs when no authoring source is found" {
|
||||
# Deliberately NOT built with make_fixture_tree: this skill sits in a bare
|
||||
# temp directory with no plugins/*/.apm/ above it and no .git, so the resolver
|
||||
# legitimately has no universe. That is a real path (a skill being drafted
|
||||
# outside any repo), and the required behaviour is to DECLINE OUT LOUD rather
|
||||
# than either false-FAIL or pass in silence — silence is what let a whole gate
|
||||
# family go missing unnoticed. So the INFO text and the named unchecked target
|
||||
# are both asserted, not just the absence of a failure.
|
||||
local skill="$TMPDIR/orphan/my-skill"
|
||||
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use some-other-skill instead." 10
|
||||
run bash "$SCRIPT" "$skill"
|
||||
assert_success
|
||||
refute_output --partial "routes to"
|
||||
assert_output --partial "boundary-target resolution DID NOT RUN"
|
||||
assert_output --partial "Unchecked target(s): some-other-skill"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user