fix(kyberforge): close the vacuous-pass paths in the ADR-0020 gate scripts
Three ways the gates could report green having measured nothing. All three were
invisible to a passing test suite, because pre-commit prints nothing at all for a
hook that exits 0 — a gate that declines to check and a gate that checked and
passed produce the identical signal.
- A UTF-8 BOM, a leading blank line, a trailing space after a `---` marker or
CRLF line endings defeated the `^---\n` frontmatter matcher. Every ADR-0020
check was then skipped and the file passed: measured at the time, a
550-character description with a 1,000-word body exited 0 behind a BOM.
All four shapes are now tolerated, and frontmatter that genuinely cannot be
parsed is a hard ERROR rather than a silent skip.
- An agent file with a valueless `description:` followed by another key let a
line regex capture the *next* key, which looked non-empty, so the
missing-or-empty branch never fired and every gate below it early-returned on
the empty folded value — zero output, exit 0, on a blocking gate. The one
field this contract is entirely about was the one field a gate could fail to
notice was absent. Presence is now decided on the YAML-folded value and
nowhere else, and a missing or empty description is a hard FAIL in all three
validators.
- The hand-rolled frontmatter fallback disagreed with PyYAML across the FAIL
boundary on folded scalars, so which reader happened to be available decided
the verdict. A fallback that mis-parses a scalar shape reports a vacuous pass,
which is worse than not running, so it is deleted: python3 and PyYAML are hard
requirements that fail loudly with an install pointer.
Boundary-target resolution no longer derives its universe from its own location.
A `${BASH_SOURCE}`-relative repo root leaked this repo's 39-skill universe into
every consumer repo running the hook through pre-commit, so a consumer skill
routing to `skill-audit` resolved against a plugin it had never installed. The
interim form resolved through `.claude/` and `.agents/`, which are gitignored
`apm install` output — the same commit reported 2 dangling targets on a machine
that had run the install and 6 on a fresh clone. Resolution now walks up from the
file being checked to an authoring root (nearest ancestor holding
`plugins/*/.apm/{skills,agents}`, else the nearest `.git`, in two passes so a
nested `.git` cannot outrank a real monorepo root); the universe is every skill
and agent under `<root>/plugins/*/` plus the file's own apm package and that
package's declared `dependencies.apm`. Deployed trees are consulted only when no
authoring root exists at all — the consumer case. One commit now gets one verdict,
which a gate shipping hot with no baseline file has to.
Narrowed in the same pass: a routing target inferred from the prose boundary form
and corroborated by nothing else reports at SUGGESTION instead of blocking. A
blocking check with no escape hatch is the wrong trade when the inference from
prose is the weak part of it.
New deterministic checks, all previously untested or absent: every
`references/<file>.md` a body names must exist (ERROR — a broken pointer is not a
style opinion); a description with no boundary clause at all, a Gotchas section
over five entries, and a Gotchas section over 25% of the body are SUGGESTIONs.
Where no universe can be determined the target check prints `INFO ... DID NOT
RUN` rather than passing quietly. Each prose-scanning check needed its own
false-positive fix — a fenced example of a Gotchas section was being read as the
section itself — and those fixes are pinned rather than assumed.
The resolver is one block copied verbatim into all three scripts between
BEGIN/END markers, because a cache-installed plugin's scripts cannot read outside
their own plugin directory. Nothing asserted the copies were still identical; a
one-line edit to a single copy passed every constant-agreement assertion, since
constants are not what drifts.
Tests land here rather than in a later commit. The existing suites assert the old
behaviour and go red against these scripts, so splitting them would leave a commit
whose own `run-tests` pre-push gate fails in isolation.
Refs: ADR-0020
This commit is contained in:
@@ -35,6 +35,24 @@ You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: a description of EXACTLY <n> characters that carries a boundary
|
||||
# clause and names no routing target. ADR-0020's missing-boundary-clause
|
||||
# SUGGESTION fires on any description without one, so a fixture that omits it
|
||||
# is never "otherwise clean" and a test refuting SUGGESTION would be asserting
|
||||
# the boundary check's absence instead of the thing it names. The clause is
|
||||
# paid for out of the measured budget rather than appended to it, because
|
||||
# these tests measure the description LENGTH. "anything else" is not
|
||||
# hyphenated, so no routing target comes with it.
|
||||
desc_of_length() {
|
||||
python3 - "$1" <<'PY'
|
||||
import sys
|
||||
n = int(sys.argv[1])
|
||||
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
||||
print(prefix + 'x' * (n - len(prefix)))
|
||||
PY
|
||||
}
|
||||
|
||||
# Helper: same shape as make_apm_agent, but the description is supplied
|
||||
# verbatim — used by the ADR-0020 description-budget tests.
|
||||
make_apm_agent_with_desc() {
|
||||
@@ -637,7 +655,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: agent description of exactly 250 chars raises no suggestion" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 250)")"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 250)"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
refute_output --partial "SUGGESTION"
|
||||
@@ -645,7 +663,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: agent description of 251 chars raises a SUGGESTION and still exits 0" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 251)")"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 251)"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output --partial "SUGGESTION"
|
||||
@@ -654,7 +672,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: agent description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 400)")"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 400)"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output --partial "SUGGESTION"
|
||||
@@ -662,7 +680,7 @@ EOF
|
||||
|
||||
@test "ADR-0020: agent description of 401 chars FAILs and exits non-zero" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 401)")"
|
||||
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 401)"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "description is 401 chars"
|
||||
@@ -732,10 +750,18 @@ EOF
|
||||
# body becomes the system prompt of a fresh context. ADR-0020 gates the
|
||||
# former at 900 words and explicitly declines to gate the latter. If a body
|
||||
# word gate is ever added here, it contradicts the ADR.
|
||||
#
|
||||
# The description carries a boundary clause so the ONLY thing this test can
|
||||
# go red on is a body finding. Without one, the missing-boundary-clause
|
||||
# SUGGESTION fires and the blanket `refute_output --partial "SUGGESTION"`
|
||||
# below trips for a reason that has nothing to do with body length — which
|
||||
# would look like the invariant breaking while proving nothing about it.
|
||||
# AGENTS.md cites this test as the pin for that invariant, so it has to fail
|
||||
# for one reason and one reason only.
|
||||
{
|
||||
echo "---"
|
||||
echo "name: my-agent"
|
||||
echo "description: A valid agent description."
|
||||
echo "description: A valid agent description. Do not use for anything else."
|
||||
echo "---"
|
||||
echo ""
|
||||
python3 -c "print(' '.join(['word'] * 1500))"
|
||||
@@ -743,7 +769,13 @@ EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
# A 1,500-word body is 667% of the skill ceiling. Nothing may be said about
|
||||
# it at any tier: not a FAIL, not a SUGGESTION, and not the word-count
|
||||
# wording either tier would use if a gate were quietly added later.
|
||||
refute_output --partial "SUGGESTION"
|
||||
refute_output --partial "1500 words"
|
||||
refute_output --partial "900-word"
|
||||
refute_output --partial "body is"
|
||||
}
|
||||
|
||||
@test "a bare plugin.json with no apm.yml is no longer plugin scope — falls through to project scope" {
|
||||
|
||||
Reference in New Issue
Block a user