fix(kyberforge): close the vacuous-pass paths in the ADR-0020 gate scripts
Three ways the gates could report green having measured nothing. All three were
invisible to a passing test suite, because pre-commit prints nothing at all for a
hook that exits 0 — a gate that declines to check and a gate that checked and
passed produce the identical signal.
- A UTF-8 BOM, a leading blank line, a trailing space after a `---` marker or
CRLF line endings defeated the `^---\n` frontmatter matcher. Every ADR-0020
check was then skipped and the file passed: measured at the time, a
550-character description with a 1,000-word body exited 0 behind a BOM.
All four shapes are now tolerated, and frontmatter that genuinely cannot be
parsed is a hard ERROR rather than a silent skip.
- An agent file with a valueless `description:` followed by another key let a
line regex capture the *next* key, which looked non-empty, so the
missing-or-empty branch never fired and every gate below it early-returned on
the empty folded value — zero output, exit 0, on a blocking gate. The one
field this contract is entirely about was the one field a gate could fail to
notice was absent. Presence is now decided on the YAML-folded value and
nowhere else, and a missing or empty description is a hard FAIL in all three
validators.
- The hand-rolled frontmatter fallback disagreed with PyYAML across the FAIL
boundary on folded scalars, so which reader happened to be available decided
the verdict. A fallback that mis-parses a scalar shape reports a vacuous pass,
which is worse than not running, so it is deleted: python3 and PyYAML are hard
requirements that fail loudly with an install pointer.
Boundary-target resolution no longer derives its universe from its own location.
A `${BASH_SOURCE}`-relative repo root leaked this repo's 39-skill universe into
every consumer repo running the hook through pre-commit, so a consumer skill
routing to `skill-audit` resolved against a plugin it had never installed. The
interim form resolved through `.claude/` and `.agents/`, which are gitignored
`apm install` output — the same commit reported 2 dangling targets on a machine
that had run the install and 6 on a fresh clone. Resolution now walks up from the
file being checked to an authoring root (nearest ancestor holding
`plugins/*/.apm/{skills,agents}`, else the nearest `.git`, in two passes so a
nested `.git` cannot outrank a real monorepo root); the universe is every skill
and agent under `<root>/plugins/*/` plus the file's own apm package and that
package's declared `dependencies.apm`. Deployed trees are consulted only when no
authoring root exists at all — the consumer case. One commit now gets one verdict,
which a gate shipping hot with no baseline file has to.
Narrowed in the same pass: a routing target inferred from the prose boundary form
and corroborated by nothing else reports at SUGGESTION instead of blocking. A
blocking check with no escape hatch is the wrong trade when the inference from
prose is the weak part of it.
New deterministic checks, all previously untested or absent: every
`references/<file>.md` a body names must exist (ERROR — a broken pointer is not a
style opinion); a description with no boundary clause at all, a Gotchas section
over five entries, and a Gotchas section over 25% of the body are SUGGESTIONs.
Where no universe can be determined the target check prints `INFO ... DID NOT
RUN` rather than passing quietly. Each prose-scanning check needed its own
false-positive fix — a fenced example of a Gotchas section was being read as the
section itself — and those fixes are pinned rather than assumed.
The resolver is one block copied verbatim into all three scripts between
BEGIN/END markers, because a cache-installed plugin's scripts cannot read outside
their own plugin directory. Nothing asserted the copies were still identical; a
one-line edit to a single copy passed every constant-agreement assertion, since
constants are not what drifts.
Tests land here rather than in a later commit. The existing suites assert the old
behaviour and go red against these scripts, so splitting them would leave a commit
whose own `run-tests` pre-push gate fails in isolation.
Refs: ADR-0020
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -35,6 +35,24 @@ You are a test agent. When invoked, do the thing.
|
|||||||
EOF
|
EOF
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Helper: a description of EXACTLY <n> characters that carries a boundary
|
||||||
|
# clause and names no routing target. ADR-0020's missing-boundary-clause
|
||||||
|
# SUGGESTION fires on any description without one, so a fixture that omits it
|
||||||
|
# is never "otherwise clean" and a test refuting SUGGESTION would be asserting
|
||||||
|
# the boundary check's absence instead of the thing it names. The clause is
|
||||||
|
# paid for out of the measured budget rather than appended to it, because
|
||||||
|
# these tests measure the description LENGTH. "anything else" is not
|
||||||
|
# hyphenated, so no routing target comes with it.
|
||||||
|
desc_of_length() {
|
||||||
|
python3 - "$1" <<'PY'
|
||||||
|
import sys
|
||||||
|
n = int(sys.argv[1])
|
||||||
|
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||||
|
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
||||||
|
print(prefix + 'x' * (n - len(prefix)))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
# Helper: same shape as make_apm_agent, but the description is supplied
|
# Helper: same shape as make_apm_agent, but the description is supplied
|
||||||
# verbatim — used by the ADR-0020 description-budget tests.
|
# verbatim — used by the ADR-0020 description-budget tests.
|
||||||
make_apm_agent_with_desc() {
|
make_apm_agent_with_desc() {
|
||||||
@@ -637,7 +655,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: agent description of exactly 250 chars raises no suggestion" {
|
@test "ADR-0020: agent description of exactly 250 chars raises no suggestion" {
|
||||||
local root="$TMPDIR/pkg"
|
local root="$TMPDIR/pkg"
|
||||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 250)")"
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 250)"
|
||||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||||
assert_success
|
assert_success
|
||||||
refute_output --partial "SUGGESTION"
|
refute_output --partial "SUGGESTION"
|
||||||
@@ -645,7 +663,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: agent description of 251 chars raises a SUGGESTION and still exits 0" {
|
@test "ADR-0020: agent description of 251 chars raises a SUGGESTION and still exits 0" {
|
||||||
local root="$TMPDIR/pkg"
|
local root="$TMPDIR/pkg"
|
||||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 251)")"
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 251)"
|
||||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||||
assert_success
|
assert_success
|
||||||
assert_output --partial "SUGGESTION"
|
assert_output --partial "SUGGESTION"
|
||||||
@@ -654,7 +672,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: agent description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
@test "ADR-0020: agent description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
||||||
local root="$TMPDIR/pkg"
|
local root="$TMPDIR/pkg"
|
||||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 400)")"
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 400)"
|
||||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||||
assert_success
|
assert_success
|
||||||
assert_output --partial "SUGGESTION"
|
assert_output --partial "SUGGESTION"
|
||||||
@@ -662,7 +680,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: agent description of 401 chars FAILs and exits non-zero" {
|
@test "ADR-0020: agent description of 401 chars FAILs and exits non-zero" {
|
||||||
local root="$TMPDIR/pkg"
|
local root="$TMPDIR/pkg"
|
||||||
make_apm_agent_with_desc "$root" "my-agent" "$(python3 -c "print('x' * 401)")"
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 401)"
|
||||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||||
assert_failure
|
assert_failure
|
||||||
assert_output --partial "description is 401 chars"
|
assert_output --partial "description is 401 chars"
|
||||||
@@ -732,10 +750,18 @@ EOF
|
|||||||
# body becomes the system prompt of a fresh context. ADR-0020 gates the
|
# body becomes the system prompt of a fresh context. ADR-0020 gates the
|
||||||
# former at 900 words and explicitly declines to gate the latter. If a body
|
# former at 900 words and explicitly declines to gate the latter. If a body
|
||||||
# word gate is ever added here, it contradicts the ADR.
|
# word gate is ever added here, it contradicts the ADR.
|
||||||
|
#
|
||||||
|
# The description carries a boundary clause so the ONLY thing this test can
|
||||||
|
# go red on is a body finding. Without one, the missing-boundary-clause
|
||||||
|
# SUGGESTION fires and the blanket `refute_output --partial "SUGGESTION"`
|
||||||
|
# below trips for a reason that has nothing to do with body length — which
|
||||||
|
# would look like the invariant breaking while proving nothing about it.
|
||||||
|
# AGENTS.md cites this test as the pin for that invariant, so it has to fail
|
||||||
|
# for one reason and one reason only.
|
||||||
{
|
{
|
||||||
echo "---"
|
echo "---"
|
||||||
echo "name: my-agent"
|
echo "name: my-agent"
|
||||||
echo "description: A valid agent description."
|
echo "description: A valid agent description. Do not use for anything else."
|
||||||
echo "---"
|
echo "---"
|
||||||
echo ""
|
echo ""
|
||||||
python3 -c "print(' '.join(['word'] * 1500))"
|
python3 -c "print(' '.join(['word'] * 1500))"
|
||||||
@@ -743,7 +769,13 @@ EOF
|
|||||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||||
assert_success
|
assert_success
|
||||||
refute_output --partial "FAIL"
|
refute_output --partial "FAIL"
|
||||||
|
# A 1,500-word body is 667% of the skill ceiling. Nothing may be said about
|
||||||
|
# it at any tier: not a FAIL, not a SUGGESTION, and not the word-count
|
||||||
|
# wording either tier would use if a gate were quietly added later.
|
||||||
refute_output --partial "SUGGESTION"
|
refute_output --partial "SUGGESTION"
|
||||||
|
refute_output --partial "1500 words"
|
||||||
|
refute_output --partial "900-word"
|
||||||
|
refute_output --partial "body is"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "a bare plugin.json with no apm.yml is no longer plugin scope — falls through to project scope" {
|
@test "a bare plugin.json with no apm.yml is no longer plugin scope — falls through to project scope" {
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -8,7 +8,14 @@ setup() {
|
|||||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
||||||
TMPDIR="$(mktemp -d)"
|
TMPDIR="$(mktemp -d)"
|
||||||
|
|
||||||
# Helper: create a minimal valid skill directory
|
# Helper: create a minimal valid skill directory.
|
||||||
|
#
|
||||||
|
# The description carries a boundary clause deliberately. ADR-0020's
|
||||||
|
# missing-boundary-clause SUGGESTION fires on any description without one, so
|
||||||
|
# a fixture that omits it is never "otherwise clean" — every test asserting
|
||||||
|
# SUGGESTION-freedom would be asserting the boundary check's absence instead
|
||||||
|
# of the thing it names. "anything else" is not hyphenated, so the clause adds
|
||||||
|
# a boundary marker without adding a routing target to resolve.
|
||||||
make_valid_skill() {
|
make_valid_skill() {
|
||||||
local dir="$1"
|
local dir="$1"
|
||||||
local name
|
local name
|
||||||
@@ -17,7 +24,7 @@ setup() {
|
|||||||
cat > "$dir/SKILL.md" <<EOF
|
cat > "$dir/SKILL.md" <<EOF
|
||||||
---
|
---
|
||||||
name: $name
|
name: $name
|
||||||
description: A valid skill description that is well within the limit.
|
description: A valid skill description that is well within the limit. Do not use for anything else.
|
||||||
---
|
---
|
||||||
|
|
||||||
## Step 1
|
## Step 1
|
||||||
@@ -26,6 +33,20 @@ Do the thing.
|
|||||||
EOF
|
EOF
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Helper: a description of EXACTLY <n> characters that carries a boundary
|
||||||
|
# clause and names no routing target. The tests below measure the description
|
||||||
|
# LENGTH, so the clause has to be paid for out of the same budget rather than
|
||||||
|
# appended to it — hence the padding arithmetic instead of a fixed suffix.
|
||||||
|
desc_of_length() {
|
||||||
|
python3 - "$1" <<'PY'
|
||||||
|
import sys
|
||||||
|
n = int(sys.argv[1])
|
||||||
|
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||||
|
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
||||||
|
print(prefix + 'x' * (n - len(prefix)))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
# Helper: create a skill directory with an exact description length and an
|
# Helper: create a skill directory with an exact description length and an
|
||||||
# exact body word count. <desc> is used verbatim; <body_words> "word"
|
# exact body word count. <desc> is used verbatim; <body_words> "word"
|
||||||
# tokens follow the frontmatter. Used by the ADR-0020 boundary tests.
|
# tokens follow the frontmatter. Used by the ADR-0020 boundary tests.
|
||||||
@@ -300,7 +321,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: description of exactly 250 chars raises no suggestion" {
|
@test "ADR-0020: description of exactly 250 chars raises no suggestion" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 250)")" 10
|
make_sized_skill "$skill" "$(desc_of_length 250)" 10
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_success
|
assert_success
|
||||||
refute_output --partial "SUGGESTION"
|
refute_output --partial "SUGGESTION"
|
||||||
@@ -308,7 +329,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: description of 251 chars raises a SUGGESTION and still exits 0" {
|
@test "ADR-0020: description of 251 chars raises a SUGGESTION and still exits 0" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 251)")" 10
|
make_sized_skill "$skill" "$(desc_of_length 251)" 10
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_success
|
assert_success
|
||||||
assert_output --partial "SUGGESTION"
|
assert_output --partial "SUGGESTION"
|
||||||
@@ -318,7 +339,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
@test "ADR-0020: description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 400)")" 10
|
make_sized_skill "$skill" "$(desc_of_length 400)" 10
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_success
|
assert_success
|
||||||
assert_output --partial "SUGGESTION"
|
assert_output --partial "SUGGESTION"
|
||||||
@@ -326,7 +347,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: description of 401 chars FAILs and exits non-zero" {
|
@test "ADR-0020: description of 401 chars FAILs and exits non-zero" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "$(python3 -c "print('x' * 401)")" 10
|
make_sized_skill "$skill" "$(desc_of_length 401)" 10
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_failure
|
assert_failure
|
||||||
assert_output --partial "description is 401 chars"
|
assert_output --partial "description is 401 chars"
|
||||||
@@ -364,7 +385,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: body of exactly 600 words raises no suggestion" {
|
@test "ADR-0020: body of exactly 600 words raises no suggestion" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "A short valid description." 600
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 600
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_success
|
assert_success
|
||||||
refute_output --partial "SUGGESTION"
|
refute_output --partial "SUGGESTION"
|
||||||
@@ -372,7 +393,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: body of 601 words raises a SUGGESTION and still exits 0" {
|
@test "ADR-0020: body of 601 words raises a SUGGESTION and still exits 0" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "A short valid description." 601
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 601
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_success
|
assert_success
|
||||||
assert_output --partial "body is 601 words"
|
assert_output --partial "body is 601 words"
|
||||||
@@ -381,7 +402,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: body of exactly 900 words is a SUGGESTION, not a FAIL" {
|
@test "ADR-0020: body of exactly 900 words is a SUGGESTION, not a FAIL" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "A short valid description." 900
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 900
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_success
|
assert_success
|
||||||
assert_output --partial "body is 900 words"
|
assert_output --partial "body is 900 words"
|
||||||
@@ -389,7 +410,7 @@ EOF
|
|||||||
|
|
||||||
@test "ADR-0020: body of 901 words FAILs and exits non-zero" {
|
@test "ADR-0020: body of 901 words FAILs and exits non-zero" {
|
||||||
local skill="$TMPDIR/my-skill"
|
local skill="$TMPDIR/my-skill"
|
||||||
make_sized_skill "$skill" "A short valid description." 901
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 901
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_failure
|
assert_failure
|
||||||
assert_output --partial "body is 901 words"
|
assert_output --partial "body is 901 words"
|
||||||
@@ -425,15 +446,33 @@ EOF
|
|||||||
assert_output --partial "boundary target(s) resolve"
|
assert_output --partial "boundary target(s) resolve"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "ADR-0020: a boundary target naming a non-existent skill FAILs" {
|
@test "ADR-0020: a boundary target naming a non-existent skill FAILs when its sentence names one that resolves" {
|
||||||
local skill
|
local skill
|
||||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||||
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-missing-skill instead." 10
|
# `fixture-sibling-skill` is the corroborator: a prose-form target only earns
|
||||||
|
# a FAIL when its own sentence proves it is a routing sentence. See the
|
||||||
|
# shared resolver's CORROBORATION note, and the uncorroborated case below.
|
||||||
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-sibling-skill or fixture-missing-skill instead." 10
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_failure
|
assert_failure
|
||||||
assert_output --partial "routes to 'fixture-missing-skill'"
|
assert_output --partial "routes to 'fixture-missing-skill'"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "ADR-0020: a LONE boundary target naming a non-existent skill is a SUGGESTION, not a FAIL" {
|
||||||
|
local skill
|
||||||
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||||
|
# Same grammar as the case above and as "run \`pre-commit\` instead" — a
|
||||||
|
# route verb, a hyphenated name, terminal position. Nothing local separates a
|
||||||
|
# broken route from a tool name, so the target is named on every run but does
|
||||||
|
# not block: this gate ships with no baseline and no suppression mechanism.
|
||||||
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-missing-skill instead." 10
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_success
|
||||||
|
assert_output --partial "SUGGESTION"
|
||||||
|
assert_output --partial "routes to 'fixture-missing-skill'"
|
||||||
|
refute_output --partial "FAIL description routes to"
|
||||||
|
}
|
||||||
|
|
||||||
@test "ADR-0020: a boundary target naming an AGENT file resolves (agents are valid routing targets)" {
|
@test "ADR-0020: a boundary target naming an AGENT file resolves (agents are valid routing targets)" {
|
||||||
local skill
|
local skill
|
||||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||||
@@ -452,15 +491,27 @@ EOF
|
|||||||
assert_output --partial "routes to 'fixture-missing-improve'"
|
assert_output --partial "routes to 'fixture-missing-improve'"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "ADR-0020: a backticked name that does not resolve FAILs" {
|
@test "ADR-0020: a backticked name that does not resolve FAILs when its sentence names one that resolves" {
|
||||||
local skill
|
local skill
|
||||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||||
make_sized_skill "$skill" "Use when doing the thing. Composes \`fixture-missing-helper\` for the shared part." 10
|
make_sized_skill "$skill" "Use when doing the thing. Composes \`fixture-sibling-skill\` and \`fixture-missing-helper\` for the shared part." 10
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_failure
|
assert_failure
|
||||||
assert_output --partial "routes to 'fixture-missing-helper'"
|
assert_output --partial "routes to 'fixture-missing-helper'"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@test "ADR-0020: a /slash-command target is route NOTATION and FAILs on its own, uncorroborated" {
|
||||||
|
local skill
|
||||||
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||||
|
# The escape hatch from the SUGGESTION tier: `/name` and `-> name` are never
|
||||||
|
# how prose cites a tool, so they are exempt from corroboration. An author
|
||||||
|
# who wants a route checked unconditionally writes one of those two forms.
|
||||||
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use /fixture-missing-notation instead." 10
|
||||||
|
run bash "$SCRIPT" "$skill"
|
||||||
|
assert_failure
|
||||||
|
assert_output --partial "routes to 'fixture-missing-notation'"
|
||||||
|
}
|
||||||
|
|
||||||
@test "ADR-0020: a bare hyphenated word outside a boundary sentence is not read as a routing target" {
|
@test "ADR-0020: a bare hyphenated word outside a boundary sentence is not read as a routing target" {
|
||||||
local skill
|
local skill
|
||||||
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
||||||
@@ -502,9 +553,18 @@ EOF
|
|||||||
}
|
}
|
||||||
|
|
||||||
@test "ADR-0020: the boundary check declines rather than false-FAILs when no authoring source is found" {
|
@test "ADR-0020: the boundary check declines rather than false-FAILs when no authoring source is found" {
|
||||||
|
# Deliberately NOT built with make_fixture_tree: this skill sits in a bare
|
||||||
|
# temp directory with no plugins/*/.apm/ above it and no .git, so the resolver
|
||||||
|
# legitimately has no universe. That is a real path (a skill being drafted
|
||||||
|
# outside any repo), and the required behaviour is to DECLINE OUT LOUD rather
|
||||||
|
# than either false-FAIL or pass in silence — silence is what let a whole gate
|
||||||
|
# family go missing unnoticed. So the INFO text and the named unchecked target
|
||||||
|
# are both asserted, not just the absence of a failure.
|
||||||
local skill="$TMPDIR/orphan/my-skill"
|
local skill="$TMPDIR/orphan/my-skill"
|
||||||
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use some-other-skill instead." 10
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use some-other-skill instead." 10
|
||||||
run bash "$SCRIPT" "$skill"
|
run bash "$SCRIPT" "$skill"
|
||||||
assert_success
|
assert_success
|
||||||
refute_output --partial "routes to"
|
refute_output --partial "routes to"
|
||||||
|
assert_output --partial "boundary-target resolution DID NOT RUN"
|
||||||
|
assert_output --partial "Unchecked target(s): some-other-skill"
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
406
tests/test-adr0020-body-checks.sh
Executable file
406
tests/test-adr0020-body-checks.sh
Executable file
@@ -0,0 +1,406 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Regression test for the ADR-0020 body-shape checks and, just as importantly,
|
||||||
|
# for the false-positive fixes each of them needed. Every check here was
|
||||||
|
# completely untested.
|
||||||
|
#
|
||||||
|
# * Gotchas section over 5 entries — SUGGESTION
|
||||||
|
# * Gotchas section over 25% of the body — SUGGESTION
|
||||||
|
# * a references/<file>.md named but absent — ERROR (a broken pointer is not a
|
||||||
|
# style opinion)
|
||||||
|
# * description with no boundary clause — SUGGESTION
|
||||||
|
#
|
||||||
|
# The false-positive half is not optional extra coverage. Each of these checks
|
||||||
|
# scans prose, and the first naive version of each one fired on ordinary writing:
|
||||||
|
# a ```-fenced EXAMPLE of a Gotchas section became the section itself, indented
|
||||||
|
# child bullets were counted as top-level entries, `## Gotcha handling` was read
|
||||||
|
# as the Gotchas section, and a documented-then-removed references/ file became a
|
||||||
|
# hard ERROR. The skills most likely to carry such an example are skill-author and
|
||||||
|
# skill-audit — the two that DOCUMENT these conventions — so a gate that fires on
|
||||||
|
# them is a gate nobody can turn on.
|
||||||
|
#
|
||||||
|
# Every case is a matched pair: the check fires just over its boundary, and stays
|
||||||
|
# silent just under it (or on the shape it must not match). A test asserting only
|
||||||
|
# that a bad file fails proves nothing about a check that fires on everything.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||||
|
PASS=0
|
||||||
|
FAIL=0
|
||||||
|
|
||||||
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||||
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||||
|
|
||||||
|
TMPDIR_T="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||||
|
|
||||||
|
# A description with a boundary clause and no routing target — the neutral
|
||||||
|
# default, so a fixture about Gotchas or references does not also trip the
|
||||||
|
# missing-boundary-clause SUGGESTION and stop isolating what it names.
|
||||||
|
CLEAN_DESC="Use when doing the thing. Do not use for anything else."
|
||||||
|
|
||||||
|
# make_skill <name> <desc> — SKILL.md with the body read from stdin. Echoes the
|
||||||
|
# path. Each skill gets its own directory so references/ fixtures are isolated.
|
||||||
|
make_skill() {
|
||||||
|
local name="$1" desc="$2" dir
|
||||||
|
dir="$TMPDIR_T/$name"
|
||||||
|
mkdir -p "$dir"
|
||||||
|
{
|
||||||
|
echo "---"
|
||||||
|
echo "name: $name"
|
||||||
|
echo "description: $desc"
|
||||||
|
echo "---"
|
||||||
|
cat
|
||||||
|
} > "$dir/SKILL.md"
|
||||||
|
echo "$dir/SKILL.md"
|
||||||
|
}
|
||||||
|
|
||||||
|
# expect <label> <file> <expect: silent|suggests|errors> [needle] [absent-needle]
|
||||||
|
#
|
||||||
|
# `silent` is the strict one: exit 0 AND completely empty output. It is what
|
||||||
|
# makes every "must not fire" case below real — a check that fired with some
|
||||||
|
# other wording would still be caught.
|
||||||
|
expect() {
|
||||||
|
local label="$1" file="$2" mode="$3" needle="${4:-}" absent="${5:-}" out status=0
|
||||||
|
set +e
|
||||||
|
out="$(bash "$HOOK" "$file" 2>&1)"
|
||||||
|
status=$?
|
||||||
|
set -e
|
||||||
|
case "$mode" in
|
||||||
|
silent)
|
||||||
|
if [[ $status -eq 0 && -z "$out" ]]; then
|
||||||
|
pass "$label"
|
||||||
|
else
|
||||||
|
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
suggests)
|
||||||
|
if [[ $status -ne 0 ]]; then
|
||||||
|
fail "$label — a SUGGESTION must never change the exit code (exit $status, output: $out)"
|
||||||
|
elif [[ "$out" != *"SUGGESTION"* || "$out" != *"$needle"* ]]; then
|
||||||
|
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||||
|
elif [[ -n "$absent" && "$out" == *"$absent"* ]]; then
|
||||||
|
fail "$label — output also contained '$absent', which must not fire here: $out"
|
||||||
|
else
|
||||||
|
pass "$label"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
errors)
|
||||||
|
if [[ $status -eq 0 ]]; then
|
||||||
|
fail "$label — expected a hard ERROR, got exit 0 (output: ${out:-<empty>})"
|
||||||
|
elif [[ "$out" != *"$needle"* ]]; then
|
||||||
|
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||||
|
else
|
||||||
|
pass "$label"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
quiet-about)
|
||||||
|
# Exit 0 and the named text absent, but other output permitted. Used where
|
||||||
|
# a second, unrelated finding legitimately fires.
|
||||||
|
if [[ $status -ne 0 ]]; then
|
||||||
|
fail "$label (exit $status, output: $out)"
|
||||||
|
elif [[ "$out" == *"$needle"* ]]; then
|
||||||
|
fail "$label — '$needle' fired when it must not: $out"
|
||||||
|
else
|
||||||
|
pass "$label"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
filler() { python3 -c "print(' '.join(['word'] * $1))"; }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Gotchas: entry count (guideline 5)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The filler after the section keeps the 25% fraction check well clear, so these
|
||||||
|
# two cases isolate the ENTRY count. Without it a six-entry section in a short
|
||||||
|
# body would fire both and the pair would not distinguish them.
|
||||||
|
echo ""
|
||||||
|
echo "--- Gotchas entry count: 5 is fine, 6 is a SUGGESTION ---"
|
||||||
|
F_FIVE="$(make_skill gotchas-five "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- first trap here
|
||||||
|
- second trap here
|
||||||
|
- third trap here
|
||||||
|
- fourth trap here
|
||||||
|
- fifth trap here
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
$(filler 200)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a Gotchas section with exactly 5 entries is silent" "$F_FIVE" silent
|
||||||
|
|
||||||
|
F_SIX="$(make_skill gotchas-six "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- first trap here
|
||||||
|
- second trap here
|
||||||
|
- third trap here
|
||||||
|
- fourth trap here
|
||||||
|
- fifth trap here
|
||||||
|
- sixth trap here
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
$(filler 200)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a Gotchas section with 6 entries raises a SUGGESTION and still exits 0" \
|
||||||
|
"$F_SIX" suggests "Gotchas section has 6 entries" "over the 25% guideline"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Gotchas: share of the body (guideline 25%)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Exact boundary arithmetic, not an approximation. The section is prose (no list
|
||||||
|
# items) so the entry check cannot fire and confuse the result; body words are
|
||||||
|
# then section + filler + the two two-word headings. At a body of 100 words a
|
||||||
|
# 25-word section is exactly the guideline (inclusive — `>` is the comparison, so
|
||||||
|
# it passes) and a 26-word section is one word past it.
|
||||||
|
echo ""
|
||||||
|
echo "--- Gotchas share of body: exactly 25% is fine, 26% is a SUGGESTION ---"
|
||||||
|
F_AT="$(make_skill gotchas-at-fraction "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
$(filler 25)
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
$(filler 71)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a Gotchas section at exactly 25% of the body is silent" "$F_AT" silent
|
||||||
|
|
||||||
|
F_OVER="$(make_skill gotchas-over-fraction "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
$(filler 26)
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
$(filler 70)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a Gotchas section at 26% of the body raises a SUGGESTION" \
|
||||||
|
"$F_OVER" suggests "Gotchas section is 26 of 100 body words (26%)" "entries"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Gotchas: false positives
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "--- a ## Gotchas heading inside a fenced block is not the Gotchas section ---"
|
||||||
|
# skill-author and skill-audit both document this convention by showing it. If a
|
||||||
|
# fenced example counted, the two skills that define the rule would be the two
|
||||||
|
# most likely to fail it.
|
||||||
|
F_FENCED_HEADING="$(make_skill gotchas-fenced-heading "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## How to write one
|
||||||
|
|
||||||
|
\`\`\`markdown
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- example one
|
||||||
|
- example two
|
||||||
|
- example three
|
||||||
|
- example four
|
||||||
|
- example five
|
||||||
|
- example six
|
||||||
|
- example seven
|
||||||
|
\`\`\`
|
||||||
|
|
||||||
|
$(filler 200)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a fenced ## Gotchas heading is not read as the section" \
|
||||||
|
"$F_FENCED_HEADING" quiet-about "Gotchas section"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- list items inside a fenced block do not count as Gotchas entries ---"
|
||||||
|
F_FENCED_ENTRIES="$(make_skill gotchas-fenced-entries "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- a real trap
|
||||||
|
- another real trap
|
||||||
|
|
||||||
|
Shown as an example of what NOT to write:
|
||||||
|
|
||||||
|
\`\`\`markdown
|
||||||
|
- fake one
|
||||||
|
- fake two
|
||||||
|
- fake three
|
||||||
|
- fake four
|
||||||
|
- fake five
|
||||||
|
- fake six
|
||||||
|
- fake seven
|
||||||
|
- fake eight
|
||||||
|
\`\`\`
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
$(filler 300)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "eight fenced bullets plus two real ones counts as two entries, not ten" \
|
||||||
|
"$F_FENCED_ENTRIES" quiet-about "Gotchas section has"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- the heading must END in gotcha(s): '## Gotcha handling' is not the section ---"
|
||||||
|
# `## Gotcha handling` and `## Why gotchas matter` are prose sections. Treating
|
||||||
|
# one as the Gotchas section measures a span that was never a gotcha list.
|
||||||
|
F_HANDLING="$(make_skill gotcha-handling "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## Gotcha handling
|
||||||
|
|
||||||
|
- item one
|
||||||
|
- item two
|
||||||
|
- item three
|
||||||
|
- item four
|
||||||
|
- item five
|
||||||
|
- item six
|
||||||
|
- item seven
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
$(filler 200)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "'## Gotcha handling' is not matched as the Gotchas section" \
|
||||||
|
"$F_HANDLING" quiet-about "Gotchas section"
|
||||||
|
|
||||||
|
# The control for the three cases above. Without it, "no Gotchas finding" could
|
||||||
|
# equally mean the whole check is dead, and all three would still be green.
|
||||||
|
F_CONTROL="$(make_skill gotchas-control "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
## Common gotchas
|
||||||
|
|
||||||
|
- item one
|
||||||
|
- item two
|
||||||
|
- item three
|
||||||
|
- item four
|
||||||
|
- item five
|
||||||
|
- item six
|
||||||
|
- item seven
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
$(filler 200)
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "control: a real '## Common gotchas' heading with 7 entries IS matched" \
|
||||||
|
"$F_CONTROL" suggests "Gotchas section has 7 entries"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# references/<file>.md pointers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "--- a references/ pointer that is not on disk is a hard ERROR ---"
|
||||||
|
F_REF_MISSING="$(make_skill ref-missing "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
If the caller needs the long form, read references/nowhere.md first.
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a body pointing at an absent references/nowhere.md ERRORs" \
|
||||||
|
"$F_REF_MISSING" errors "points at references/nowhere.md"
|
||||||
|
|
||||||
|
F_REF_PRESENT="$(make_skill ref-present "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
If the caller needs the long form, read references/here.md first.
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
mkdir -p "$TMPDIR_T/ref-present/references"
|
||||||
|
echo "content" > "$TMPDIR_T/ref-present/references/here.md"
|
||||||
|
expect "the same pointer is silent once the file exists" "$F_REF_PRESENT" silent
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- a references/ pointer inside a fenced block is not a dispatch entry ---"
|
||||||
|
F_REF_FENCED="$(make_skill ref-fenced "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
Dispatch tables look like this:
|
||||||
|
|
||||||
|
\`\`\`markdown
|
||||||
|
If X, read references/example-file.md.
|
||||||
|
\`\`\`
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a fenced references/example-file.md does not ERROR" "$F_REF_FENCED" silent
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- a references/ pointer in a same-line removal context is history, not dispatch ---"
|
||||||
|
# Narrow on purpose: a live dispatch table never describes its own target as
|
||||||
|
# removed, so the exemption costs no recall. Each phrasing is checked separately
|
||||||
|
# because they are separate alternatives in one regex, and a typo in any one of
|
||||||
|
# them turns ordinary prose back into a hard ERROR.
|
||||||
|
#
|
||||||
|
# The file names are deliberately NEUTRAL (detail-a.md, not gone-a.md). An
|
||||||
|
# earlier draft of this block named them gone-*.md and every case passed for the
|
||||||
|
# wrong reason: "gone" is itself one of the removal words, so the exemption fired
|
||||||
|
# off the FILENAME and the phrase under test was never exercised. The control
|
||||||
|
# below is what surfaced that — it is the assertion that keeps these five honest.
|
||||||
|
i=0
|
||||||
|
for phrase in \
|
||||||
|
"The old references/detail-a.md was removed in v2." \
|
||||||
|
"references/detail-b.md is no longer part of this skill." \
|
||||||
|
"references/detail-c.md is deprecated and should not be read." \
|
||||||
|
"references/detail-d.md was renamed, so nothing points at it now." \
|
||||||
|
"references/detail-e.md has been superseded by the body itself."; do
|
||||||
|
i=$((i + 1))
|
||||||
|
F_REF_PAST="$(make_skill "ref-past-$i" "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
$phrase
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "removal-context pointer is not an ERROR: \"$phrase\"" "$F_REF_PAST" silent
|
||||||
|
done
|
||||||
|
|
||||||
|
# The control: the SAME sentence shape without a removal word must still ERROR,
|
||||||
|
# or the exemption above has swallowed the check rather than narrowed it.
|
||||||
|
F_REF_LIVE="$(make_skill ref-live "$CLEAN_DESC" <<EOF
|
||||||
|
|
||||||
|
The details live in references/detail-a.md, which the agent should read first.
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "control: the same pointer with no removal word still ERRORs" \
|
||||||
|
"$F_REF_LIVE" errors "points at references/detail-a.md"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Missing boundary clause
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "--- a description with no boundary clause raises a SUGGESTION ---"
|
||||||
|
F_NO_BOUNDARY="$(make_skill no-boundary "Use when the user wants the thing done." <<EOF
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "a description with no boundary clause raises a SUGGESTION and still exits 0" \
|
||||||
|
"$F_NO_BOUNDARY" suggests "description has no boundary clause"
|
||||||
|
|
||||||
|
# Both accepted shapes, asserted separately: the prose markers and ADR-0020's
|
||||||
|
# compressed arrow form. Dropping either from the detector would leave the other
|
||||||
|
# green.
|
||||||
|
F_PROSE_BOUNDARY="$(make_skill prose-boundary "Use when the user wants the thing done. Do not use for anything else." <<EOF
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "the prose boundary form satisfies the check" "$F_PROSE_BOUNDARY" silent
|
||||||
|
|
||||||
|
F_ARROW_BOUNDARY="$(make_skill arrow-boundary "Use when the user wants the thing done. Not the other thing -> sibling-skill." <<EOF
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
)"
|
||||||
|
expect "ADR-0020's compressed 'Not X -> y' form satisfies the check" \
|
||||||
|
"$F_ARROW_BOUNDARY" quiet-about "no boundary clause"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Results: $PASS passed, $FAIL failed"
|
||||||
|
[[ $FAIL -eq 0 ]]
|
||||||
331
tests/test-adr0020-contract.sh
Executable file
331
tests/test-adr0020-contract.sh
Executable file
@@ -0,0 +1,331 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Regression test for the three STRUCTURAL claims the ADR-0020 gate family makes
|
||||||
|
# about itself. None of them was pinned anywhere before this file, and each one
|
||||||
|
# fails silently — which is the whole reason they need a test rather than a
|
||||||
|
# comment:
|
||||||
|
#
|
||||||
|
# 1. "ONE resolver, embedded VERBATIM in three scripts." The block between the
|
||||||
|
# BEGIN/END markers is copied, not imported, because a cache-installed
|
||||||
|
# plugin's scripts cannot read files outside their own plugin directory.
|
||||||
|
# Nothing but this file asserts the three copies are still identical, and a
|
||||||
|
# one-line edit to a single copy is invisible: every constant-agreement
|
||||||
|
# assertion in tests/test-skill-size-check.sh still passes, because the
|
||||||
|
# CONSTANTS are not what drifted.
|
||||||
|
# 2. Both interpreter preflights, in all three scripts. python3 and PyYAML are
|
||||||
|
# declared HARD dependencies precisely so a missing one cannot turn into a
|
||||||
|
# vacuous pass, and the two are checked separately so the message names the
|
||||||
|
# thing to install rather than the wrong one.
|
||||||
|
# 3. `verbose: true` on the skill-size-check hook. It is the ENTIRE delivery
|
||||||
|
# mechanism for the SUGGESTION tier: pre-commit prints nothing at all for a
|
||||||
|
# passing hook, and a SUGGESTION deliberately does not fail, so dropping
|
||||||
|
# one word from the config silences the tier ADR-0020 depends on while
|
||||||
|
# every test and every hook still reports green.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||||
|
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||||
|
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||||
|
PASS=0
|
||||||
|
FAIL=0
|
||||||
|
|
||||||
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||||
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||||
|
|
||||||
|
TMPDIR_T="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||||
|
|
||||||
|
BEGIN_MARKER='# ===== BEGIN ADR-0020 SHARED BOUNDARY RESOLVER ====='
|
||||||
|
END_MARKER='# ===== END ADR-0020 SHARED BOUNDARY RESOLVER ====='
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. The shared resolver block is byte-identical in all three scripts
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "--- the ADR-0020 shared resolver block is byte-identical in all three scripts ---"
|
||||||
|
|
||||||
|
# Marker discipline first. An unbalanced or duplicated marker pair makes the
|
||||||
|
# extraction below silently measure the wrong span — a sed range that never
|
||||||
|
# closes swallows the rest of the file, and one that opens twice concatenates
|
||||||
|
# two spans. Both would still compare "equal" if all three were mangled the
|
||||||
|
# same way, so the shape is asserted before the contents.
|
||||||
|
MARKERS_OK=true
|
||||||
|
for f in "$HOOK" "$SKILL_VALIDATE" "$AGENT_VALIDATE"; do
|
||||||
|
if [[ ! -f "$f" ]]; then
|
||||||
|
fail "script not found: $f"
|
||||||
|
MARKERS_OK=false
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
b="$(grep -cFx "$BEGIN_MARKER" "$f" || true)"
|
||||||
|
e="$(grep -cFx "$END_MARKER" "$f" || true)"
|
||||||
|
if [[ "$b" == "1" && "$e" == "1" ]]; then
|
||||||
|
pass "${f#"$REPO_ROOT/"} carries exactly one BEGIN and one END marker"
|
||||||
|
else
|
||||||
|
fail "${f#"$REPO_ROOT/"} has $b BEGIN and $e END markers, expected 1 and 1"
|
||||||
|
MARKERS_OK=false
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ! $MARKERS_OK; then
|
||||||
|
fail "skipping the byte-identity comparison — the marker pairs are not well-formed, so any extraction would measure the wrong span"
|
||||||
|
else
|
||||||
|
HASHES=()
|
||||||
|
LINECOUNTS=()
|
||||||
|
for f in "$HOOK" "$SKILL_VALIDATE" "$AGENT_VALIDATE"; do
|
||||||
|
out="$TMPDIR_T/block-$(echo "$f" | md5sum | cut -c1-8).txt"
|
||||||
|
sed -n "/^${BEGIN_MARKER}\$/,/^${END_MARKER}\$/p" "$f" > "$out"
|
||||||
|
HASHES+=("$(md5sum < "$out" | cut -d' ' -f1)")
|
||||||
|
LINECOUNTS+=("$(wc -l < "$out" | tr -d ' ')")
|
||||||
|
done
|
||||||
|
if [[ "${HASHES[0]}" == "${HASHES[1]}" && "${HASHES[1]}" == "${HASHES[2]}" ]]; then
|
||||||
|
pass "all three copies hash to ${HASHES[0]} (${LINECOUNTS[0]} lines) — agreement by construction, not by coincidence"
|
||||||
|
else
|
||||||
|
fail "the shared resolver has DRIFTED: skill-size-check=${HASHES[0]} (${LINECOUNTS[0]} lines), skill-audit=${HASHES[1]} (${LINECOUNTS[1]} lines), agent-audit=${HASHES[2]} (${LINECOUNTS[2]} lines). Edit one copy, then paste it over the other two."
|
||||||
|
fi
|
||||||
|
# A block that has been emptied out would hash equal in all three and pass the
|
||||||
|
# comparison above while enforcing nothing. The resolver is ~570 lines; 100 is
|
||||||
|
# a floor low enough never to need maintenance and high enough that a gutted
|
||||||
|
# block cannot sneak past.
|
||||||
|
if [[ "${LINECOUNTS[0]}" -gt 100 ]]; then
|
||||||
|
pass "the extracted block is ${LINECOUNTS[0]} lines — the comparison is over real content, not an empty span"
|
||||||
|
else
|
||||||
|
fail "the extracted shared block is only ${LINECOUNTS[0]} lines — three identical empty spans would compare equal and assert nothing"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. Both interpreter preflights, in all three scripts
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The two are checked separately on purpose: `python3 -c 'import yaml'` fails
|
||||||
|
# identically whether python3 is missing or PyYAML is, and naming the wrong one
|
||||||
|
# sends the reader to install the wrong thing.
|
||||||
|
REAL_PYTHON="$(command -v python3)"
|
||||||
|
# Absolute path, deliberately. The no-python3 fixture below replaces PATH
|
||||||
|
# wholesale, so a bare `bash` (or `/usr/bin/env bash`) would be resolved against
|
||||||
|
# that stripped PATH and die with "No such file or directory" before the script
|
||||||
|
# under test ever starts -- a 127 that looks like the preflight firing.
|
||||||
|
BASH_BIN="$(command -v bash)"
|
||||||
|
|
||||||
|
# A PATH that genuinely has no python3 on it. Built by symlinking the handful of
|
||||||
|
# binaries the three scripts touch before their own preflight rather than by
|
||||||
|
# hiding python3 from a full PATH, because there is no portable way to subtract
|
||||||
|
# one entry from a directory. `bash` is invoked by absolute path below so the
|
||||||
|
# interpreter itself does not have to be on this PATH.
|
||||||
|
NOPY_BIN="$TMPDIR_T/nopython-bin"
|
||||||
|
mkdir -p "$NOPY_BIN"
|
||||||
|
for b in awk cat cut dirname basename grep sed pwd rm mkdir tr; do
|
||||||
|
src="$(command -v "$b" 2>/dev/null || true)"
|
||||||
|
[[ -n "$src" ]] && ln -sf "$src" "$NOPY_BIN/$b"
|
||||||
|
done
|
||||||
|
|
||||||
|
# A python3 that runs but cannot import yaml. A shim on PATH re-execs the real
|
||||||
|
# interpreter with a PYTHONPATH entry holding a `yaml` module that raises on
|
||||||
|
# import; PYTHONPATH precedes site-packages on sys.path, so it shadows a real
|
||||||
|
# PyYAML install without touching it.
|
||||||
|
SHADOW="$TMPDIR_T/shadow"
|
||||||
|
mkdir -p "$SHADOW"
|
||||||
|
printf 'raise ImportError("PyYAML deliberately unavailable in this fixture")\n' \
|
||||||
|
> "$SHADOW/yaml.py"
|
||||||
|
NOYAML_BIN="$TMPDIR_T/noyaml-bin"
|
||||||
|
mkdir -p "$NOYAML_BIN"
|
||||||
|
cat > "$NOYAML_BIN/python3" <<EOF
|
||||||
|
#!/bin/sh
|
||||||
|
PYTHONPATH="$SHADOW\${PYTHONPATH:+:\$PYTHONPATH}" exec "$REAL_PYTHON" "\$@"
|
||||||
|
EOF
|
||||||
|
chmod +x "$NOYAML_BIN/python3"
|
||||||
|
|
||||||
|
# Sanity-check the two fixtures themselves before trusting any verdict they
|
||||||
|
# produce. A shim that silently still imports yaml would make every PyYAML
|
||||||
|
# assertion below pass for the wrong reason.
|
||||||
|
if PATH="$NOYAML_BIN:$PATH" python3 -c 'import yaml' 2>/dev/null; then
|
||||||
|
fail "the no-PyYAML shim does not actually shadow PyYAML — every PyYAML assertion below would be vacuous"
|
||||||
|
else
|
||||||
|
pass "fixture check: the no-PyYAML shim makes 'import yaml' fail while python3 still runs"
|
||||||
|
fi
|
||||||
|
if PATH="$NOPY_BIN" command -v python3 > /dev/null 2>&1; then
|
||||||
|
fail "the no-python3 PATH still resolves python3 — every python3 assertion below would be vacuous"
|
||||||
|
else
|
||||||
|
pass "fixture check: the no-python3 PATH resolves no python3"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# A minimal, entirely clean subject for each script. The preflight must fire
|
||||||
|
# before any measurement, so the subject's own content is irrelevant — which is
|
||||||
|
# exactly what makes a clean one the right choice: nothing else can produce the
|
||||||
|
# non-zero exit these cases assert.
|
||||||
|
SUBJECT_SKILL_DIR="$TMPDIR_T/subject/my-skill"
|
||||||
|
mkdir -p "$SUBJECT_SKILL_DIR"
|
||||||
|
cat > "$SUBJECT_SKILL_DIR/SKILL.md" <<'EOF'
|
||||||
|
---
|
||||||
|
name: my-skill
|
||||||
|
description: A short valid description. Do not use for anything else.
|
||||||
|
---
|
||||||
|
|
||||||
|
Do the thing.
|
||||||
|
EOF
|
||||||
|
SUBJECT_AGENT_ROOT="$TMPDIR_T/subject-agent"
|
||||||
|
mkdir -p "$SUBJECT_AGENT_ROOT/.apm/agents"
|
||||||
|
cat > "$SUBJECT_AGENT_ROOT/apm.yml" <<'EOF'
|
||||||
|
name: test-package
|
||||||
|
version: 0.1.0
|
||||||
|
type: skill
|
||||||
|
EOF
|
||||||
|
cat > "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md" <<'EOF'
|
||||||
|
---
|
||||||
|
name: my-agent
|
||||||
|
description: A short valid description. Do not use for anything else.
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a test agent. When invoked, do the thing.
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# probe_preflight <label> <env-kind: nopython|noyaml> <expect-needle> <cmd...>
|
||||||
|
probe_preflight() {
|
||||||
|
local label="$1" kind="$2" needle="$3"
|
||||||
|
shift 3
|
||||||
|
local out status=0
|
||||||
|
set +e
|
||||||
|
if [[ "$kind" == nopython ]]; then
|
||||||
|
out="$(env -i PATH="$NOPY_BIN" HOME="$HOME" "$BASH_BIN" "$@" 2>&1)"
|
||||||
|
else
|
||||||
|
out="$(env PATH="$NOYAML_BIN:$PATH" "$BASH_BIN" "$@" 2>&1)"
|
||||||
|
fi
|
||||||
|
status=$?
|
||||||
|
set -e
|
||||||
|
if [[ $status -eq 0 ]]; then
|
||||||
|
fail "$label exited 0 — a missing hard dependency became a vacuous pass (output: ${out:-<empty>})"
|
||||||
|
elif [[ "$out" != *"$needle"* ]]; then
|
||||||
|
# The needle is the DIAGNOSTIC ("python3 is required"), not the bare word.
|
||||||
|
# Deleting the preflight entirely would still produce a non-zero exit and a
|
||||||
|
# message mentioning python3 -- bash's own "python3: command not found" --
|
||||||
|
# so a bare-word needle would go green on a script with no preflight at all.
|
||||||
|
fail "$label exited $status but never produced the '$needle' diagnostic (output: ${out:-<empty>})"
|
||||||
|
elif [[ "$kind" == nopython && "$out" == *PyYAML* ]]; then
|
||||||
|
fail "$label reported PyYAML when python3 itself is missing — that sends the reader to install the wrong thing (output: $out)"
|
||||||
|
else
|
||||||
|
pass "$label"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- a PATH with no python3 is a hard failure in all three scripts, naming python3 ---"
|
||||||
|
probe_preflight "scripts/skill-size-check.sh reports missing python3" \
|
||||||
|
nopython "python3 is required" \
|
||||||
|
"$HOOK" "$SUBJECT_SKILL_DIR/SKILL.md"
|
||||||
|
probe_preflight "skill-audit/scripts/validate.sh reports missing python3" \
|
||||||
|
nopython "python3 is required" \
|
||||||
|
"$SKILL_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||||
|
probe_preflight "agent-audit/scripts/validate.sh reports missing python3" \
|
||||||
|
nopython "python3 is required" \
|
||||||
|
"$AGENT_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- a python3 that cannot import yaml is a hard failure in all three scripts, naming PyYAML ---"
|
||||||
|
probe_preflight "scripts/skill-size-check.sh reports missing PyYAML" \
|
||||||
|
noyaml "PyYAML is required" \
|
||||||
|
"$HOOK" "$SUBJECT_SKILL_DIR/SKILL.md"
|
||||||
|
probe_preflight "skill-audit/scripts/validate.sh reports missing PyYAML" \
|
||||||
|
noyaml "PyYAML is required" \
|
||||||
|
"$SKILL_VALIDATE" "$SUBJECT_SKILL_DIR"
|
||||||
|
probe_preflight "agent-audit/scripts/validate.sh reports missing PyYAML" \
|
||||||
|
noyaml "PyYAML is required" \
|
||||||
|
"$AGENT_VALIDATE" "$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"
|
||||||
|
|
||||||
|
# The control. Without it, "fails when the dependency is missing" is satisfied by
|
||||||
|
# a script that fails unconditionally, and the two cases above would be green on
|
||||||
|
# a gate that never runs at all.
|
||||||
|
echo ""
|
||||||
|
echo "--- control: with both dependencies present the same subjects pass ---"
|
||||||
|
for probe in "$HOOK:$SUBJECT_SKILL_DIR/SKILL.md" \
|
||||||
|
"$SKILL_VALIDATE:$SUBJECT_SKILL_DIR" \
|
||||||
|
"$AGENT_VALIDATE:$SUBJECT_AGENT_ROOT/.apm/agents/my-agent.agent.md"; do
|
||||||
|
script="${probe%%:*}"
|
||||||
|
arg="${probe#*:}"
|
||||||
|
set +e
|
||||||
|
ctl_out="$(bash "$script" "$arg" 2>&1)"
|
||||||
|
ctl_rc=$?
|
||||||
|
set -e
|
||||||
|
if [[ $ctl_rc -eq 0 ]]; then
|
||||||
|
pass "${script#"$REPO_ROOT/"} exits 0 on a clean subject with python3 and PyYAML available"
|
||||||
|
else
|
||||||
|
fail "${script#"$REPO_ROOT/"} failed a clean subject (exit $ctl_rc): $ctl_out"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. verbose: true on the skill-size-check hook, in BOTH manifests
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# .pre-commit-config.yaml governs this repo; .pre-commit-hooks.yaml is what a
|
||||||
|
# CONSUMER repo gets when it points at this one. Dropping the flag from either
|
||||||
|
# silences the SUGGESTION tier for that audience alone, which is the hardest
|
||||||
|
# version of the defect to notice.
|
||||||
|
echo ""
|
||||||
|
echo "--- the skill-size-check hook declares verbose: true in both manifests ---"
|
||||||
|
VERBOSE_REPORT="$(python3 - "$REPO_ROOT" <<'PY'
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
root = sys.argv[1]
|
||||||
|
|
||||||
|
|
||||||
|
def emit(status, msg):
|
||||||
|
print("%s\t%s" % (status, msg))
|
||||||
|
|
||||||
|
|
||||||
|
# Repo config: nested repos[].hooks[].
|
||||||
|
path = os.path.join(root, '.pre-commit-config.yaml')
|
||||||
|
try:
|
||||||
|
with open(path, encoding='utf-8') as fh:
|
||||||
|
cfg = yaml.safe_load(fh) or {}
|
||||||
|
except Exception as exc:
|
||||||
|
emit('FAIL', '.pre-commit-config.yaml did not parse: %s' % exc)
|
||||||
|
cfg = {}
|
||||||
|
found = None
|
||||||
|
for repo in cfg.get('repos') or []:
|
||||||
|
for hook in (repo.get('hooks') or []):
|
||||||
|
if hook.get('id') == 'skill-size-check':
|
||||||
|
found = hook
|
||||||
|
if found is None:
|
||||||
|
emit('FAIL', '.pre-commit-config.yaml declares no hook with id skill-size-check')
|
||||||
|
elif found.get('verbose') is True:
|
||||||
|
emit('PASS', '.pre-commit-config.yaml: skill-size-check is verbose: true')
|
||||||
|
else:
|
||||||
|
emit('FAIL', '.pre-commit-config.yaml: skill-size-check has verbose=%r — '
|
||||||
|
'pre-commit prints nothing for a passing hook, so every '
|
||||||
|
'ADR-0020 SUGGESTION is swallowed' % (found.get('verbose'),))
|
||||||
|
|
||||||
|
# Consumer manifest: a flat list of hooks.
|
||||||
|
path = os.path.join(root, '.pre-commit-hooks.yaml')
|
||||||
|
try:
|
||||||
|
with open(path, encoding='utf-8') as fh:
|
||||||
|
hooks = yaml.safe_load(fh) or []
|
||||||
|
except Exception as exc:
|
||||||
|
emit('FAIL', '.pre-commit-hooks.yaml did not parse: %s' % exc)
|
||||||
|
hooks = []
|
||||||
|
found = None
|
||||||
|
for hook in hooks:
|
||||||
|
if isinstance(hook, dict) and hook.get('id') == 'kyberforge-skill-size-check':
|
||||||
|
found = hook
|
||||||
|
if found is None:
|
||||||
|
emit('FAIL', '.pre-commit-hooks.yaml declares no hook with id kyberforge-skill-size-check')
|
||||||
|
elif found.get('verbose') is True:
|
||||||
|
emit('PASS', '.pre-commit-hooks.yaml: kyberforge-skill-size-check is verbose: true')
|
||||||
|
else:
|
||||||
|
emit('FAIL', '.pre-commit-hooks.yaml: kyberforge-skill-size-check has verbose=%r — '
|
||||||
|
'a consumer repo would never see the SUGGESTION tier'
|
||||||
|
% (found.get('verbose'),))
|
||||||
|
PY
|
||||||
|
)"
|
||||||
|
while IFS=$'\t' read -r status msg; do
|
||||||
|
[[ -n "$status" ]] || continue
|
||||||
|
if [[ "$status" == PASS ]]; then
|
||||||
|
pass "$msg"
|
||||||
|
else
|
||||||
|
fail "$msg"
|
||||||
|
fi
|
||||||
|
done <<< "$VERBOSE_REPORT"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Results: $PASS passed, $FAIL failed"
|
||||||
|
[[ $FAIL -eq 0 ]]
|
||||||
365
tests/test-adr0020-differential.sh
Executable file
365
tests/test-adr0020-differential.sh
Executable file
@@ -0,0 +1,365 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Differential test: scripts/skill-size-check.sh (the pre-commit hook) and
|
||||||
|
# skill-audit/scripts/validate.sh (the in-skill auditor) must reach the SAME
|
||||||
|
# ADR-0020 verdict on the same file.
|
||||||
|
#
|
||||||
|
# Why this exists as a separate suite. tests/test-skill-size-check.sh already
|
||||||
|
# asserts the two agree on their CONSTANTS, and that assertion is necessary but
|
||||||
|
# demonstrably not sufficient: a previous review found the two scripts disagreeing
|
||||||
|
# on real files while every constant matched perfectly. Constants are one of the
|
||||||
|
# ways two hand-duplicated implementations diverge; comparison operators, message
|
||||||
|
# wording, which value gets measured, and which branch runs first are the others,
|
||||||
|
# and none of them is visible to a constant check.
|
||||||
|
#
|
||||||
|
# The consequence of divergence is specific and bad: skill-audit reports a skill
|
||||||
|
# ready to ship and the commit hook then rejects it, or worse, the reverse. So the
|
||||||
|
# comparison here is over VERDICTS on files, not over source text.
|
||||||
|
#
|
||||||
|
# Scope: the ADR-0020 axes the two scripts share — description length and tier,
|
||||||
|
# body word count and tier, dangling routing targets, missing references/
|
||||||
|
# pointers, the two Gotchas suggestions, the missing-boundary-clause suggestion,
|
||||||
|
# a declined resolution, and an empty description. The two scripts legitimately
|
||||||
|
# differ elsewhere (validate.sh also checks name/directory agreement, script
|
||||||
|
# executability and the 1024-char spec backstop; the hook checks whole-file lines
|
||||||
|
# and words), and those lines are ignored rather than being forced into a shared
|
||||||
|
# shape they were never meant to have.
|
||||||
|
#
|
||||||
|
# Run over the real 39-skill corpus AND over purpose-built fixtures that sit ON
|
||||||
|
# each boundary. The corpus alone is not enough — it happens not to contain a
|
||||||
|
# file at exactly 900 body words, which is precisely where an inclusive/exclusive
|
||||||
|
# comparison mismatch would hide.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||||
|
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||||
|
|
||||||
|
TMPDIR_T="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Fixtures: one per ADR-0020 axis, placed ON the boundary wherever there is one.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Built inside a synthetic plugin monorepo so boundary-target resolution actually
|
||||||
|
# runs for both scripts (in a bare temp dir both would decline, and "both
|
||||||
|
# declined" is agreement about nothing).
|
||||||
|
FIXTURE_ROOT="$TMPDIR_T/fixtures"
|
||||||
|
FX="$FIXTURE_ROOT/plugins/fixture-plugin/.apm/skills"
|
||||||
|
mkdir -p "$FX/sibling-skill" "$FIXTURE_ROOT/plugins/fixture-plugin/.apm/agents"
|
||||||
|
|
||||||
|
make_fx() {
|
||||||
|
local name="$1" desc="$2" body_words="$3"
|
||||||
|
mkdir -p "$FX/$name"
|
||||||
|
{
|
||||||
|
echo "---"
|
||||||
|
echo "name: $name"
|
||||||
|
echo "description: $desc"
|
||||||
|
echo "---"
|
||||||
|
echo ""
|
||||||
|
python3 -c "print(' '.join(['word'] * $body_words))"
|
||||||
|
} > "$FX/$name/SKILL.md"
|
||||||
|
}
|
||||||
|
desc_of_length() {
|
||||||
|
python3 - "$1" <<'PY'
|
||||||
|
import sys
|
||||||
|
n = int(sys.argv[1])
|
||||||
|
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||||
|
print(prefix + 'x' * (n - len(prefix)))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
CLEAN="Use when doing the thing. Do not use for anything else."
|
||||||
|
|
||||||
|
# Description tier boundaries, both sides of both thresholds.
|
||||||
|
make_fx desc-249 "$(desc_of_length 249)" 10
|
||||||
|
make_fx desc-250 "$(desc_of_length 250)" 10
|
||||||
|
make_fx desc-251 "$(desc_of_length 251)" 10
|
||||||
|
make_fx desc-400 "$(desc_of_length 400)" 10
|
||||||
|
make_fx desc-401 "$(desc_of_length 401)" 10
|
||||||
|
# Body tier boundaries, both sides of both thresholds.
|
||||||
|
make_fx body-599 "$CLEAN" 599
|
||||||
|
make_fx body-600 "$CLEAN" 600
|
||||||
|
make_fx body-601 "$CLEAN" 601
|
||||||
|
make_fx body-900 "$CLEAN" 900
|
||||||
|
make_fx body-901 "$CLEAN" 901
|
||||||
|
# Folding: the value has to be measured after YAML folding in both scripts.
|
||||||
|
mkdir -p "$FX/folded-desc"
|
||||||
|
{
|
||||||
|
echo "---"
|
||||||
|
echo "name: folded-desc"
|
||||||
|
echo "description: >"
|
||||||
|
python3 -c "print('\n'.join([' ' + 'x' * 40] * 11))"
|
||||||
|
echo "---"
|
||||||
|
echo ""
|
||||||
|
echo "Do the thing."
|
||||||
|
} > "$FX/folded-desc/SKILL.md"
|
||||||
|
# Routing targets, one per tier the resolver can produce: resolves, dangles with
|
||||||
|
# in-sentence corroboration (ERROR/FAIL), dangles alone (SUGGESTION on both
|
||||||
|
# sides), route notation (ERROR/FAIL without corroboration), attributive
|
||||||
|
# (silent). Each tier is here because the two scripts have to agree on the TIER,
|
||||||
|
# not merely on the finding — a copy that promoted or demoted one of them would
|
||||||
|
# otherwise pass this comparison.
|
||||||
|
make_fx target-resolves "Use when doing the thing. Do not use for the other thing — use sibling-skill instead." 10
|
||||||
|
make_fx target-dangles "Use when doing the thing. Do not use for the other thing — use sibling-skill or no-such-skill instead." 10
|
||||||
|
make_fx target-dangles-lone "Use when doing the thing. Do not use for the other thing — use no-such-lone-skill instead." 10
|
||||||
|
make_fx target-dangles-notation "Use when doing the thing. Do not use for the other thing — use /no-such-notation-skill instead." 10
|
||||||
|
make_fx target-attributive "Use when doing the thing. Use pre-commit hooks instead of ad-hoc scripts." 10
|
||||||
|
# No boundary clause at all.
|
||||||
|
make_fx no-boundary "Use when the user wants the thing done." 10
|
||||||
|
# A missing references/ pointer, and a present one.
|
||||||
|
make_fx ref-missing "$CLEAN" 10
|
||||||
|
printf '\nIf the caller needs detail, read references/absent.md first.\n' >> "$FX/ref-missing/SKILL.md"
|
||||||
|
make_fx ref-present "$CLEAN" 10
|
||||||
|
printf '\nIf the caller needs detail, read references/there.md first.\n' >> "$FX/ref-present/SKILL.md"
|
||||||
|
mkdir -p "$FX/ref-present/references"
|
||||||
|
echo "detail" > "$FX/ref-present/references/there.md"
|
||||||
|
# Gotchas, over each guideline.
|
||||||
|
make_fx gotchas-many "$CLEAN" 0
|
||||||
|
cat >> "$FX/gotchas-many/SKILL.md" <<'EOF'
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
- one trap here
|
||||||
|
- two trap here
|
||||||
|
- three trap here
|
||||||
|
- four trap here
|
||||||
|
- five trap here
|
||||||
|
- six trap here
|
||||||
|
- seven trap here
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
EOF
|
||||||
|
python3 -c "print(' '.join(['word'] * 200))" >> "$FX/gotchas-many/SKILL.md"
|
||||||
|
# Gotchas over the 25% body-fraction guideline. Prose, not list items, so the
|
||||||
|
# entry guideline cannot fire and the two suggestions stay separable: 26 section
|
||||||
|
# words in a 100-word body is one word past the threshold.
|
||||||
|
make_fx gotchas-fraction "$CLEAN" 0
|
||||||
|
{
|
||||||
|
echo ""
|
||||||
|
echo "## Gotchas"
|
||||||
|
echo ""
|
||||||
|
python3 -c "print(' '.join(['word'] * 26))"
|
||||||
|
echo ""
|
||||||
|
echo "## Notes"
|
||||||
|
echo ""
|
||||||
|
python3 -c "print(' '.join(['word'] * 70))"
|
||||||
|
} >> "$FX/gotchas-fraction/SKILL.md"
|
||||||
|
|
||||||
|
# Empty description — the shape that used to exit 0 in silence.
|
||||||
|
mkdir -p "$FX/empty-desc"
|
||||||
|
printf -- '---\nname: empty-desc\ndescription:\nmodel: sonnet\n---\n\nDo the thing.\n' \
|
||||||
|
> "$FX/empty-desc/SKILL.md"
|
||||||
|
# Every boundary shape at once, so a divergence that only appears when several
|
||||||
|
# findings fire together is not missed.
|
||||||
|
make_fx combined "$(desc_of_length 401)" 901
|
||||||
|
printf '\nIf the caller needs detail, read references/absent.md first.\n' >> "$FX/combined/SKILL.md"
|
||||||
|
|
||||||
|
# A skill with routing targets and NO authoring root above it — deliberately
|
||||||
|
# OUTSIDE the fixture plugin tree. Both scripts must decline out loud, and both
|
||||||
|
# must decline identically; "both declined" is only meaningful as agreement if
|
||||||
|
# the declining path is exercised on purpose somewhere.
|
||||||
|
ORPHAN_ROOT="$TMPDIR_T/orphan"
|
||||||
|
mkdir -p "$ORPHAN_ROOT/no-universe"
|
||||||
|
{
|
||||||
|
echo "---"
|
||||||
|
echo "name: no-universe"
|
||||||
|
echo "description: Use when doing the thing. Do not use for the other thing — use some-other-skill instead."
|
||||||
|
echo "---"
|
||||||
|
echo ""
|
||||||
|
echo "Do the thing."
|
||||||
|
} > "$ORPHAN_ROOT/no-universe/SKILL.md"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The comparison
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
python3 - "$HOOK" "$SKILL_VALIDATE" "$REPO_ROOT" "$FX" "$ORPHAN_ROOT" <<'PYTHON'
|
||||||
|
import glob
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
hook, validate, repo_root, fixture_dir, orphan_dir = sys.argv[1:6]
|
||||||
|
|
||||||
|
passes = 0
|
||||||
|
failures = 0
|
||||||
|
|
||||||
|
|
||||||
|
def ok(msg):
|
||||||
|
global passes
|
||||||
|
passes += 1
|
||||||
|
print(" PASS: %s" % msg)
|
||||||
|
|
||||||
|
|
||||||
|
def bad(msg):
|
||||||
|
global failures
|
||||||
|
failures += 1
|
||||||
|
print(" FAIL: %s" % msg)
|
||||||
|
|
||||||
|
|
||||||
|
# Tier prefixes. The hook writes `ERROR: ` / `SUGGESTION: ` / `INFO: `; the
|
||||||
|
# auditor writes `FAIL ` / `SUGGESTION ` / `INFO ` and additionally `PASS `
|
||||||
|
# lines, which carry no finding and are dropped.
|
||||||
|
TIERS = (
|
||||||
|
('ERROR', ('ERROR:', 'FAIL ')),
|
||||||
|
('SUGGESTION', ('SUGGESTION:', 'SUGGESTION ')),
|
||||||
|
('INFO', ('INFO:', 'INFO ')),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Each rule turns a finding line into a canonical token. Wording differs between
|
||||||
|
# the two scripts by design (one addresses a committer, the other an auditor), so
|
||||||
|
# the tokens deliberately capture the MEASUREMENT and not the sentence.
|
||||||
|
RULES = (
|
||||||
|
('DESC_CHARS', re.compile(r'description is (\d+) char')),
|
||||||
|
('BODY_WORDS', re.compile(r'body is (\d+) words')),
|
||||||
|
('ROUTE', re.compile(r"routes to '([^']+)'")),
|
||||||
|
('MISSING_REF', re.compile(r'points at (references/[^\s,]+)')),
|
||||||
|
('GOTCHA_ENTRIES', re.compile(r'Gotchas section has (\d+) entries')),
|
||||||
|
('GOTCHA_FRACTION', re.compile(r'Gotchas section is (\d+) of (\d+) body words')),
|
||||||
|
('NO_BOUNDARY_CLAUSE', re.compile(r'(description has no boundary clause)')),
|
||||||
|
('RESOLUTION_DECLINED', re.compile(r'(boundary-target resolution DID NOT RUN)')),
|
||||||
|
('DESC_EMPTY', re.compile(r'(description field is missing or empty)')),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def verdict(output):
|
||||||
|
"""The set of ADR-0020 findings in a script's output, tier included.
|
||||||
|
|
||||||
|
Lines that match no rule are dropped rather than compared: the two scripts
|
||||||
|
legitimately check different things outside ADR-0020 (name/directory
|
||||||
|
agreement, script executability, the 1024-char spec backstop, whole-file
|
||||||
|
line and word ceilings), and forcing those into the comparison would report
|
||||||
|
a difference that is not a disagreement.
|
||||||
|
"""
|
||||||
|
found = set()
|
||||||
|
for raw in output.splitlines():
|
||||||
|
line = raw.strip()
|
||||||
|
tier = None
|
||||||
|
for name, prefixes in TIERS:
|
||||||
|
if any(line.startswith(p) for p in prefixes):
|
||||||
|
tier = name
|
||||||
|
break
|
||||||
|
if tier is None:
|
||||||
|
continue
|
||||||
|
for token, pattern in RULES:
|
||||||
|
match = pattern.search(line)
|
||||||
|
if match:
|
||||||
|
found.add((tier, token) + tuple(match.groups()))
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def run(cmd):
|
||||||
|
proc = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
|
||||||
|
return proc.returncode, proc.stdout.decode('utf-8', 'replace')
|
||||||
|
|
||||||
|
|
||||||
|
def compare(label, skill_dir):
|
||||||
|
skill_md = os.path.join(skill_dir, 'SKILL.md')
|
||||||
|
hook_rc, hook_out = run(['bash', hook, skill_md])
|
||||||
|
audit_rc, audit_out = run(['bash', validate, skill_dir])
|
||||||
|
hook_v = verdict(hook_out)
|
||||||
|
audit_v = verdict(audit_out)
|
||||||
|
|
||||||
|
problems = []
|
||||||
|
only_hook = sorted(hook_v - audit_v)
|
||||||
|
only_audit = sorted(audit_v - hook_v)
|
||||||
|
if only_hook:
|
||||||
|
problems.append('only the hook reported %s' % (only_hook,))
|
||||||
|
if only_audit:
|
||||||
|
problems.append('only skill-audit reported %s' % (only_audit,))
|
||||||
|
|
||||||
|
# Exit codes are compared on the ADR-0020 axis only: an ERROR-tier ADR-0020
|
||||||
|
# finding must make BOTH scripts non-zero, and neither may be turned
|
||||||
|
# non-zero by a SUGGESTION. The raw codes cannot be compared directly —
|
||||||
|
# validate.sh also fails on checks the hook does not run at all.
|
||||||
|
hook_err = any(t == 'ERROR' for t, *_ in hook_v)
|
||||||
|
audit_err = any(t == 'ERROR' for t, *_ in audit_v)
|
||||||
|
if hook_err and hook_rc == 0:
|
||||||
|
problems.append('the hook reported an ADR-0020 ERROR but exited 0')
|
||||||
|
if audit_err and audit_rc == 0:
|
||||||
|
problems.append('skill-audit reported an ADR-0020 FAIL but exited 0')
|
||||||
|
if not hook_err and hook_rc != 0 and not _non_adr_hook_error(hook_out):
|
||||||
|
problems.append('the hook exited %d with no ADR-0020 ERROR and no spec-ceiling ERROR'
|
||||||
|
% hook_rc)
|
||||||
|
|
||||||
|
if problems:
|
||||||
|
bad('%s: %s' % (label, '; '.join(problems)))
|
||||||
|
else:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _non_adr_hook_error(output):
|
||||||
|
"""True if the hook failed on a spec ceiling rather than an ADR-0020 gate.
|
||||||
|
|
||||||
|
MAX_LINES / MAX_WORDS are the hook's other ERROR sources and are outside
|
||||||
|
this comparison, so a non-zero exit explained by one of them is not a
|
||||||
|
disagreement.
|
||||||
|
"""
|
||||||
|
return bool(re.search(r'ERROR: .*(-line ceiling|-word ceiling \(~5,000 tokens)', output))
|
||||||
|
|
||||||
|
|
||||||
|
# --- The real corpus -------------------------------------------------------
|
||||||
|
corpus = sorted(glob.glob(os.path.join(repo_root, 'plugins', '*', '.apm', 'skills', '*')))
|
||||||
|
corpus = [d for d in corpus if os.path.isfile(os.path.join(d, 'SKILL.md'))]
|
||||||
|
print("")
|
||||||
|
print("--- the two scripts agree on every skill in the live corpus (%d files) ---" % len(corpus))
|
||||||
|
if len(corpus) < 30:
|
||||||
|
bad('only %d corpus skills were discovered — the glob is wrong, so this leg '
|
||||||
|
'proves nothing' % len(corpus))
|
||||||
|
else:
|
||||||
|
ok('discovered %d corpus skills to compare' % len(corpus))
|
||||||
|
agreed = 0
|
||||||
|
for skill_dir in corpus:
|
||||||
|
rel = os.path.relpath(skill_dir, repo_root)
|
||||||
|
if compare(rel, skill_dir):
|
||||||
|
agreed += 1
|
||||||
|
if agreed == len(corpus):
|
||||||
|
ok('all %d corpus skills produce identical ADR-0020 verdicts from both scripts' % agreed)
|
||||||
|
|
||||||
|
# --- Boundary fixtures -----------------------------------------------------
|
||||||
|
fixtures = sorted(d for d in (glob.glob(os.path.join(fixture_dir, '*'))
|
||||||
|
+ glob.glob(os.path.join(orphan_dir, '*')))
|
||||||
|
if os.path.isfile(os.path.join(d, 'SKILL.md')))
|
||||||
|
print("")
|
||||||
|
print("--- the two scripts agree on every boundary fixture (%d files) ---" % len(fixtures))
|
||||||
|
if len(fixtures) < 15:
|
||||||
|
bad('only %d fixtures were built — the fixture set is incomplete, so the '
|
||||||
|
'boundaries the corpus does not cover are untested' % len(fixtures))
|
||||||
|
else:
|
||||||
|
ok('built %d boundary fixtures to compare' % len(fixtures))
|
||||||
|
fx_agreed = 0
|
||||||
|
for skill_dir in fixtures:
|
||||||
|
if compare(os.path.basename(skill_dir), skill_dir):
|
||||||
|
fx_agreed += 1
|
||||||
|
if fx_agreed == len(fixtures):
|
||||||
|
ok('all %d boundary fixtures produce identical ADR-0020 verdicts from both scripts' % fx_agreed)
|
||||||
|
|
||||||
|
# --- The comparison must not be vacuous ------------------------------------
|
||||||
|
# Everything above would also pass if verdict() extracted nothing at all. So the
|
||||||
|
# fixtures are required to have produced findings across every axis this suite
|
||||||
|
# claims to compare — if a rule stops matching (a reworded message, say), that is
|
||||||
|
# a silent loss of coverage and it fails here instead.
|
||||||
|
print("")
|
||||||
|
print("--- the comparison actually extracted findings on every axis it claims to cover ---")
|
||||||
|
seen_tokens = set()
|
||||||
|
for skill_dir in fixtures:
|
||||||
|
_, out = run(['bash', hook, os.path.join(skill_dir, 'SKILL.md')])
|
||||||
|
for entry in verdict(out):
|
||||||
|
seen_tokens.add(entry[1])
|
||||||
|
_, out = run(['bash', validate, skill_dir])
|
||||||
|
for entry in verdict(out):
|
||||||
|
seen_tokens.add(entry[1])
|
||||||
|
expected_tokens = {t for t, _ in RULES}
|
||||||
|
missing = sorted(expected_tokens - seen_tokens)
|
||||||
|
if missing:
|
||||||
|
bad('no fixture produced a finding for %s — verdict() may no longer match '
|
||||||
|
'those messages, and any disagreement on them would go unseen' % missing)
|
||||||
|
else:
|
||||||
|
ok('every one of the %d compared axes was exercised by at least one fixture'
|
||||||
|
% len(expected_tokens))
|
||||||
|
|
||||||
|
print("")
|
||||||
|
print("Results: %d passed, %d failed" % (passes, failures))
|
||||||
|
sys.exit(1 if failures else 0)
|
||||||
|
PYTHON
|
||||||
286
tests/test-adr0020-frontmatter.sh
Executable file
286
tests/test-adr0020-frontmatter.sh
Executable file
@@ -0,0 +1,286 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Regression test for the two ways an ADR-0020 gate can be made to check NOTHING
|
||||||
|
# while still exiting 0. Both were live defects, both were silent, and both sit
|
||||||
|
# in the shared resolver block that all three scripts embed verbatim — so every
|
||||||
|
# case below runs against all three.
|
||||||
|
#
|
||||||
|
# 1. THE FRONTMATTER BLOCKER. The frontmatter matcher used to be `^---\n`. A
|
||||||
|
# UTF-8 BOM, a leading blank line, a trailing space after either marker, or
|
||||||
|
# CRLF line endings all defeated it, and the miss was not reported: every
|
||||||
|
# ADR-0020 check was skipped and the file passed. Measured at the time: a
|
||||||
|
# 550-character description with a 1,000-word body exited 0 behind a BOM.
|
||||||
|
# So this file asserts two complementary things — that each of those four
|
||||||
|
# shapes is now TOLERATED (the findings actually fire), and that
|
||||||
|
# frontmatter which genuinely cannot be parsed is a hard ERROR rather than
|
||||||
|
# a quiet skip. A file that cannot be measured must never report green.
|
||||||
|
#
|
||||||
|
# 2. THE VALUELESS DESCRIPTION. `description:` with no value, followed by
|
||||||
|
# another key, let a line regex's `\s*` cross the newline and capture the
|
||||||
|
# NEXT key. The value then looked present (so "missing or empty" never
|
||||||
|
# fired) and was empty once folded (so every ADR-0020 gate early-returned).
|
||||||
|
# An agent file with one exited 0 with zero output through a BLOCKING
|
||||||
|
# pre-push gate. All five spellings of "no value" are pinned here.
|
||||||
|
#
|
||||||
|
# Both fixtures carry an over-ceiling description AND an over-ceiling body on
|
||||||
|
# purpose: asserting a non-zero exit alone would be satisfied by the "cannot
|
||||||
|
# parse" error itself, so the tolerated shapes are asserted on the CONTENT of
|
||||||
|
# the findings, not on the exit code.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||||
|
SKILL_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh"
|
||||||
|
AGENT_VALIDATE="$REPO_ROOT/plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh"
|
||||||
|
PASS=0
|
||||||
|
FAIL=0
|
||||||
|
|
||||||
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||||
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||||
|
|
||||||
|
TMPDIR_T="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||||
|
|
||||||
|
DESC_CHARS=450
|
||||||
|
BODY_WORDS=1000
|
||||||
|
|
||||||
|
# write_fixture <kind> <path> <name> — one generator for both file shapes.
|
||||||
|
#
|
||||||
|
# Byte-level control is the point: BOM placement, line endings and trailing
|
||||||
|
# whitespace are exactly what is under test, so the file is emitted in binary
|
||||||
|
# mode rather than through a shell heredoc that would normalise them.
|
||||||
|
write_fixture() {
|
||||||
|
python3 - "$1" "$2" "$3" "$DESC_CHARS" "$BODY_WORDS" <<'PY'
|
||||||
|
import sys
|
||||||
|
|
||||||
|
kind, path, name, desc_chars, body_words = sys.argv[1:6]
|
||||||
|
desc = 'x' * int(desc_chars)
|
||||||
|
body = ' '.join(['word'] * int(body_words))
|
||||||
|
|
||||||
|
# The default, well-formed shape. Variants below mutate it.
|
||||||
|
open_marker = '---'
|
||||||
|
close_marker = '---'
|
||||||
|
prefix = ''
|
||||||
|
newline = '\n'
|
||||||
|
fm_lines = ['name: ' + name, 'description: ' + desc]
|
||||||
|
|
||||||
|
if kind == 'plain':
|
||||||
|
pass
|
||||||
|
elif kind == 'bom':
|
||||||
|
prefix = ''
|
||||||
|
elif kind == 'leading-blanks':
|
||||||
|
prefix = '\n\n \n'
|
||||||
|
elif kind == 'trailing-ws':
|
||||||
|
open_marker = '--- '
|
||||||
|
close_marker = '---\t '
|
||||||
|
elif kind == 'crlf':
|
||||||
|
newline = '\r\n'
|
||||||
|
elif kind == 'no-close':
|
||||||
|
close_marker = None
|
||||||
|
elif kind == 'yaml-list':
|
||||||
|
fm_lines = ['- one', '- two']
|
||||||
|
elif kind == 'yaml-string':
|
||||||
|
fm_lines = ['just a bare scalar, not a mapping']
|
||||||
|
elif kind == 'yaml-none':
|
||||||
|
fm_lines = []
|
||||||
|
elif kind == 'yaml-malformed':
|
||||||
|
fm_lines = ['name: ' + name, 'description: "unterminated', 'tabs:\t- a']
|
||||||
|
elif kind == 'desc-no-value':
|
||||||
|
# The exact shape that exited 0 with zero output: a line regex's `\s*`
|
||||||
|
# crosses the newline and captures `model: sonnet` as the description.
|
||||||
|
fm_lines = ['name: ' + name, 'description:', 'model: sonnet']
|
||||||
|
elif kind == 'desc-null':
|
||||||
|
fm_lines = ['name: ' + name, 'description: null']
|
||||||
|
elif kind == 'desc-single-quoted-empty':
|
||||||
|
fm_lines = ['name: ' + name, "description: ''"]
|
||||||
|
elif kind == 'desc-double-quoted-empty':
|
||||||
|
fm_lines = ['name: ' + name, 'description: ""']
|
||||||
|
elif kind == 'desc-empty-fold':
|
||||||
|
fm_lines = ['name: ' + name, 'description: >']
|
||||||
|
else:
|
||||||
|
raise SystemExit('unknown fixture kind: %s' % kind)
|
||||||
|
|
||||||
|
parts = [prefix, open_marker, newline]
|
||||||
|
for line in fm_lines:
|
||||||
|
parts.append(line)
|
||||||
|
parts.append(newline)
|
||||||
|
if close_marker is not None:
|
||||||
|
parts.append(close_marker)
|
||||||
|
parts.append(newline)
|
||||||
|
parts.append(newline)
|
||||||
|
parts.append(body)
|
||||||
|
parts.append(newline)
|
||||||
|
|
||||||
|
with open(path, 'wb') as fh:
|
||||||
|
fh.write(''.join(parts).encode('utf-8'))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# Builds all three subjects for one fixture kind and echoes nothing; the paths
|
||||||
|
# are fixed by convention so the probes below can find them.
|
||||||
|
#
|
||||||
|
# skill-audit takes a DIRECTORY (SKILL.md inside it, name matching the dir);
|
||||||
|
# agent-audit takes a FILE inside an apm package. The hook takes the SKILL.md
|
||||||
|
# directly, so it and skill-audit share one file.
|
||||||
|
build_subjects() {
|
||||||
|
local kind="$1" base="$TMPDIR_T/$1"
|
||||||
|
rm -rf "$base"
|
||||||
|
mkdir -p "$base/skill/my-skill" "$base/agent/.apm/agents"
|
||||||
|
cat > "$base/agent/apm.yml" <<'EOF'
|
||||||
|
name: test-package
|
||||||
|
version: 0.1.0
|
||||||
|
type: skill
|
||||||
|
EOF
|
||||||
|
write_fixture "$kind" "$base/skill/my-skill/SKILL.md" my-skill
|
||||||
|
write_fixture "$kind" "$base/agent/.apm/agents/my-agent.agent.md" my-agent
|
||||||
|
}
|
||||||
|
|
||||||
|
# probe_all <label> <kind> <needle>... — runs all three scripts over the fixture
|
||||||
|
# and requires every one of them to exit non-zero AND report every needle. One
|
||||||
|
# assertion per script would let two of them drift apart while the suite stayed
|
||||||
|
# green; the whole point of the shared resolver block is that they cannot.
|
||||||
|
#
|
||||||
|
# A needle written `@skills:<text>` is asserted for the hook and skill-audit but
|
||||||
|
# NOT for agent-audit. There is exactly one such needle in this file — the body
|
||||||
|
# word ceiling — and the exemption is the ADR, not a workaround: ADR-0020 gives
|
||||||
|
# agents the description gates and deliberately NO body word gate, because an
|
||||||
|
# agent body becomes the system prompt of a fresh context rather than competing
|
||||||
|
# with the caller's live conversation. Demanding a body finding from agent-audit
|
||||||
|
# would be demanding the ADR be contradicted.
|
||||||
|
probe_all() {
|
||||||
|
local label="$1" kind="$2"
|
||||||
|
shift 2
|
||||||
|
local base="$TMPDIR_T/$kind"
|
||||||
|
local -a targets=(
|
||||||
|
"hook|$HOOK|$base/skill/my-skill/SKILL.md"
|
||||||
|
"skill-audit|$SKILL_VALIDATE|$base/skill/my-skill"
|
||||||
|
"agent-audit|$AGENT_VALIDATE|$base/agent/.apm/agents/my-agent.agent.md"
|
||||||
|
)
|
||||||
|
local problems=""
|
||||||
|
for target in "${targets[@]}"; do
|
||||||
|
local who="${target%%|*}" rest="${target#*|}"
|
||||||
|
local script="${rest%%|*}" arg="${rest#*|}"
|
||||||
|
local out status=0
|
||||||
|
set +e
|
||||||
|
out="$(bash "$script" "$arg" 2>&1)"
|
||||||
|
status=$?
|
||||||
|
set -e
|
||||||
|
if [[ $status -eq 0 ]]; then
|
||||||
|
problems="$problems [$who exited 0: ${out:-<no output>}]"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
for needle in "$@"; do
|
||||||
|
if [[ "$needle" == @skills:* ]]; then
|
||||||
|
# Spelled as a full `if`, not `[[ ... ]] && continue`. Under `set -e` the
|
||||||
|
# short form's exit status is the test's when it is false, and relying on
|
||||||
|
# the &&-list exemption to keep that from aborting the run is a footgun
|
||||||
|
# one edit away from biting.
|
||||||
|
if [[ "$who" == agent-audit ]]; then
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
needle="${needle#@skills:}"
|
||||||
|
fi
|
||||||
|
if [[ "$out" != *"$needle"* ]]; then
|
||||||
|
problems="$problems [$who never said '$needle': $out]"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
done
|
||||||
|
if [[ -z "$problems" ]]; then
|
||||||
|
pass "$label"
|
||||||
|
else
|
||||||
|
fail "$label —$problems"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1a. Tolerated frontmatter shapes — the gates must RUN, not merely not-pass
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The needles are the FINDINGS, not the exit code. A script that rejected the BOM
|
||||||
|
# outright would exit non-zero too, and would still be skipping every ADR-0020
|
||||||
|
# measurement — which is the defect, one error message later.
|
||||||
|
echo ""
|
||||||
|
echo "--- a BOM, leading blanks, trailing marker whitespace and CRLF are all tolerated, and the gates still fire ---"
|
||||||
|
for kind in plain bom leading-blanks trailing-ws crlf; do
|
||||||
|
build_subjects "$kind"
|
||||||
|
done
|
||||||
|
probe_all "control: a well-formed over-ceiling file fails on BOTH the description and the body" \
|
||||||
|
plain "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||||
|
probe_all "a UTF-8 BOM does not hide an over-ceiling description or body" \
|
||||||
|
bom "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||||
|
probe_all "leading blank lines before the opening --- do not hide the findings" \
|
||||||
|
leading-blanks "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||||
|
probe_all "trailing whitespace after either --- marker does not hide the findings" \
|
||||||
|
trailing-ws "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||||
|
# Note on what this last one can and cannot detect. read_text() opens the file in
|
||||||
|
# TEXT mode, so Python's universal-newline translation turns \r\n into \n before
|
||||||
|
# the frontmatter matcher ever sees it — verified by mutation: reverting
|
||||||
|
# FRONTMATTER_RE to the old `^---\n(.*?)\n---` breaks the leading-blanks and
|
||||||
|
# trailing-whitespace cases above but NOT this one. So this case pins the
|
||||||
|
# end-to-end behaviour (a CRLF file is measured, not skipped) rather than the
|
||||||
|
# `\r?\n` alternations in the regex, and it would catch a future switch to binary
|
||||||
|
# reads or to a newline='' open. Kept for that reason, and labelled so nobody
|
||||||
|
# reads it as covering more than it does.
|
||||||
|
probe_all "CRLF line endings do not hide the findings" \
|
||||||
|
crlf "description is $DESC_CHARS char" "@skills:body is $BODY_WORDS words"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1b. Unparseable frontmatter is a hard ERROR, never a quiet skip
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "--- genuinely unparseable frontmatter exits non-zero with a message, rather than passing quietly ---"
|
||||||
|
for kind in no-close yaml-list yaml-string yaml-none yaml-malformed; do
|
||||||
|
build_subjects "$kind"
|
||||||
|
done
|
||||||
|
probe_all "frontmatter with no closing --- is reported, not skipped" \
|
||||||
|
no-close "frontmatter"
|
||||||
|
probe_all "frontmatter that parses to a LIST is reported, not skipped" \
|
||||||
|
yaml-list "frontmatter"
|
||||||
|
probe_all "frontmatter that parses to a STRING is reported, not skipped" \
|
||||||
|
yaml-string "frontmatter"
|
||||||
|
probe_all "frontmatter that parses to None (empty block) is reported, not skipped" \
|
||||||
|
yaml-none "frontmatter"
|
||||||
|
probe_all "malformed YAML in the frontmatter is reported, not skipped" \
|
||||||
|
yaml-malformed "frontmatter"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. A valueless description is a hard FAIL in all three scripts
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# All five spellings mean the same thing to a YAML parser — an empty value — and
|
||||||
|
# all five have to be decided on the FOLDED value rather than on a line regex.
|
||||||
|
# `description:` followed by `model: sonnet` is the one that shipped: it made the
|
||||||
|
# value look present, skipped the "missing or empty" failure, and then
|
||||||
|
# early-returned out of every ADR-0020 gate on the genuinely empty folded value.
|
||||||
|
echo ""
|
||||||
|
echo "--- every spelling of a valueless description hard-FAILs in all three scripts ---"
|
||||||
|
for kind in desc-no-value desc-null desc-single-quoted-empty desc-double-quoted-empty desc-empty-fold; do
|
||||||
|
build_subjects "$kind"
|
||||||
|
done
|
||||||
|
probe_all "'description:' with no value (next key not captured as the value) FAILs" \
|
||||||
|
desc-no-value "description field is missing or empty"
|
||||||
|
probe_all "'description: null' FAILs" \
|
||||||
|
desc-null "description field is missing or empty"
|
||||||
|
probe_all "\"description: ''\" FAILs" \
|
||||||
|
desc-single-quoted-empty "description field is missing or empty"
|
||||||
|
probe_all "'description: \"\"' FAILs" \
|
||||||
|
desc-double-quoted-empty "description field is missing or empty"
|
||||||
|
probe_all "'description: >' with nothing folded under it FAILs" \
|
||||||
|
desc-empty-fold "description field is missing or empty"
|
||||||
|
|
||||||
|
# The specific regression, spelled out: the valueless-description agent file must
|
||||||
|
# not merely fail — it must not be SILENT. Zero output on a blocking gate is what
|
||||||
|
# made this un-diagnosable, so the output is asserted non-empty independently.
|
||||||
|
echo ""
|
||||||
|
echo "--- the valueless-description agent file produces output, not silence ---"
|
||||||
|
build_subjects desc-no-value
|
||||||
|
set +e
|
||||||
|
SILENT_OUT="$(bash "$AGENT_VALIDATE" "$TMPDIR_T/desc-no-value/agent/.apm/agents/my-agent.agent.md" 2>&1)"
|
||||||
|
SILENT_RC=$?
|
||||||
|
set -e
|
||||||
|
if [[ $SILENT_RC -ne 0 && -n "$SILENT_OUT" ]]; then
|
||||||
|
pass "agent-audit reports a valueless description rather than exiting 0 with zero output"
|
||||||
|
else
|
||||||
|
fail "agent-audit exited $SILENT_RC with output '${SILENT_OUT:-<empty>}' — the original defect was exit 0 and total silence on a blocking pre-push gate"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Results: $PASS passed, $FAIL failed"
|
||||||
|
[[ $FAIL -eq 0 ]]
|
||||||
409
tests/test-adr0020-targets.sh
Executable file
409
tests/test-adr0020-targets.sh
Executable file
@@ -0,0 +1,409 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Regression test for the two properties of ADR-0020 boundary-target resolution
|
||||||
|
# that decide whether the gate can be trusted at all.
|
||||||
|
#
|
||||||
|
# 1. MACHINE INDEPENDENCE. The resolution universe is derived by walking up
|
||||||
|
# FROM THE TARGET FILE to an authoring root, and when one is found the
|
||||||
|
# deployed .claude/ and .agents/ trees are deliberately NOT consulted. Those
|
||||||
|
# trees are `apm install` output — gitignored, and present only on a machine
|
||||||
|
# that has run it. Four cross-plugin targets in this repo resolved through
|
||||||
|
# .claude/skills/ alone, so the same commit measured 2 dangling targets on a
|
||||||
|
# developer machine and 6 on a fresh clone. A gate shipping hot with no
|
||||||
|
# baseline cannot give two answers, so this file asserts the verdict is
|
||||||
|
# identical with and without a deployed tree — on a synthetic fixture AND on
|
||||||
|
# the real 39-skill corpus.
|
||||||
|
#
|
||||||
|
# 2. THE BARE-TARGET GRAMMAR RULE. A hyphenated token used as a compound
|
||||||
|
# MODIFIER ("pre-commit hooks", "pull-request template") is prose, not a
|
||||||
|
# route; a terminal one is a real target. Getting this wrong in either
|
||||||
|
# direction is fatal: firing on prose makes the gate untrustworthy and it
|
||||||
|
# gets turned off, while suppressing too much deletes the only two true
|
||||||
|
# positives the corpus has. Both live true positives are BARE, which is why
|
||||||
|
# the rule keys on the FOLLOWER TOKEN rather than on marking, and why they
|
||||||
|
# are pinned by name below — a future false-positive fix must not be able to
|
||||||
|
# quietly take them with it.
|
||||||
|
#
|
||||||
|
# 3. IN-SENTENCE CORROBORATION. Terminal position alone is not evidence of a
|
||||||
|
# route: "run `pre-commit` instead", "see `commit-msg`", "use the clean-up
|
||||||
|
# instead" and "run unit-tests" are all terminal, all prose, and all were
|
||||||
|
# hard FAILs with no suppression mechanism anywhere in the gate. A
|
||||||
|
# prose-form target therefore blocks only when its own sentence names
|
||||||
|
# another target that RESOLVES; otherwise it is reported at SUGGESTION tier
|
||||||
|
# and the commit proceeds. Route NOTATION (`/name`, `-> name`) is exempt
|
||||||
|
# and always blocks. Both halves are asserted below: the prose class must
|
||||||
|
# report-not-block, and the notation and corroborated forms must still
|
||||||
|
# ERROR, or the fix would have eaten the gate rather than narrowed it.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||||
|
HOOK="$REPO_ROOT/scripts/skill-size-check.sh"
|
||||||
|
PASS=0
|
||||||
|
FAIL=0
|
||||||
|
|
||||||
|
pass() { echo " PASS: $1"; PASS=$((PASS + 1)); }
|
||||||
|
fail() { echo " FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||||
|
|
||||||
|
TMPDIR_T="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$TMPDIR_T"' EXIT
|
||||||
|
|
||||||
|
# write_skill <skill-dir> <name> <desc>
|
||||||
|
write_skill() {
|
||||||
|
mkdir -p "$1"
|
||||||
|
{
|
||||||
|
echo "---"
|
||||||
|
echo "name: $2"
|
||||||
|
echo "description: $3"
|
||||||
|
echo "---"
|
||||||
|
echo ""
|
||||||
|
echo "Do the thing."
|
||||||
|
} > "$1/SKILL.md"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Machine independence — synthetic fixture
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Two trees, identical except that one also carries a deployed .claude/ tree
|
||||||
|
# holding a skill and an agent that exist NOWHERE in plugins/. The subject routes
|
||||||
|
# to one name that lives in a sibling plugin (must resolve in both) and one that
|
||||||
|
# lives only in .claude/ (must DANGLE in both — an authoring root exists, so the
|
||||||
|
# deployed tree is not part of the universe).
|
||||||
|
#
|
||||||
|
# If the deployed tree were consulted, the second target would resolve on the
|
||||||
|
# machine that has run `apm install` and dangle on a fresh clone. That is the
|
||||||
|
# 2-vs-6 defect exactly, at fixture scale.
|
||||||
|
echo ""
|
||||||
|
echo "--- the same file gets the same verdict with and without a deployed .claude/ tree ---"
|
||||||
|
build_tree() {
|
||||||
|
local root="$1"
|
||||||
|
write_skill "$root/plugins/other-plugin/.apm/skills/cross-plugin-skill" cross-plugin-skill \
|
||||||
|
"Use when doing the other thing. Do not use for anything else."
|
||||||
|
write_skill "$root/plugins/subject-plugin/.apm/skills/my-skill" my-skill \
|
||||||
|
"Use when doing the thing. Do not use for the other thing — use cross-plugin-skill or deployed-only-skill instead."
|
||||||
|
}
|
||||||
|
build_tree "$TMPDIR_T/no-claude"
|
||||||
|
build_tree "$TMPDIR_T/with-claude"
|
||||||
|
# The deployed tree, present only in the second root. Both a skill and an agent,
|
||||||
|
# because both are valid routing targets and both would leak.
|
||||||
|
mkdir -p "$TMPDIR_T/with-claude/.claude/skills/deployed-only-skill" \
|
||||||
|
"$TMPDIR_T/with-claude/.claude/agents"
|
||||||
|
: > "$TMPDIR_T/with-claude/.claude/agents/deployed-only-agent.md"
|
||||||
|
|
||||||
|
run_subject() {
|
||||||
|
local root="$1" out
|
||||||
|
set +e
|
||||||
|
out="$(bash "$HOOK" "$root/plugins/subject-plugin/.apm/skills/my-skill/SKILL.md" 2>&1)"
|
||||||
|
set -e
|
||||||
|
# Normalise the tree root out of the paths so the two runs are comparable.
|
||||||
|
printf '%s\n' "$out" | sed "s#$root#<ROOT>#g"
|
||||||
|
}
|
||||||
|
NO_CLAUDE_OUT="$(run_subject "$TMPDIR_T/no-claude")"
|
||||||
|
WITH_CLAUDE_OUT="$(run_subject "$TMPDIR_T/with-claude")"
|
||||||
|
|
||||||
|
if [[ "$NO_CLAUDE_OUT" == "$WITH_CLAUDE_OUT" ]]; then
|
||||||
|
pass "identical output with and without a deployed .claude/ tree"
|
||||||
|
else
|
||||||
|
fail "the deployed .claude/ tree changed the verdict — without: [$NO_CLAUDE_OUT] with: [$WITH_CLAUDE_OUT]"
|
||||||
|
fi
|
||||||
|
# Identical-but-wrong is still possible (both could resolve everything, or
|
||||||
|
# neither could resolve anything), so the CONTENT is asserted too: the
|
||||||
|
# sibling-plugin name must resolve and the deployed-only name must not.
|
||||||
|
if [[ "$WITH_CLAUDE_OUT" == *"routes to 'deployed-only-skill'"* ]]; then
|
||||||
|
pass "a name that exists only in .claude/ still dangles when an authoring root is present"
|
||||||
|
else
|
||||||
|
fail "the deployed-only target did not dangle — the deployed tree is being read into the universe: $WITH_CLAUDE_OUT"
|
||||||
|
fi
|
||||||
|
if [[ "$WITH_CLAUDE_OUT" != *"routes to 'cross-plugin-skill'"* ]]; then
|
||||||
|
pass "a name in a SIBLING PLUGIN resolves, so the comparison above is not 'nothing resolves'"
|
||||||
|
else
|
||||||
|
fail "the sibling-plugin target dangled — the monorepo universe is not being built: $WITH_CLAUDE_OUT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The other half of the rule: with NO authoring root, deployed trees ARE the
|
||||||
|
# universe. That is the consumer case, and without this the rule above could be
|
||||||
|
# implemented as "never read .claude/", which would leave consumers with no
|
||||||
|
# resolution at all.
|
||||||
|
echo ""
|
||||||
|
echo "--- with no authoring root, a deployed .claude/ tree IS the universe ---"
|
||||||
|
CONSUMER="$TMPDIR_T/consumer"
|
||||||
|
mkdir -p "$CONSUMER/.claude/skills/deployed-only-skill"
|
||||||
|
write_skill "$CONSUMER/.claude/skills/my-skill" my-skill \
|
||||||
|
"Use when doing the thing. Do not use for the other thing — use deployed-only-skill instead."
|
||||||
|
set +e
|
||||||
|
CONSUMER_OUT="$(bash "$HOOK" "$CONSUMER/.claude/skills/my-skill/SKILL.md" 2>&1)"
|
||||||
|
CONSUMER_RC=$?
|
||||||
|
set -e
|
||||||
|
if [[ $CONSUMER_RC -eq 0 && "$CONSUMER_OUT" != *"routes to"* && "$CONSUMER_OUT" != *"DID NOT RUN"* ]]; then
|
||||||
|
pass "a sibling in a deployed .claude/skills/ tree resolves when there is no authoring root"
|
||||||
|
else
|
||||||
|
fail "the consumer path did not resolve through the deployed tree (exit $CONSUMER_RC): ${CONSUMER_OUT:-<empty>}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1b. Machine independence — the real corpus
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The fixture above proves the rule; this proves it at the scale where it broke.
|
||||||
|
#
|
||||||
|
# The A/B is built rather than borrowed. plugins/ is copied TWICE: once bare (a
|
||||||
|
# fresh clone), and once with a synthetic .claude/skills/ tree deployed beside it
|
||||||
|
# holding a directory for every name the corpus currently reports as dangling. If
|
||||||
|
# deployed trees leaked back into the universe, the second copy would resolve
|
||||||
|
# those names and report an empty dangling set while the first reported two —
|
||||||
|
# 2-vs-6, reproduced deterministically.
|
||||||
|
#
|
||||||
|
# Deliberately NOT keyed on whether THIS machine has run `apm install`. Doing that
|
||||||
|
# would make the suite fail on a fresh clone (where there is no .claude/ to
|
||||||
|
# contrast against) — a test of machine independence that is itself
|
||||||
|
# machine-dependent. The live tree is still compared, as a third data point, but
|
||||||
|
# nothing here requires it to be in either state.
|
||||||
|
echo ""
|
||||||
|
echo "--- the real corpus reports the same dangling targets with and without a deployed tree ---"
|
||||||
|
dangling_set() {
|
||||||
|
local -a files=()
|
||||||
|
local f
|
||||||
|
# Collected with a `while read` loop, not `mapfile`: macOS ships bash 3.2,
|
||||||
|
# which has no `mapfile`, and tests/test-vale-wrap.sh scans tests/*.sh for
|
||||||
|
# exactly that hazard. `find` rather than a glob so both roots walk identically.
|
||||||
|
while IFS= read -r f; do
|
||||||
|
files+=("$f")
|
||||||
|
done < <(find "$1" -path '*/.apm/skills/*/SKILL.md' | sort)
|
||||||
|
if [[ ${#files[@]} -eq 0 ]]; then
|
||||||
|
echo "NO-FILES-FOUND"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
set +e
|
||||||
|
bash "$HOOK" ${files[@]+"${files[@]}"} 2>&1 \
|
||||||
|
| grep -oE "routes to '[^']+'" \
|
||||||
|
| sed "s/routes to '//; s/'//" \
|
||||||
|
| sort -u
|
||||||
|
set -e
|
||||||
|
}
|
||||||
|
LIVE_DANGLING="$(dangling_set "$REPO_ROOT/plugins")"
|
||||||
|
|
||||||
|
FRESH_ROOT="$TMPDIR_T/fresh-clone"
|
||||||
|
mkdir -p "$FRESH_ROOT"
|
||||||
|
cp -R "$REPO_ROOT/plugins" "$FRESH_ROOT/plugins"
|
||||||
|
[[ -f "$REPO_ROOT/apm.yml" ]] && cp "$REPO_ROOT/apm.yml" "$FRESH_ROOT/apm.yml"
|
||||||
|
FRESH_DANGLING="$(dangling_set "$FRESH_ROOT/plugins")"
|
||||||
|
|
||||||
|
DEPLOYED_ROOT="$TMPDIR_T/deployed-clone"
|
||||||
|
mkdir -p "$DEPLOYED_ROOT/.claude/skills" "$DEPLOYED_ROOT/.claude/agents"
|
||||||
|
cp -R "$REPO_ROOT/plugins" "$DEPLOYED_ROOT/plugins"
|
||||||
|
[[ -f "$REPO_ROOT/apm.yml" ]] && cp "$REPO_ROOT/apm.yml" "$DEPLOYED_ROOT/apm.yml"
|
||||||
|
# Deploy exactly the names that currently dangle. That is the strongest possible
|
||||||
|
# bait: if the deployed tree were consulted, every one of them would resolve and
|
||||||
|
# the dangling set would collapse to empty.
|
||||||
|
DEPLOY_COUNT=0
|
||||||
|
while IFS= read -r name; do
|
||||||
|
[[ -n "$name" ]] || continue
|
||||||
|
mkdir -p "$DEPLOYED_ROOT/.claude/skills/$name"
|
||||||
|
DEPLOY_COUNT=$((DEPLOY_COUNT + 1))
|
||||||
|
done <<< "$FRESH_DANGLING"
|
||||||
|
DEPLOYED_DANGLING="$(dangling_set "$DEPLOYED_ROOT/plugins")"
|
||||||
|
|
||||||
|
if [[ "$DEPLOY_COUNT" -gt 0 ]]; then
|
||||||
|
pass "precondition: $DEPLOY_COUNT dangling name(s) deployed into the contrast tree's .claude/skills/, so the A/B has something to distinguish"
|
||||||
|
else
|
||||||
|
fail "no dangling names to deploy — the corpus reports none, so this A/B distinguishes nothing. Deploy a known-absent name explicitly instead of deriving one."
|
||||||
|
fi
|
||||||
|
if [[ ! -d "$FRESH_ROOT/.claude" && ! -d "$FRESH_ROOT/.agents" ]]; then
|
||||||
|
pass "precondition: the fresh-clone copy has no deployed tree of its own"
|
||||||
|
else
|
||||||
|
fail "the fresh-clone copy picked up a deployed tree — it is not a fresh-clone fixture"
|
||||||
|
fi
|
||||||
|
if [[ "$FRESH_DANGLING" == "$DEPLOYED_DANGLING" ]]; then
|
||||||
|
pass "deploying every dangling name into .claude/skills/ changes nothing: $(echo "$FRESH_DANGLING" | tr '\n' ' ')"
|
||||||
|
else
|
||||||
|
fail "the corpus verdict depends on whether apm install has been run — fresh clone: [$(echo "$FRESH_DANGLING" | tr '\n' ' ')] with a deployed tree: [$(echo "$DEPLOYED_DANGLING" | tr '\n' ' ')]"
|
||||||
|
fi
|
||||||
|
# Third data point: whatever state THIS machine happens to be in, the live tree
|
||||||
|
# must agree with a bare copy of the same plugins/. No precondition on that state
|
||||||
|
# — see the section header.
|
||||||
|
if [[ "$LIVE_DANGLING" == "$FRESH_DANGLING" ]]; then
|
||||||
|
pass "the live tree agrees with a bare copy (this machine $( [[ -d "$REPO_ROOT/.claude/skills" ]] && echo "HAS" || echo "has no" ) deployed .claude/skills/ tree)"
|
||||||
|
else
|
||||||
|
fail "the live tree disagrees with a bare copy of the same plugins/ — live: [$(echo "$LIVE_DANGLING" | tr '\n' ' ')] fresh clone: [$(echo "$FRESH_DANGLING" | tr '\n' ' ')]"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1c. The two live true positives, pinned by name
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ADR-0020 records these as real broken routing targets and splits fixing them
|
||||||
|
# into Gitea issue #100. Until that lands they are the ONLY evidence the dangling
|
||||||
|
# check finds anything at all in real prose, so they are asserted as an exact set
|
||||||
|
# rather than a "contains" — a false-positive fix that suppressed one of them
|
||||||
|
# would otherwise land green.
|
||||||
|
#
|
||||||
|
# `gitea-labels` is the subtler of the two and is worth keeping: it is not
|
||||||
|
# written anywhere as `gitea-labels`. gitea-issues' description says "Composes
|
||||||
|
# `gitea-labels-\n milestones`" in a `>`-folded scalar, and the fold joins the
|
||||||
|
# lines into "gitea-labels- milestones" — the trailing hyphen is what keeps the
|
||||||
|
# token terminal and therefore danglable.
|
||||||
|
#
|
||||||
|
# WHEN ISSUE #100 IS FIXED: update EXPECTED_DANGLING to match. Do not delete the
|
||||||
|
# assertion — an empty expected set is fine and still pins that no NEW dangling
|
||||||
|
# target appeared.
|
||||||
|
echo ""
|
||||||
|
echo "--- the two live dangling targets in the corpus are exactly the two ADR-0020 records ---"
|
||||||
|
EXPECTED_DANGLING="$(printf '%s\n' gitea-labels neuledge-context)"
|
||||||
|
if [[ "$LIVE_DANGLING" == "$EXPECTED_DANGLING" ]]; then
|
||||||
|
pass "the corpus dangling set is exactly {gitea-labels, neuledge-context}"
|
||||||
|
else
|
||||||
|
fail "the corpus dangling set changed — expected [$(echo "$EXPECTED_DANGLING" | tr '\n' ' ')], got [$(echo "$LIVE_DANGLING" | tr '\n' ' ')]. If a retrofit fixed one, update EXPECTED_DANGLING; if a false-positive fix silently deleted one, that is the regression this asserts."
|
||||||
|
fi
|
||||||
|
for probe in \
|
||||||
|
"plugins/bin/.apm/skills/research/SKILL.md:neuledge-context" \
|
||||||
|
"plugins/gitea/.apm/skills/gitea-issues/SKILL.md:gitea-labels"; do
|
||||||
|
probe_file="$REPO_ROOT/${probe%%:*}"
|
||||||
|
probe_name="${probe##*:}"
|
||||||
|
if [[ ! -f "$probe_file" ]]; then
|
||||||
|
fail "the true-positive fixture ${probe%%:*} no longer exists — this pin has become vacuous"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
set +e
|
||||||
|
probe_out="$(bash "$HOOK" "$probe_file" 2>&1)"
|
||||||
|
set -e
|
||||||
|
if [[ "$probe_out" == *"routes to '$probe_name'"* ]]; then
|
||||||
|
pass "detects the dangling '$probe_name' target in ${probe%%:*}"
|
||||||
|
else
|
||||||
|
fail "did not detect the dangling '$probe_name' target in ${probe%%:*} — a false-positive fix has taken a true positive with it: $probe_out"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. The bare-target grammar rule
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Every fixture is built inside a real plugin tree. In a bare temp directory the
|
||||||
|
# resolver would decline ("DID NOT RUN") and every must-not-error case would pass
|
||||||
|
# vacuously, proving nothing about extraction.
|
||||||
|
echo ""
|
||||||
|
echo "--- attributive compound modifiers are prose, not routing targets ---"
|
||||||
|
GRAMMAR_ROOT="$TMPDIR_T/grammar"
|
||||||
|
write_skill "$GRAMMAR_ROOT/plugins/p/.apm/skills/sibling-skill" sibling-skill \
|
||||||
|
"Use when doing the other thing. Do not use for anything else."
|
||||||
|
|
||||||
|
# grammar_case <slug> <expect: silent|errors> <needle> <description>
|
||||||
|
grammar_case() {
|
||||||
|
local slug="$1" mode="$2" needle="$3" desc="$4" out status=0
|
||||||
|
write_skill "$GRAMMAR_ROOT/plugins/p/.apm/skills/$slug" "$slug" "$desc"
|
||||||
|
set +e
|
||||||
|
out="$(bash "$HOOK" "$GRAMMAR_ROOT/plugins/p/.apm/skills/$slug/SKILL.md" 2>&1)"
|
||||||
|
status=$?
|
||||||
|
set -e
|
||||||
|
if [[ "$out" == *"DID NOT RUN"* ]]; then
|
||||||
|
fail "\"$desc\" — the resolver declined, so this case asserts nothing about extraction: $out"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
case "$mode" in
|
||||||
|
silent)
|
||||||
|
if [[ $status -eq 0 && -z "$out" ]]; then
|
||||||
|
pass "not a dangling target: \"$desc\""
|
||||||
|
else
|
||||||
|
fail "\"$desc\" (exit $status, output: ${out:-<empty>})"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
errors)
|
||||||
|
if [[ $status -ne 0 && "$out" == *"$needle"* ]]; then
|
||||||
|
pass "still a dangling target: \"$desc\""
|
||||||
|
else
|
||||||
|
fail "\"$desc\" should have ERRORed with $needle (exit $status, output: ${out:-<empty>})"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
suggests)
|
||||||
|
# Reported, not blocking. Both halves matter: an ERROR here would be the
|
||||||
|
# unsuppressable false positive this tier exists to remove, and silence
|
||||||
|
# would mean the gate stopped noticing the target at all.
|
||||||
|
if [[ $status -eq 0 && "$out" == *"SUGGESTION"*"$needle"* && "$out" != *"ERROR"* ]]; then
|
||||||
|
pass "reported but not blocking: \"$desc\""
|
||||||
|
else
|
||||||
|
fail "\"$desc\" should have exited 0 with a SUGGESTION naming $needle (exit $status, output: ${out:-<empty>})"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# The four phrasings that were hard dangling FAILs with no suppression. All four
|
||||||
|
# are lifted from real descriptions in this corpus.
|
||||||
|
grammar_case fp-precommit-hooks silent "" \
|
||||||
|
"Use when running the linter. Use pre-commit hooks instead of ad-hoc scripts."
|
||||||
|
grammar_case fp-pull-request silent "" \
|
||||||
|
"Use when opening changes. Invoke the pull-request template instead of writing one by hand."
|
||||||
|
grammar_case fp-conventional silent "" \
|
||||||
|
"Use when writing history. Use conventional-commits formatting rather than free-form messages."
|
||||||
|
grammar_case fp-prepush-backticked silent "" \
|
||||||
|
"Use when checking a branch. Do not use for local edits — run the \`pre-push\` hooks instead."
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- a lone unresolvable token in terminal position is REPORTED, not blocking ---"
|
||||||
|
# The class this tier was added for. Every one of these is grammatically
|
||||||
|
# identical to a real broken route — "route verb + name + terminal" is also how
|
||||||
|
# prose cites a hook, a linter, a file format or an English compound — and every
|
||||||
|
# one of them was a hard FAIL with no suppression mechanism anywhere in the gate.
|
||||||
|
# The skills most exposed are exactly the ones the ADR-0020 retrofit sends
|
||||||
|
# authors back to rewrite first: pc-run, pc-author, vale-run, vale-config and the
|
||||||
|
# apm-* family are all ABOUT hyphenated tools.
|
||||||
|
grammar_case fp-precommit-terminal suggests "routes to 'pre-commit'" \
|
||||||
|
"Use when running the linter. Do not use for running hooks — run \`pre-commit\` instead."
|
||||||
|
grammar_case fp-commit-msg suggests "routes to 'commit-msg'" \
|
||||||
|
"Use when writing history. Do not use for the commit message — see \`commit-msg\`."
|
||||||
|
grammar_case fp-type-check suggests "routes to 'type-check'" \
|
||||||
|
"Use when compiling. Do not use for type errors — run \`type-check\` first."
|
||||||
|
grammar_case fp-semantic-release suggests "routes to 'semantic-release'" \
|
||||||
|
"Use when tagging a version. Instead, use \`semantic-release\`."
|
||||||
|
# Single-word tool names are the same defect: `eslint` in terminal position hit
|
||||||
|
# the marked-target path and hard-FAILed just as `pre-commit` did.
|
||||||
|
grammar_case fp-eslint suggests "routes to 'eslint'" \
|
||||||
|
"Use when linting JS. Do not use for style — run \`eslint\` instead."
|
||||||
|
# Bare English compounds, which the backtick path never sees at all.
|
||||||
|
grammar_case fp-clean-up suggests "routes to 'clean-up'" \
|
||||||
|
"Use when doing the thing. Do not use for the old flow — use the clean-up instead."
|
||||||
|
grammar_case fp-built-in suggests "routes to 'built-in'" \
|
||||||
|
"Use when doing the thing. Do not use for the custom path — Instead, prefer the built-in."
|
||||||
|
grammar_case fp-write-up suggests "routes to 'write-up'" \
|
||||||
|
"Use when doing the thing. Do not use for the summary — see the write-up."
|
||||||
|
grammar_case fp-front-end suggests "routes to 'front-end'" \
|
||||||
|
"Use when doing the thing. Do not use for the API layer — use the front-end."
|
||||||
|
grammar_case fp-unit-tests suggests "routes to 'unit-tests'" \
|
||||||
|
"Use when testing. Do not run end-to-end, run unit-tests."
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- terminal targets that resolve to nothing still ERROR ---"
|
||||||
|
# The controls. Without them the cases above are satisfied by a check that never
|
||||||
|
# fires, and the narrowing would have eaten the gate rather than sharpened it.
|
||||||
|
# Three forms, three code paths:
|
||||||
|
# * route NOTATION — `/name` and `-> name` — is exempt from corroboration and
|
||||||
|
# blocks on its own. Nobody writes `/pre-commit` or `-> pre-commit` to mean
|
||||||
|
# the hook, so there is no ambiguity to resolve, and an author who wants a
|
||||||
|
# route checked unconditionally has two ways to say so.
|
||||||
|
# * a PROSE-form target — backticked or bare — blocks when its own sentence
|
||||||
|
# names another target that resolves. `sibling-skill` is that corroborator
|
||||||
|
# here; it is the same shape as both live true positives, which sit beside
|
||||||
|
# `write-docs` and `gitea-labels-milestones` respectively.
|
||||||
|
grammar_case tp-arrow errors "routes to 'no-such-arrow-target'" \
|
||||||
|
"Use when doing the thing. Not the other thing → no-such-arrow-target."
|
||||||
|
grammar_case tp-slash errors "routes to 'no-such-slash-skill'" \
|
||||||
|
"Use when doing the thing. Do not use for improvements — use /no-such-slash-skill instead."
|
||||||
|
grammar_case tp-backticked errors "routes to 'no-such-backticked-skill'" \
|
||||||
|
"Use when doing the thing. Do not use for improvements — use \`sibling-skill\` or \`no-such-backticked-skill\` instead."
|
||||||
|
grammar_case tp-bare-terminal errors "routes to 'no-such-bare-skill'" \
|
||||||
|
"Use when doing the thing. Do not use for improvements — use sibling-skill or no-such-bare-skill instead."
|
||||||
|
|
||||||
|
# And the confirming half of the grammar rule: a compound-modifier target is
|
||||||
|
# CONFIRM-ONLY, not ignored. When the name does exist it still counts as a route
|
||||||
|
# — the rule suppresses the ERROR, it does not delete the target.
|
||||||
|
echo ""
|
||||||
|
echo "--- an attributive target that DOES resolve is still a route, not a discarded token ---"
|
||||||
|
write_skill "$GRAMMAR_ROOT/plugins/p/.apm/skills/attributive-subject" attributive-subject \
|
||||||
|
"Use when doing the thing. Do not use for the other thing — use the sibling-skill helper instead."
|
||||||
|
set +e
|
||||||
|
ATTR_OUT="$(bash "$HOOK" "$GRAMMAR_ROOT/plugins/p/.apm/skills/attributive-subject/SKILL.md" 2>&1)"
|
||||||
|
ATTR_RC=$?
|
||||||
|
set -e
|
||||||
|
if [[ $ATTR_RC -eq 0 && -z "$ATTR_OUT" ]]; then
|
||||||
|
pass "a resolving attributive target neither errors nor is reported"
|
||||||
|
else
|
||||||
|
fail "an attributive target naming a REAL skill produced output (exit $ATTR_RC): $ATTR_OUT"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Results: $PASS passed, $FAIL failed"
|
||||||
|
[[ $FAIL -eq 0 ]]
|
||||||
@@ -294,6 +294,61 @@ make_budget_fixture() {
|
|||||||
echo "$file"
|
echo "$file"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# desc_of_length <n> — a description of EXACTLY n characters that carries a
|
||||||
|
# boundary clause and names no routing target.
|
||||||
|
#
|
||||||
|
# ADR-0020's missing-boundary-clause SUGGESTION fires on every description
|
||||||
|
# without one, so a fixture that omits it is never "otherwise clean": a test
|
||||||
|
# asserting silence would be asserting the boundary check's ABSENCE rather than
|
||||||
|
# the length boundary it names. The clause is paid for out of the same budget
|
||||||
|
# being measured (padding arithmetic, not a fixed suffix) so the character count
|
||||||
|
# stays exact. "anything else" is unhyphenated, so no routing target rides along.
|
||||||
|
desc_of_length() {
|
||||||
|
python3 - "$1" <<'PY'
|
||||||
|
import sys
|
||||||
|
n = int(sys.argv[1])
|
||||||
|
prefix = 'Use when doing the thing. Do not use for anything else. '
|
||||||
|
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
||||||
|
print(prefix + 'x' * (n - len(prefix)))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# make_tree_fixture <label> <desc> <body_words> — a SKILL.md inside a synthetic
|
||||||
|
# apm plugin monorepo, so the boundary-target resolver has a universe.
|
||||||
|
#
|
||||||
|
# Resolution walks up FROM THE TARGET FILE to an authoring root (the nearest
|
||||||
|
# ancestor holding plugins/*/.apm/{skills,agents}, falling back to .git); it is
|
||||||
|
# never derived from the checker's own location, because deriving it from
|
||||||
|
# ${BASH_SOURCE} leaked this repo's 39-skill universe into every consumer repo
|
||||||
|
# running the hook. A fixture in a bare mktemp -d therefore has NO universe and
|
||||||
|
# correctly reports "DID NOT RUN" — that is not a bug to paper over with a
|
||||||
|
# looser assertion, it is why the fixture has to be a real tree:
|
||||||
|
#
|
||||||
|
# <root>/plugins/subject-plugin/.apm/skills/<label>/SKILL.md <- the subject
|
||||||
|
# <root>/plugins/subject-plugin/.apm/skills/sibling-skill/ <- same package
|
||||||
|
# <root>/plugins/subject-plugin/.apm/agents/sibling-agent.agent.md
|
||||||
|
# <root>/plugins/other-plugin/.apm/skills/cross-plugin-skill/ <- sibling plugin
|
||||||
|
#
|
||||||
|
# The sibling plugin is what makes "every plugin in the monorepo contributes its
|
||||||
|
# names" testable; without it a cross-plugin target and a typo are the same.
|
||||||
|
make_tree_fixture() {
|
||||||
|
local label="$1" desc="$2" body_words="$3" root apm
|
||||||
|
root="$TMPDIR/tree-$label"
|
||||||
|
apm="$root/plugins/subject-plugin/.apm"
|
||||||
|
mkdir -p "$apm/skills/$label" "$apm/skills/sibling-skill" "$apm/agents" \
|
||||||
|
"$root/plugins/other-plugin/.apm/skills/cross-plugin-skill"
|
||||||
|
: > "$apm/agents/sibling-agent.agent.md"
|
||||||
|
{
|
||||||
|
echo "---"
|
||||||
|
echo "name: $label"
|
||||||
|
echo "description: $desc"
|
||||||
|
echo "---"
|
||||||
|
echo ""
|
||||||
|
python3 -c "print(' '.join(['word'] * $body_words))"
|
||||||
|
} > "$apm/skills/$label/SKILL.md"
|
||||||
|
echo "$apm/skills/$label/SKILL.md"
|
||||||
|
}
|
||||||
|
|
||||||
# expect_gate <label> <expected: pass|suggest|fail> <file> [needle]
|
# expect_gate <label> <expected: pass|suggest|fail> <file> [needle]
|
||||||
expect_gate() {
|
expect_gate() {
|
||||||
local label="$1" expected="$2" file="$3" needle="${4:-}" out status
|
local label="$1" expected="$2" file="$3" needle="${4:-}" out status
|
||||||
@@ -323,15 +378,26 @@ expect_gate() {
|
|||||||
fail "$label (exit $status, output: ${out:-<empty>})"
|
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
|
# A check that DECLINED to run must say so and must not fail the file. The
|
||||||
|
# ERROR guard is the point: a declined check that also errored would satisfy
|
||||||
|
# a bare "output contains INFO" assertion.
|
||||||
|
info)
|
||||||
|
if [[ $status -eq 0 && "$out" == *"INFO"* && "$out" == *"$needle"* \
|
||||||
|
&& "$out" != *"ERROR"* ]]; then
|
||||||
|
pass "$label"
|
||||||
|
else
|
||||||
|
fail "$label (exit $status, output: ${out:-<empty>})"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "--- description budget: $DESC_SUGGEST_CHARS SUGGESTION / $DESC_MAX_CHARS FAIL, both inclusive ---"
|
echo "--- description budget: $DESC_SUGGEST_CHARS SUGGESTION / $DESC_MAX_CHARS FAIL, both inclusive ---"
|
||||||
D_AT_SUGGEST="$(python3 -c "print('x' * $DESC_SUGGEST_CHARS)")"
|
D_AT_SUGGEST="$(desc_of_length "$DESC_SUGGEST_CHARS")"
|
||||||
D_OVER_SUGGEST="$(python3 -c "print('x' * $((DESC_SUGGEST_CHARS + 1)))")"
|
D_OVER_SUGGEST="$(desc_of_length "$((DESC_SUGGEST_CHARS + 1))")"
|
||||||
D_AT_MAX="$(python3 -c "print('x' * $DESC_MAX_CHARS)")"
|
D_AT_MAX="$(desc_of_length "$DESC_MAX_CHARS")"
|
||||||
D_OVER_MAX="$(python3 -c "print('x' * $((DESC_MAX_CHARS + 1)))")"
|
D_OVER_MAX="$(desc_of_length "$((DESC_MAX_CHARS + 1))")"
|
||||||
expect_gate "description at exactly $DESC_SUGGEST_CHARS chars is silent" \
|
expect_gate "description at exactly $DESC_SUGGEST_CHARS chars is silent" \
|
||||||
pass "$(make_budget_fixture desc-at-suggest "$D_AT_SUGGEST" 10)"
|
pass "$(make_budget_fixture desc-at-suggest "$D_AT_SUGGEST" 10)"
|
||||||
expect_gate "description at $((DESC_SUGGEST_CHARS + 1)) chars suggests and exits 0" \
|
expect_gate "description at $((DESC_SUGGEST_CHARS + 1)) chars suggests and exits 0" \
|
||||||
@@ -359,18 +425,23 @@ FOLDED="$TMPDIR/folded.md"
|
|||||||
expect_gate "a >-folded 450-char description fails (raw first line would read as 1 char)" \
|
expect_gate "a >-folded 450-char description fails (raw first line would read as 1 char)" \
|
||||||
fail "$FOLDED" "description is 450 characters"
|
fail "$FOLDED" "description is 450 characters"
|
||||||
|
|
||||||
|
# Every body fixture below carries a boundary clause for the same reason
|
||||||
|
# desc_of_length() does: without one the missing-boundary-clause SUGGESTION
|
||||||
|
# fires and a body-budget test that asserts silence stops isolating the body
|
||||||
|
# budget. It is short, so the description gate stays quiet too.
|
||||||
|
CLEAN_DESC="Short valid description. Do not use for anything else."
|
||||||
echo ""
|
echo ""
|
||||||
echo "--- body budget: $BODY_SUGGEST_WORDS SUGGESTION / $BODY_MAX_WORDS FAIL, body only, both inclusive ---"
|
echo "--- body budget: $BODY_SUGGEST_WORDS SUGGESTION / $BODY_MAX_WORDS FAIL, body only, both inclusive ---"
|
||||||
expect_gate "body at exactly $BODY_SUGGEST_WORDS words is silent" \
|
expect_gate "body at exactly $BODY_SUGGEST_WORDS words is silent" \
|
||||||
pass "$(make_budget_fixture body-at-suggest "Short valid description." "$BODY_SUGGEST_WORDS")"
|
pass "$(make_budget_fixture body-at-suggest "$CLEAN_DESC" "$BODY_SUGGEST_WORDS")"
|
||||||
expect_gate "body at $((BODY_SUGGEST_WORDS + 1)) words suggests and exits 0" \
|
expect_gate "body at $((BODY_SUGGEST_WORDS + 1)) words suggests and exits 0" \
|
||||||
suggest "$(make_budget_fixture body-over-suggest "Short valid description." "$((BODY_SUGGEST_WORDS + 1))")" \
|
suggest "$(make_budget_fixture body-over-suggest "$CLEAN_DESC" "$((BODY_SUGGEST_WORDS + 1))")" \
|
||||||
"body is $((BODY_SUGGEST_WORDS + 1)) words"
|
"body is $((BODY_SUGGEST_WORDS + 1)) words"
|
||||||
expect_gate "body at exactly $BODY_MAX_WORDS words suggests, does not fail" \
|
expect_gate "body at exactly $BODY_MAX_WORDS words suggests, does not fail" \
|
||||||
suggest "$(make_budget_fixture body-at-max "Short valid description." "$BODY_MAX_WORDS")" \
|
suggest "$(make_budget_fixture body-at-max "$CLEAN_DESC" "$BODY_MAX_WORDS")" \
|
||||||
"body is $BODY_MAX_WORDS words"
|
"body is $BODY_MAX_WORDS words"
|
||||||
expect_gate "body at $((BODY_MAX_WORDS + 1)) words fails" \
|
expect_gate "body at $((BODY_MAX_WORDS + 1)) words fails" \
|
||||||
fail "$(make_budget_fixture body-over-max "Short valid description." "$((BODY_MAX_WORDS + 1))")" \
|
fail "$(make_budget_fixture body-over-max "$CLEAN_DESC" "$((BODY_MAX_WORDS + 1))")" \
|
||||||
"$BODY_MAX_WORDS-word ceiling"
|
"$BODY_MAX_WORDS-word ceiling"
|
||||||
|
|
||||||
# The two word gates measure different things and must stay separable: a file
|
# The two word gates measure different things and must stay separable: a file
|
||||||
@@ -382,7 +453,7 @@ BODY_ONLY_DESC="$(python3 -c "print(' '.join(['w'] * 100))")"
|
|||||||
expect_gate "frontmatter words do not count toward the $BODY_MAX_WORDS-word body ceiling" \
|
expect_gate "frontmatter words do not count toward the $BODY_MAX_WORDS-word body ceiling" \
|
||||||
suggest "$(make_budget_fixture body-independent "$BODY_ONLY_DESC" "$((BODY_MAX_WORDS - 5))")" \
|
suggest "$(make_budget_fixture body-independent "$BODY_ONLY_DESC" "$((BODY_MAX_WORDS - 5))")" \
|
||||||
"words"
|
"words"
|
||||||
BIG_BODY="$(make_budget_fixture body-over-not-whole-file "Short valid description." "$((BODY_MAX_WORDS + 1))")"
|
BIG_BODY="$(make_budget_fixture body-over-not-whole-file "$CLEAN_DESC" "$((BODY_MAX_WORDS + 1))")"
|
||||||
BIG_BODY_WORDS="$(wc -w < "$BIG_BODY")"
|
BIG_BODY_WORDS="$(wc -w < "$BIG_BODY")"
|
||||||
if [[ "$BIG_BODY_WORDS" -le "$MAX_WORDS" ]]; then
|
if [[ "$BIG_BODY_WORDS" -le "$MAX_WORDS" ]]; then
|
||||||
pass "the body-gate fixture is $BIG_BODY_WORDS whole-file words, well under MAX_WORDS=$MAX_WORDS — it fails on the body gate alone"
|
pass "the body-gate fixture is $BIG_BODY_WORDS whole-file words, well under MAX_WORDS=$MAX_WORDS — it fails on the body gate alone"
|
||||||
@@ -393,21 +464,37 @@ fi
|
|||||||
echo ""
|
echo ""
|
||||||
echo "--- resolvable boundary targets ---"
|
echo "--- resolvable boundary targets ---"
|
||||||
# Resolution is against the AUTHORING SOURCE (plugins/*/.apm/skills/ and
|
# Resolution is against the AUTHORING SOURCE (plugins/*/.apm/skills/ and
|
||||||
# plugins/*/.apm/agents/), found here via the script's own repo root — these
|
# plugins/*/.apm/agents/), reached by walking up FROM THE SKILL FILE. These
|
||||||
# fixtures live in a temp dir with no plugin tree of their own, so a resolving
|
# fixtures therefore build their own synthetic monorepo (make_tree_fixture) and
|
||||||
# target proves the repo-root path works.
|
# name only fixture-local targets: they must not depend on this repo's live
|
||||||
expect_gate "a boundary target naming a real skill resolves" \
|
# skills, or renaming git-commits would break a test about extraction grammar.
|
||||||
pass "$(make_budget_fixture target-ok \
|
expect_gate "a boundary target naming a sibling skill in the same package resolves" \
|
||||||
"Use when doing the thing. Do not use for commits — use git-commits instead." 10)"
|
pass "$(make_tree_fixture target-ok \
|
||||||
expect_gate "a boundary target naming a real AGENT resolves (agents are valid targets)" \
|
"Use when doing the thing. Do not use for commits — use sibling-skill instead." 10)"
|
||||||
pass "$(make_budget_fixture target-agent-ok \
|
expect_gate "a boundary target naming a skill in a SIBLING PLUGIN resolves (that is what a monorepo means)" \
|
||||||
"Use when doing the thing. Do not use when the caller is an agent — invoke git-orchestrate instead." 10)"
|
pass "$(make_tree_fixture target-cross-plugin \
|
||||||
expect_gate "a boundary target that resolves to nothing fails" \
|
"Use when doing the thing. Do not use for the other thing — use cross-plugin-skill instead." 10)"
|
||||||
fail "$(make_budget_fixture target-missing \
|
expect_gate "a boundary target naming an AGENT resolves (agents are valid targets)" \
|
||||||
"Use when doing the thing. Do not use for improvements — use no-such-skill-anywhere instead." 10)" \
|
pass "$(make_tree_fixture target-agent-ok \
|
||||||
|
"Use when doing the thing. Do not use when the caller is an agent — invoke sibling-agent instead." 10)"
|
||||||
|
# CORROBORATED: `sibling-skill` resolves in the same sentence, which is what
|
||||||
|
# promotes a prose-form target from "reported" to "blocking". A lone prose-form
|
||||||
|
# target is deliberately not fatal — see the case below and the shared resolver's
|
||||||
|
# CORROBORATION note.
|
||||||
|
expect_gate "a boundary target that resolves to nothing fails when its sentence names one that does" \
|
||||||
|
fail "$(make_tree_fixture target-missing \
|
||||||
|
"Use when doing the thing. Do not use for improvements — use sibling-skill or no-such-skill-anywhere instead." 10)" \
|
||||||
"routes to 'no-such-skill-anywhere'"
|
"routes to 'no-such-skill-anywhere'"
|
||||||
|
# UNCORROBORATED: identical grammar to the case above, and identical grammar to
|
||||||
|
# "run `pre-commit` instead". Reported at SUGGESTION tier, exit 0 — a gate that
|
||||||
|
# ships hot with no baseline and no suppression mechanism must not block a commit
|
||||||
|
# on a token it cannot tell from a tool name.
|
||||||
|
expect_gate "a lone boundary target that resolves to nothing is reported, not fatal" \
|
||||||
|
suggest "$(make_tree_fixture target-missing-lone \
|
||||||
|
"Use when doing the thing. Do not use for improvements — use no-such-lone-skill instead." 10)" \
|
||||||
|
"routes to 'no-such-lone-skill'"
|
||||||
expect_gate "a /slash-command boundary target that resolves to nothing fails" \
|
expect_gate "a /slash-command boundary target that resolves to nothing fails" \
|
||||||
fail "$(make_budget_fixture target-missing-slash \
|
fail "$(make_tree_fixture target-missing-slash \
|
||||||
"Use when doing the thing. Do not use for improvements — use /no-such-slash-skill instead." 10)" \
|
"Use when doing the thing. Do not use for improvements — use /no-such-slash-skill instead." 10)" \
|
||||||
"routes to 'no-such-slash-skill'"
|
"routes to 'no-such-slash-skill'"
|
||||||
# False-positive guards. These phrasings are lifted from real descriptions:
|
# False-positive guards. These phrasings are lifted from real descriptions:
|
||||||
@@ -415,16 +502,38 @@ expect_gate "a /slash-command boundary target that resolves to nothing fails" \
|
|||||||
# gitea-files says "(use Read/Write/Edit)", gitea-labels-milestones says
|
# gitea-files says "(use Read/Write/Edit)", gitea-labels-milestones says
|
||||||
# "through `issue_write`/`pull_request_write`". None of them is a routing
|
# "through `issue_write`/`pull_request_write`". None of them is a routing
|
||||||
# target, and reading any of them as one makes the gate untrustworthy.
|
# target, and reading any of them as one makes the gate untrustworthy.
|
||||||
|
#
|
||||||
|
# Each carries a boundary clause in a SEPARATE sentence. That is not decoration:
|
||||||
|
# target extraction is decided per sentence, so the clause satisfies the
|
||||||
|
# missing-boundary-clause SUGGESTION (keeping the expected output empty) while
|
||||||
|
# leaving the sentence under test outside a boundary context, which is the exact
|
||||||
|
# condition each of these is about. They are built as trees so a universe exists
|
||||||
|
# — in a bare temp dir the resolver would decline and the guard would pass
|
||||||
|
# vacuously, proving nothing about extraction.
|
||||||
expect_gate "'run pre-commit hooks' outside a boundary sentence is not a routing target" \
|
expect_gate "'run pre-commit hooks' outside a boundary sentence is not a routing target" \
|
||||||
pass "$(make_budget_fixture fp-precommit \
|
pass "$(make_tree_fixture fp-precommit \
|
||||||
"Use when the user wants to run pre-commit hooks or install git hooks." 10)"
|
"Use when the user wants to run pre-commit hooks or install git hooks. Do not use for anything else." 10)"
|
||||||
expect_gate "an arrow chain outside a boundary clause is not a routing target" \
|
expect_gate "an arrow chain outside a boundary clause is not a routing target" \
|
||||||
pass "$(make_budget_fixture fp-arrow \
|
pass "$(make_tree_fixture fp-arrow \
|
||||||
"Reproduce → minimise → instrument → fix → regression-test. Use when a bug is reported." 10)"
|
"Reproduce → minimise → instrument → fix → regression-test. Use when a bug is reported. Do not use for anything else." 10)"
|
||||||
expect_gate "tool names and MCP tool names are not routing targets" \
|
expect_gate "tool names and MCP tool names are not routing targets" \
|
||||||
pass "$(make_budget_fixture fp-tools \
|
pass "$(make_tree_fixture fp-tools \
|
||||||
"Use when writing issues. Do not use for local files (use Read/Write/Edit) — that write goes through \`issue_write\`/\`pull_request_write\` instead." 10)"
|
"Use when writing issues. Do not use for local files (use Read/Write/Edit) — that write goes through \`issue_write\`/\`pull_request_write\` instead." 10)"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "--- with NO authoring root the resolver declines OUT LOUD and does not fail the file ---"
|
||||||
|
# The consumer/draft case, and a real one: a SKILL.md in a bare directory with no
|
||||||
|
# plugins/*/.apm/ above it and no .git has no universe to resolve against. The
|
||||||
|
# required behaviour is neither a false FAIL nor silence — silence is how a whole
|
||||||
|
# gate family goes missing unnoticed — so the INFO and the named unchecked target
|
||||||
|
# are both asserted, along with exit 0. This is the same path make_tree_fixture
|
||||||
|
# exists to escape, kept pinned so a future "just use the repo root" shortcut
|
||||||
|
# (the ${BASH_SOURCE} universe leak ADR-0020 removed) fails here.
|
||||||
|
expect_gate "a fixture with no authoring root reports DID NOT RUN and exits 0" \
|
||||||
|
info "$(make_budget_fixture no-universe \
|
||||||
|
"Use when doing the thing. Do not use for improvements — use some-other-skill instead." 10)" \
|
||||||
|
"Unchecked target(s): some-other-skill"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo "--- the three live dangling routing targets are caught (issue #100) ---"
|
echo "--- the three live dangling routing targets are caught (issue #100) ---"
|
||||||
# ADR-0020 records four broken routing targets and splits fixing them into its
|
# ADR-0020 records four broken routing targets and splits fixing them into its
|
||||||
|
|||||||
Reference in New Issue
Block a user