Retrofits all 39 skills to ADR-0020's description/body context contract, then fixes what six rounds of independent review found in that retrofit — including four ways the hot gate itself failed open. Closes #99, #107, #108, #110, #111, #114, #115, #120. ## The retrofit (waves 1-5) | | Start | Now | |---|---|---| | Description FAILs (>400 chars) | 26 | **0** | | Body FAILs (>900 words, body-only) | 9 | **0** | | Dangling routing targets | 2 | **0** | | `Kyberforge.CompositionNote` | 10 | **0** | | Preload tax | 21,005 chars | **~10,500** | Under the 12,000-char success criterion. Per-wave detail is on #99. ## The review fixes **The gate failed open four ways, three of them found after the retrofit shipped.** An unrecognised follower token made a dangling target vanish. A skill directory with no `SKILL.md` resolved as a valid target, so a commit could be green locally and red in a fresh clone — three existing fixtures were relying on that, one of which made the install-leak A/B pass vacuously. Then the free-standing `/name` sweep turned out to be gated on the sentence carrying a boundary marker, so route notation in any other sentence was invisible — not an ERROR, not a SUGGESTION, not an INFO — which left the documented "`/name` always blocks" promise false from a second direction. All four fixed and pinned. **Two checks were silently not running.** `validate-provenance.sh` checks 7-8 were dead across nine skills. Waking them exposed a deeper problem: they assume `Research doc:` names a source index, but 30 of 121 entries point at topic content documents, so every new check-7 INFO was a false positive and check 8 was saved from a false-FAIL flood only by an *unannounced* skip. Checks 7/8 are now scoped to source indexes and every skip announces itself (#121). **The retrofit's own anti-goal, four times.** ADR-0020 warns that a blunt gate gets satisfied by deleting content rather than relocating it. `diagnose` and `skill-audit` relocated prose and then read it unconditionally; `prototype` and `vale-config` deleted rules outright that survived nowhere. All four addressed. ## Verification - `bash tests/run-tests.sh --strict` — 24 suites, 0 skipped, 0 failed - `bash tests/run-bats.sh` — 325 tests, 0 failures - `pre-commit run --all-files` — 17/17 - `pre-commit run --hook-stage pre-push --all-files` — 16/16, with `apm marketplace check` and `apm pack --check-clean` run against the remote, not skipped - `scripts/skill-size-check.sh` over all 39 skills — rc 0, 0 ERROR/FAIL, SUGGESTION-only - Preload tax measured at **10,498 chars**, max description 390 — both inside budget - Every new test proven non-vacuous by a deliberate mutation of the behaviour it covers **Per-commit sync, stated accurately:** the ten commits from the latest review round each pass `check-plugin-content-sync` in isolation, verified by checking each out in a detached worktree with a clean between. The earlier gitea window (`dfacf05..bedbd1d`, nine commits) does **not** — its mirror was regenerated in one batch at `bbc7300`. An earlier revision of this description claimed the property held for every commit; it does not, and a bisect through that window lands on a red commit. **Squash-merge** to collapse it, or accept that this range is not bisectable. ## Version bump Six plugins and the catalog take a **patch**, not a minor. The branch is **89 commits — 40 `fix` / 30 `refactor` / 12 `docs` / 5 `chore` / 2 `test` — zero `feat`, zero `!`, zero `BREAKING CHANGE`** — and adds no skill, agent, command or hook. (Two earlier revisions of this section cited a stale histogram, most recently 78 commits; the figures above are measured at HEAD.) Both rules this repo ships (`forge/references/version-bump.md`, landing in this PR, and `git-commits/references/conventional-commits-spec.md`) make that a patch, and the catalog set is unchanged at 7 entries. Not settled by that: four published files were removed from the installed tree, three moved, and `caveman` gained `disable-model-invocation`, retiring its old triggers. Under a strict reading those are major-class and currently ship under `refactor:` with no marker. Whether the deployed skill surface is a public contract is written down nowhere — worth deciding, but it outlives this PR. ## Deliberately not in scope #112 (cherry-pick ownership, now resolved in favour of `git-commits`), #113 (`rtk git` normalisation), #116 (research fan-out), #101 (audit-skill merge), #122 (non-spec skill-root files), #123 (no PRD producer) stay open. #117 is the one worth reading: the contract's remedy is to move prose into `references/`, which is exactly where neither the size gate nor Vale looks — and the blind spot is wider than #117 currently records, since there is no root `.vale.ini` at all, so every ADR, `CONTEXT.md` and `README.md` is unlinted too. That blind spot let this branch carry two `level: error` `Kyberforge.SentenceOpenerThereIs` violations into `references/` files it created — `provider-adapter-author/references/provider-matrix.md:31` and `agent-audit/references/finding-criteria.md:95`. Both are reworded in `afadaae`, confirmed by routing each file through the audit's own `vale-wrap.sh` (1 error each before, 0 after). Five further occurrences sit in `references/` files already on `main`; those are the pre-existing corpus and stay with #117, which is the real fix. Also unfixed and not this PR's: `apm install` appends a duplicate `SessionStart` entry to `.claude/settings.json`, so a fresh clone cannot get pre-push green without an edit AGENTS.md warns against. Reproduces identically on `main`. Co-authored-by: Defame1297 <gitea@rkdr.net> Reviewed-on: https://git.dev.rkdr.net/Defame1297/holocron/pulls/129 Co-authored-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net> Co-committed-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net>
776 lines
30 KiB
Bash
Executable File
776 lines
30 KiB
Bash
Executable File
#!/usr/bin/env bats
|
|
|
|
setup() {
|
|
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../../" && pwd)"
|
|
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
|
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
|
|
|
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
|
TMPDIR="$(mktemp -d)"
|
|
|
|
# Helper: create a minimal valid skill directory.
|
|
#
|
|
# The description carries a boundary clause deliberately. ADR-0020's
|
|
# missing-boundary-clause SUGGESTION fires on any description without one, so
|
|
# a fixture that omits it is never "otherwise clean" — every test asserting
|
|
# SUGGESTION-freedom would be asserting the boundary check's absence instead
|
|
# of the thing it names. "anything else" is not hyphenated, so the clause adds
|
|
# a boundary marker without adding a routing target to resolve.
|
|
make_valid_skill() {
|
|
local dir="$1"
|
|
local name
|
|
name="$(basename "$dir")"
|
|
mkdir -p "$dir/scripts"
|
|
cat > "$dir/SKILL.md" <<EOF
|
|
---
|
|
name: $name
|
|
description: A valid skill description that is well within the limit. Do not use for anything else.
|
|
---
|
|
|
|
## Step 1
|
|
|
|
Do the thing.
|
|
EOF
|
|
}
|
|
|
|
# Helper: a description of EXACTLY <n> characters that carries a boundary
|
|
# clause and names no routing target. The tests below measure the description
|
|
# LENGTH, so the clause has to be paid for out of the same budget rather than
|
|
# appended to it — hence the padding arithmetic instead of a fixed suffix.
|
|
desc_of_length() {
|
|
python3 - "$1" <<'PY'
|
|
import sys
|
|
n = int(sys.argv[1])
|
|
prefix = 'Use when doing the thing. Do not use for anything else. '
|
|
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
|
print(prefix + 'x' * (n - len(prefix)))
|
|
PY
|
|
}
|
|
|
|
# Helper: create a skill directory with an exact description length and an
|
|
# exact body word count. <desc> is used verbatim; <body_words> "word"
|
|
# tokens follow the frontmatter. Used by the ADR-0020 boundary tests.
|
|
make_sized_skill() {
|
|
local dir="$1" desc="$2" body_words="$3"
|
|
local name
|
|
name="$(basename "$dir")"
|
|
mkdir -p "$dir"
|
|
{
|
|
echo "---"
|
|
echo "name: $name"
|
|
echo "description: $desc"
|
|
echo "---"
|
|
echo ""
|
|
python3 -c "print(' '.join(['word'] * $body_words))"
|
|
} > "$dir/SKILL.md"
|
|
}
|
|
|
|
# Helper: build a self-contained fixture plugin tree so the boundary-target
|
|
# resolver has a real authoring source to resolve against, independent of
|
|
# this repo's live skills. Echoes the subject skill's directory.
|
|
#
|
|
# <root>/plugins/fixture-plugin/.apm/skills/<subject>/SKILL.md
|
|
# <root>/plugins/fixture-plugin/.apm/skills/fixture-sibling-skill/SKILL.md
|
|
# <root>/plugins/fixture-plugin/.apm/agents/fixture-sibling-agent.agent.md
|
|
#
|
|
# The sibling gets a real SKILL.md, and that is load-bearing rather than
|
|
# tidiness: a directory under skills/ is a resolvable name only when it
|
|
# HOLDS one. An empty leftover directory is untracked by git, so counting
|
|
# one made a target resolve on the machine that made it and dangle in a
|
|
# fresh clone. This helper used to mkdir the sibling and write nothing into
|
|
# it, so the corroborator every blocking-tier test depends on silently
|
|
# stopped resolving the moment that rule was enforced.
|
|
make_fixture_tree() {
|
|
local root="$1" subject="$2"
|
|
local apm="$root/plugins/fixture-plugin/.apm"
|
|
mkdir -p "$apm/skills/$subject" "$apm/skills/fixture-sibling-skill" "$apm/agents"
|
|
touch "$apm/agents/fixture-sibling-agent.agent.md"
|
|
make_sized_skill "$apm/skills/fixture-sibling-skill" \
|
|
"Use when doing the other thing. Do not use for anything else." 10
|
|
echo "$apm/skills/$subject"
|
|
}
|
|
}
|
|
|
|
teardown() {
|
|
rm -rf "$TMPDIR"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Passing cases
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "passes on a valid minimal skill" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
@test "--help exits 0" {
|
|
run bash "$SCRIPT" --help
|
|
assert_success
|
|
assert_output --partial "Usage:"
|
|
}
|
|
|
|
@test "passes when scripts/ directory is absent" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
rmdir "$skill/scripts"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
@test "FILL IN: inside backticks does not fail" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
echo "Use \`FILL IN: value\` as an example." >> "$skill/SKILL.md"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
@test "the 1024-char agentskills.io spec backstop is unchanged and separate from the ADR-0020 ceiling" {
|
|
local skill="$TMPDIR/my-skill"
|
|
local name
|
|
name="$(basename "$skill")"
|
|
mkdir -p "$skill"
|
|
local desc
|
|
desc="$(python3 -c "print('x' * 1024)")"
|
|
cat > "$skill/SKILL.md" <<EOF
|
|
---
|
|
name: $name
|
|
description: $desc
|
|
---
|
|
|
|
## Step 1
|
|
|
|
Do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$skill"
|
|
# Two independent gates on one value: the spec limit still PASSES at
|
|
# exactly 1024 (its own boundary is unmoved), while ADR-0020's 400-char
|
|
# ceiling FAILs. The run fails on the second, not the first.
|
|
assert_output --partial "description length 1024 chars (agentskills.io spec limit: 1024)"
|
|
assert_output --partial "400-character ADR-0020 ceiling"
|
|
assert_failure
|
|
}
|
|
|
|
@test "passes at exactly 500 lines" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
local current
|
|
current="$(wc -l < "$skill/SKILL.md")"
|
|
local needed=$(( 500 - current ))
|
|
python3 -c "print('\n' * $needed, end='')" >> "$skill/SKILL.md"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Failing cases
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "fails when SKILL.md is missing" {
|
|
local skill="$TMPDIR/my-skill"
|
|
mkdir -p "$skill"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when name does not match directory" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
sed -i 's/^name: .*/name: wrong-name/' "$skill/SKILL.md"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when description exceeds 1024 chars" {
|
|
local skill="$TMPDIR/my-skill"
|
|
local name
|
|
name="$(basename "$skill")"
|
|
mkdir -p "$skill"
|
|
local desc
|
|
desc="$(python3 -c "print('x' * 1025)")"
|
|
cat > "$skill/SKILL.md" <<EOF
|
|
---
|
|
name: $name
|
|
description: $desc
|
|
---
|
|
|
|
## Step 1
|
|
|
|
Do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when SKILL.md exceeds 500 lines" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
python3 -c "print('\n' * 500)" >> "$skill/SKILL.md"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when body contains unfilled FILL IN: placeholder" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
echo "FILL IN: replace this" >> "$skill/SKILL.md"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when a script is not executable" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
echo "#!/usr/bin/env bash" > "$skill/scripts/helper.sh"
|
|
chmod -x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when a script has an interactive prompt" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
printf '#!/usr/bin/env bash\nread -p "Enter value: " VAL\n' > "$skill/scripts/helper.sh"
|
|
chmod +x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when a script reads a variable with no redirect" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
printf '#!/usr/bin/env bash\nread -r ANSWER\n' > "$skill/scripts/helper.sh"
|
|
chmod +x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when an interactive prompt string contains an angle bracket" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
printf '#!/usr/bin/env bash\nread -p "enter <name>: " NAME\n' > "$skill/scripts/helper.sh"
|
|
chmod +x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "passes when a script reads from a here-string" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
printf '#!/usr/bin/env bash\nLINE="a b"\nread -r X Y <<< "$LINE"\n' \
|
|
> "$skill/scripts/helper.sh"
|
|
chmod +x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
@test "passes when a script reads from a here-doc" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
printf '#!/usr/bin/env bash\nread -r X <<EOF\nvalue\nEOF\n' > "$skill/scripts/helper.sh"
|
|
chmod +x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
@test "passes when a script reads from a file redirect" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
printf '#!/usr/bin/env bash\nread -r LINE < "$1"\n' > "$skill/scripts/helper.sh"
|
|
chmod +x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
@test "passes when a script reads from a pipe continued onto the next line" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_valid_skill "$skill"
|
|
printf '#!/usr/bin/env bash\nprintf %%s "$1" |\n read -r X\n' > "$skill/scripts/helper.sh"
|
|
chmod +x "$skill/scripts/helper.sh"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
}
|
|
|
|
@test "fails when name contains consecutive hyphens" {
|
|
local skill="$TMPDIR/my--skill"
|
|
make_valid_skill "$skill"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when name has a leading hyphen" {
|
|
local skill="$TMPDIR/-my-skill"
|
|
make_valid_skill "$skill"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when no frontmatter block is present" {
|
|
local skill="$TMPDIR/my-skill"
|
|
mkdir -p "$skill"
|
|
echo "Just some content with no frontmatter." > "$skill/SKILL.md"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when no arguments are given" {
|
|
run bash "$SCRIPT"
|
|
assert_failure
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — description budget (250 SUGGESTION / 400 FAIL)
|
|
#
|
|
# These sit UNDER the agentskills.io 1024-character spec backstop above, which
|
|
# is unchanged. Both ceilings are inclusive: exactly at the number passes that
|
|
# tier, one past it trips.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "ADR-0020: description of exactly 250 chars raises no suggestion" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "$(desc_of_length 250)" 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "SUGGESTION"
|
|
}
|
|
|
|
@test "ADR-0020: description of 251 chars raises a SUGGESTION and still exits 0" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "$(desc_of_length 251)" 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "SUGGESTION"
|
|
assert_output --partial "description is 251 chars"
|
|
assert_output --partial "All checks passed (1 suggestion(s))."
|
|
}
|
|
|
|
@test "ADR-0020: description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "$(desc_of_length 400)" 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "SUGGESTION"
|
|
}
|
|
|
|
@test "ADR-0020: description of 401 chars FAILs and exits non-zero" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "$(desc_of_length 401)" 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "description is 401 chars"
|
|
assert_output --partial "400-character ADR-0020 ceiling"
|
|
}
|
|
|
|
@test "ADR-0020: description length is measured after YAML folding is resolved" {
|
|
local skill="$TMPDIR/my-skill"
|
|
mkdir -p "$skill"
|
|
# A >-folded block scalar: 11 lines of 40 chars folded with 10 joining
|
|
# spaces = 450 characters. Measured off its raw `description: >` line it is
|
|
# 1 character and passes; measured as the folded VALUE it must FAIL. This
|
|
# is exactly the case a line-wise regex gets wrong.
|
|
{
|
|
echo "---"
|
|
echo "name: my-skill"
|
|
echo "description: >"
|
|
python3 -c "print('\n'.join([' ' + 'x' * 40] * 11))"
|
|
echo "---"
|
|
echo ""
|
|
echo "Do the thing."
|
|
} > "$skill/SKILL.md"
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "description is 450 chars"
|
|
assert_output --partial "400-character ADR-0020 ceiling"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — body budget (600 SUGGESTION / 900 FAIL), body ONLY
|
|
#
|
|
# Distinct from the 2,770-word whole-file spec ceiling above, which counts
|
|
# frontmatter too and is unchanged. Do not unify them.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "ADR-0020: body of exactly 600 words raises no suggestion" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 600
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "SUGGESTION"
|
|
}
|
|
|
|
@test "ADR-0020: body of 601 words raises a SUGGESTION and still exits 0" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 601
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "body is 601 words"
|
|
assert_output --partial "All checks passed (1 suggestion(s))."
|
|
}
|
|
|
|
@test "ADR-0020: body of exactly 900 words is a SUGGESTION, not a FAIL" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 900
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "body is 900 words"
|
|
}
|
|
|
|
@test "ADR-0020: body of 901 words FAILs and exits non-zero" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" "A short valid description. Do not use for anything else." 901
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "body is 901 words"
|
|
assert_output --partial "900-word ADR-0020 ceiling"
|
|
}
|
|
|
|
@test "ADR-0020: the body gate counts the body only — frontmatter words do not count toward it" {
|
|
local skill="$TMPDIR/my-skill"
|
|
# 895 body words plus a description long enough that the WHOLE FILE is well
|
|
# over 900 words. The body gate must stay silent; the 2,770-word whole-file
|
|
# ceiling is a separate measurement and is nowhere near tripping.
|
|
make_sized_skill "$skill" "$(python3 -c "print(' '.join(['w'] * 100))")" 895
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "900-word ADR-0020 ceiling"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — resolvable boundary targets
|
|
#
|
|
# Resolved against the AUTHORING SOURCE (plugins/*/.apm/skills/ and
|
|
# plugins/*/.apm/agents/), never .claude/skills/, so the check works offline and
|
|
# before an apm install. Every fixture below builds its own plugin tree rather
|
|
# than leaning on this repo's live skills.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "ADR-0020: a boundary target naming an existing sibling skill resolves" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-sibling-skill instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "boundary target(s) resolve"
|
|
}
|
|
|
|
@test "ADR-0020: a boundary target naming a non-existent skill FAILs when its sentence names one that resolves" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# `fixture-sibling-skill` is the corroborator: a prose-form target only earns
|
|
# a FAIL when its own sentence proves it is a routing sentence. See the
|
|
# shared resolver's CORROBORATION note, and the uncorroborated case below.
|
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-sibling-skill or fixture-missing-skill instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "routes to 'fixture-missing-skill'"
|
|
}
|
|
|
|
@test "ADR-0020: a LONE boundary target naming a non-existent skill is a SUGGESTION, not a FAIL" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# Same grammar as the case above and as "run \`pre-commit\` instead" — a
|
|
# route verb, a hyphenated name, terminal position. Nothing local separates a
|
|
# broken route from a tool name, so the target is named on every run but does
|
|
# not block: this gate ships with no baseline and no suppression mechanism.
|
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use fixture-missing-skill instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "SUGGESTION"
|
|
assert_output --partial "routes to 'fixture-missing-skill'"
|
|
refute_output --partial "FAIL description routes to"
|
|
}
|
|
|
|
@test "ADR-0020: a boundary target naming an AGENT file resolves (agents are valid routing targets)" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when doing the thing. Do not use when the caller is an agent — invoke fixture-sibling-agent instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "boundary target(s) resolve"
|
|
}
|
|
|
|
@test "ADR-0020: a /slash-command boundary target that does not resolve FAILs" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when doing the thing. Do not use when improvements are wanted — use /fixture-missing-improve instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "routes to 'fixture-missing-improve'"
|
|
}
|
|
|
|
@test "ADR-0020: a backticked name that does not resolve FAILs when its sentence names one that resolves" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when doing the thing. Composes \`fixture-sibling-skill\` and \`fixture-missing-helper\` for the shared part." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "routes to 'fixture-missing-helper'"
|
|
}
|
|
|
|
@test "ADR-0020: a /slash-command target is route NOTATION and FAILs on its own, uncorroborated" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# The escape hatch from the SUGGESTION tier: `/name` and `-> name` are never
|
|
# how prose cites a tool, so they are exempt from corroboration. An author
|
|
# who wants a route checked unconditionally writes one of those two forms.
|
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use /fixture-missing-notation instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "routes to 'fixture-missing-notation'"
|
|
}
|
|
|
|
@test "ADR-0020: a bare hyphenated word outside a boundary sentence is not read as a routing target" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# "run pre-commit hooks" is pc-run's real phrasing. A naive extractor reads
|
|
# it as a route to a non-existent `pre-commit` skill.
|
|
make_sized_skill "$skill" "Use when the user wants to run pre-commit hooks or install git hooks." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "pre-commit"
|
|
}
|
|
|
|
@test "ADR-0020: an arrow chain outside a boundary clause is not read as a routing target" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# diagnose's real process chain. Only ADR-0020's `Not <thing> -> <skill>`
|
|
# form makes a bare arrow target a route.
|
|
make_sized_skill "$skill" "Reproduce → minimise → instrument → fix → regression-test. Use when a bug is reported." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "regression-test"
|
|
}
|
|
|
|
@test "ADR-0020: MCP tool names and capitalised tool names are not read as routing targets" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when writing issues. Do not use for local files (use Read/Write/Edit) — that write goes through \`issue_write\`/\`pull_request_write\` instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "routes to"
|
|
}
|
|
|
|
@test "ADR-0020: ADR's compressed boundary form (Not <thing> -> <skill>) is checked" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when doing the thing. Not the other thing → fixture-missing-target." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "routes to 'fixture-missing-target'"
|
|
}
|
|
|
|
@test "ADR-0020: the boundary check declines rather than false-FAILs when no authoring source is found" {
|
|
# Deliberately NOT built with make_fixture_tree: this skill sits in a bare
|
|
# temp directory with no plugins/*/.apm/ above it and no .git, so the resolver
|
|
# legitimately has no universe. That is a real path (a skill being drafted
|
|
# outside any repo), and the required behaviour is to DECLINE OUT LOUD rather
|
|
# than either false-FAIL or pass in silence — silence is what let a whole gate
|
|
# family go missing unnoticed. So the INFO text and the named unchecked target
|
|
# are both asserted, not just the absence of a failure.
|
|
local skill="$TMPDIR/orphan/my-skill"
|
|
make_sized_skill "$skill" "Use when doing the thing. Do not use for the other thing — use some-other-skill instead." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "routes to"
|
|
assert_output --partial "boundary-target resolution DID NOT RUN"
|
|
assert_output --partial "Unchecked target(s): some-other-skill"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — the hand-invocation carve-out (issue #108)
|
|
#
|
|
# A skill carrying `disable-model-invocation: true` is absent from the
|
|
# model-visible listing entirely: not preloaded, and the Skill tool refuses to
|
|
# call it. Its description is never matched against user intent, so
|
|
# references/description-quality.md Step 0 gives it ONE plain human-facing
|
|
# sentence — no trigger list, no boundary clause — and calls a
|
|
# missing-boundary-clause finding on such a skill "a wrong finding, not a strict
|
|
# one". Until this ran, nothing here knew the field existed, so the audit
|
|
# reported exactly the shape its own rubric mandates, with advice naming a
|
|
# router that cannot see the skill.
|
|
#
|
|
# The carve-out is narrow. Both size gates are unaffected and both are pinned
|
|
# below: the body is loaded on invocation like any other body, and the
|
|
# 400-character ceiling is an outlier stop rather than a routing budget.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
# Helper: a skill directory carrying `disable-model-invocation: true`.
|
|
make_hand_invoked_skill() {
|
|
local dir="$1" desc="$2" body_words="$3"
|
|
local name
|
|
name="$(basename "$dir")"
|
|
mkdir -p "$dir"
|
|
{
|
|
echo "---"
|
|
echo "name: $name"
|
|
echo "description: $desc"
|
|
echo "disable-model-invocation: true"
|
|
echo "---"
|
|
echo ""
|
|
python3 -c "print(' '.join(['word'] * $body_words))"
|
|
} > "$dir/SKILL.md"
|
|
}
|
|
|
|
@test "ADR-0020: a hand-invoked skill is not asked for a boundary clause" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_hand_invoked_skill "$skill" \
|
|
"Tell the agent to zoom out and give broader context or a higher level perspective." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "has no boundary clause"
|
|
assert_output --partial "hand-invoked"
|
|
}
|
|
|
|
@test "ADR-0020: the SAME description without the flag IS asked for a boundary clause" {
|
|
# The control. Without it the case above is satisfied by an audit that
|
|
# stopped checking boundary clauses altogether.
|
|
local skill="$TMPDIR/my-skill"
|
|
make_sized_skill "$skill" \
|
|
"Tell the agent to zoom out and give broader context or a higher level perspective." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "has no boundary clause"
|
|
}
|
|
|
|
@test "ADR-0020: a hand-invoked skill is exempt from the 250-character description target" {
|
|
local skill="$TMPDIR/my-skill"
|
|
make_hand_invoked_skill "$skill" \
|
|
"$(python3 -c "print('Tell the agent to zoom out. ' + 'x' * 273)")" 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "over the 250-character"
|
|
}
|
|
|
|
@test "ADR-0020: a hand-invoked description over 400 chars still FAILS" {
|
|
# The half the carve-out does NOT lift. 400 is an outlier stop, not a
|
|
# routing-quality target: a hand-invoked description is still the one line
|
|
# the user reads when choosing from the `/` menu.
|
|
local skill="$TMPDIR/my-skill"
|
|
make_hand_invoked_skill "$skill" \
|
|
"$(python3 -c "print('Tell the agent to zoom out. ' + 'x' * 374)")" 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "400-character"
|
|
}
|
|
|
|
@test "ADR-0020: a hand-invoked body over 900 words still FAILS" {
|
|
# The body is loaded on invocation exactly like any other body and competes
|
|
# with the caller's live conversation the same way, so no body tier moves.
|
|
local skill="$TMPDIR/my-skill"
|
|
make_hand_invoked_skill "$skill" "Tell the agent to zoom out." 901
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "900-word"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — one arrow, one target (issue #107)
|
|
#
|
|
# Only the FIRST target after an arrow was resolved: the conjunction
|
|
# continuation is wired to the prose route verbs and never to arrows. So this
|
|
# script printed "1 of 1 boundary target(s) resolve" on a clause naming two,
|
|
# and the second was resolved by nothing and reported by nothing. A typo in it
|
|
# shipped through a green gate. The shape is now rejected rather than the
|
|
# extractor widened.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "ADR-0020: an arrow clause naming two targets is reported, not silently half-checked" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# A bare `Not ... ->` sentence carries no BOUNDARY_MARKER, so the backtick
|
|
# sweep does not run and the second target is invisible to every other rule
|
|
# in the resolver — this is the exact shape #107 measured.
|
|
make_sized_skill "$skill" "Use when doing the thing. Not the other thing -> \`fixture-sibling-skill\` or \`fixture-missing-second\`." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
assert_output --partial "names more than one target"
|
|
}
|
|
|
|
@test "ADR-0020: one arrow per target — the convention the suggestion asks for — is silent" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when doing the thing. Not the other thing -> \`fixture-sibling-skill\`." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "names more than one target"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — a dotted filename in a boundary clause (issue #110)
|
|
#
|
|
# `[^.;]` could not cross the `.` in `AGENTS.md`, so a clause naming a dotted
|
|
# file between "Not" and the arrow was invisible. With a backticked target that
|
|
# was a MISDIAGNOSIS — "no boundary clause" reported on a clause that was
|
|
# present and working. With a BARE target it was worse: the target was never
|
|
# extracted, so the dangling check silently did not run on it.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "ADR-0020: a boundary clause naming a dotted filename is not reported as missing" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
make_sized_skill "$skill" "Use when doing the thing. Not AGENTS.md -> \`fixture-sibling-skill\`." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "has no boundary clause"
|
|
assert_output --partial "description has a boundary clause"
|
|
}
|
|
|
|
@test "ADR-0020: a BARE target after a dotted filename is extracted and checked" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# The silent half of #110: this clause produced no target at all, so it was
|
|
# neither resolved nor reported — a route to a non-existent skill shipping
|
|
# through a green gate with no finding of any kind.
|
|
make_sized_skill "$skill" "Use when doing the thing. Not AGENTS.md -> fixture-missing-dotted." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_failure
|
|
assert_output --partial "routes to 'fixture-missing-dotted'"
|
|
}
|
|
|
|
@test "ADR-0020: an arrow clause yielding no target is reported as unparsed, not as missing" {
|
|
local skill
|
|
skill="$(make_fixture_tree "$TMPDIR/tree" "my-skill")"
|
|
# A single-word target is deliberately not matchable bare, because
|
|
# `research`, `triage` and `forge` are all skill names AND ordinary English.
|
|
# The clause is present; saying it is missing sends the author to add a
|
|
# second copy of a clause that is already there.
|
|
make_sized_skill "$skill" "Use when doing the thing. Not the other thing -> forge." 10
|
|
run bash "$SCRIPT" "$skill"
|
|
assert_success
|
|
refute_output --partial "has no boundary clause"
|
|
assert_output --partial "no target could be read"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Encoding, write side: sys.stdout/stderr.reconfigure(encoding='utf-8')
|
|
#
|
|
# read_text() in the shared resolver block pins the READS to UTF-8. That moved
|
|
# the LC_ALL=C crash to the WRITE: this script's own message text carries em
|
|
# dashes (the ADR-0020 boundary SUGGESTION is one), so the streams' ASCII
|
|
# default raised UnicodeEncodeError while PRINTING — after every check had
|
|
# already run, losing the whole report at the last step.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "under LC_ALL=C the report is printed, not lost to a UnicodeEncodeError" {
|
|
local dir="$TMPDIR/locale-skill"
|
|
mkdir -p "$dir"
|
|
cat > "$dir/SKILL.md" <<EOF
|
|
---
|
|
name: locale-skill
|
|
description: A valid skill description that is well within the limit.
|
|
---
|
|
|
|
## Step 1
|
|
|
|
Do the thing.
|
|
EOF
|
|
run env LC_ALL=C PYTHONUTF8=0 bash "$SCRIPT" "$dir"
|
|
assert_success
|
|
assert_output --partial "description has no boundary clause"
|
|
refute_output --partial "UnicodeEncodeError"
|
|
refute_output --partial "Traceback"
|
|
}
|