Defame1297
b6e68e9a2b
Three ways the gates could report green having measured nothing. All three were
invisible to a passing test suite, because pre-commit prints nothing at all for a
hook that exits 0 — a gate that declines to check and a gate that checked and
passed produce the identical signal.
- A UTF-8 BOM, a leading blank line, a trailing space after a `---` marker or
CRLF line endings defeated the `^---\n` frontmatter matcher. Every ADR-0020
check was then skipped and the file passed: measured at the time, a
550-character description with a 1,000-word body exited 0 behind a BOM.
All four shapes are now tolerated, and frontmatter that genuinely cannot be
parsed is a hard ERROR rather than a silent skip.
- An agent file with a valueless `description:` followed by another key let a
line regex capture the *next* key, which looked non-empty, so the
missing-or-empty branch never fired and every gate below it early-returned on
the empty folded value — zero output, exit 0, on a blocking gate. The one
field this contract is entirely about was the one field a gate could fail to
notice was absent. Presence is now decided on the YAML-folded value and
nowhere else, and a missing or empty description is a hard FAIL in all three
validators.
- The hand-rolled frontmatter fallback disagreed with PyYAML across the FAIL
boundary on folded scalars, so which reader happened to be available decided
the verdict. A fallback that mis-parses a scalar shape reports a vacuous pass,
which is worse than not running, so it is deleted: python3 and PyYAML are hard
requirements that fail loudly with an install pointer.
Boundary-target resolution no longer derives its universe from its own location.
A `${BASH_SOURCE}`-relative repo root leaked this repo's 39-skill universe into
every consumer repo running the hook through pre-commit, so a consumer skill
routing to `skill-audit` resolved against a plugin it had never installed. The
interim form resolved through `.claude/` and `.agents/`, which are gitignored
`apm install` output — the same commit reported 2 dangling targets on a machine
that had run the install and 6 on a fresh clone. Resolution now walks up from the
file being checked to an authoring root (nearest ancestor holding
`plugins/*/.apm/{skills,agents}`, else the nearest `.git`, in two passes so a
nested `.git` cannot outrank a real monorepo root); the universe is every skill
and agent under `<root>/plugins/*/` plus the file's own apm package and that
package's declared `dependencies.apm`. Deployed trees are consulted only when no
authoring root exists at all — the consumer case. One commit now gets one verdict,
which a gate shipping hot with no baseline file has to.
Narrowed in the same pass: a routing target inferred from the prose boundary form
and corroborated by nothing else reports at SUGGESTION instead of blocking. A
blocking check with no escape hatch is the wrong trade when the inference from
prose is the weak part of it.
New deterministic checks, all previously untested or absent: every
`references/<file>.md` a body names must exist (ERROR — a broken pointer is not a
style opinion); a description with no boundary clause at all, a Gotchas section
over five entries, and a Gotchas section over 25% of the body are SUGGESTIONs.
Where no universe can be determined the target check prints `INFO ... DID NOT
RUN` rather than passing quietly. Each prose-scanning check needed its own
false-positive fix — a fenced example of a Gotchas section was being read as the
section itself — and those fixes are pinned rather than assumed.
The resolver is one block copied verbatim into all three scripts between
BEGIN/END markers, because a cache-installed plugin's scripts cannot read outside
their own plugin directory. Nothing asserted the copies were still identical; a
one-line edit to a single copy passed every constant-agreement assertion, since
constants are not what drifts.
Tests land here rather than in a later commit. The existing suites assert the old
behaviour and go red against these scripts, so splitting them would leave a commit
whose own `run-tests` pre-push gate fails in isolation.
Refs: ADR-0020
813 lines
26 KiB
Bash
813 lines
26 KiB
Bash
#!/usr/bin/env bats
|
|
|
|
setup() {
|
|
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../../" && pwd)"
|
|
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
|
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
|
|
|
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
|
TMPDIR="$(mktemp -d)"
|
|
|
|
# Helper: create an APM package root at <root> (apm.yml with a top-level
|
|
# type: line, marking it a real package manifest — not marketplace-only)
|
|
# plus a single vendor-neutral agent file at
|
|
# <root>/.apm/agents/<name>.agent.md. <extra_frontmatter>, if given, is
|
|
# inserted as additional raw frontmatter lines (used to inject fields
|
|
# under test).
|
|
make_apm_agent() {
|
|
local root="$1"
|
|
local name="$2"
|
|
local extra_frontmatter="${3:-}"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
|
|
---
|
|
name: ${name}
|
|
description: A valid agent description.
|
|
${extra_frontmatter}
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
}
|
|
|
|
# Helper: a description of EXACTLY <n> characters that carries a boundary
|
|
# clause and names no routing target. ADR-0020's missing-boundary-clause
|
|
# SUGGESTION fires on any description without one, so a fixture that omits it
|
|
# is never "otherwise clean" and a test refuting SUGGESTION would be asserting
|
|
# the boundary check's absence instead of the thing it names. The clause is
|
|
# paid for out of the measured budget rather than appended to it, because
|
|
# these tests measure the description LENGTH. "anything else" is not
|
|
# hyphenated, so no routing target comes with it.
|
|
desc_of_length() {
|
|
python3 - "$1" <<'PY'
|
|
import sys
|
|
n = int(sys.argv[1])
|
|
prefix = 'Use when doing the thing. Do not use for anything else. '
|
|
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
|
print(prefix + 'x' * (n - len(prefix)))
|
|
PY
|
|
}
|
|
|
|
# Helper: same shape as make_apm_agent, but the description is supplied
|
|
# verbatim — used by the ADR-0020 description-budget tests.
|
|
make_apm_agent_with_desc() {
|
|
local root="$1" name="$2" desc="$3"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
|
|
---
|
|
name: ${name}
|
|
description: ${desc}
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
}
|
|
}
|
|
|
|
teardown() {
|
|
rm -rf "$TMPDIR"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Passing cases — project/user scope (unchanged)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "passes on a clean project-scope pair (CC file as input)" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "user scope: \$HOME being a dotfiles .git repo does not shadow user scope" {
|
|
local fake_home="$TMPDIR/fakehome"
|
|
mkdir -p "$fake_home/.git" "$fake_home/.claude/agents" "$fake_home/.copilot/agents"
|
|
cat > "$fake_home/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$fake_home/.copilot/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "user scope: agent file directly in \$HOME (start dir IS exactly \$HOME, no walk-up) resolves to user scope" {
|
|
local fake_home="$TMPDIR/fakehome-direct"
|
|
mkdir -p "$fake_home" "$fake_home/.copilot/agents"
|
|
cat > "$fake_home/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$fake_home/.copilot/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: a nested marker-less directory walked up into \$HOME resolves to project scope, not user scope (live repro of new-agent.sh's stray-directory case)" {
|
|
local fake_home="$TMPDIR/fakehome-nested"
|
|
local nested="$fake_home/scratch/testdir"
|
|
mkdir -p "$nested/.claude/agents" "$nested/.github/agents"
|
|
cat > "$nested/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$nested/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$nested/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: nested marker-less dir under \$HOME does NOT look for a counterpart under the shared \$HOME/.copilot or \$HOME/.github dirs" {
|
|
local fake_home="$TMPDIR/fakehome-nested2"
|
|
local nested="$fake_home/scratch/testdir"
|
|
mkdir -p "$nested/.claude/agents" "$fake_home/.copilot/agents"
|
|
cat > "$nested/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
# Decoy counterpart at the *user*-scope location — if scope were
|
|
# misclassified as 'user' (the pre-fix bug), validate.sh would find this
|
|
# unrelated file and (wrongly) pass.
|
|
cat > "$fake_home/.copilot/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$nested/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "counterpart file not found"
|
|
}
|
|
|
|
@test "project scope: filesystem-root fallback (no \$HOME in path, no markers found) resolves to project scope, not user scope" {
|
|
local unrelated_home="$TMPDIR/unrelated-home-never-reached"
|
|
local root="$TMPDIR/no-home-relation/deep/proj"
|
|
mkdir -p "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$unrelated_home" bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: <root> one level below a .git ancestor resolves scope to <root>, not to wherever .git was found (subdirectory of a larger git-tracked tree)" {
|
|
local repo="$TMPDIR/repo-with-subdir"
|
|
local root="$repo/subdir"
|
|
mkdir -p "$repo/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
# new-agent.sh, invoked with <root> as its root argument, would place the
|
|
# counterpart at <root>/.github/agents — not at the repo root's
|
|
# .github/agents, even though .git lives at the repo root one level up.
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: a non-conventional path (agent file not directly under a literal 'agents' dir) falls back to the nearest .git boundary instead of two-segments-up arithmetic" {
|
|
local outer="$TMPDIR/outer-repo"
|
|
local pkg="$outer/pkgA"
|
|
mkdir -p "$pkg/.git" "$pkg/.github/agents" "$pkg/extra"
|
|
# Misplaced file: sits two path segments below $outer (pkgA/extra), which
|
|
# matches the conventional_root arithmetic by coincidence, but its
|
|
# immediate parent dir is "extra", not "agents" — conventional_shape is
|
|
# false, so the fix must fall back to the nearest .git boundary (pkgA),
|
|
# not trust $outer.
|
|
cat > "$pkg/extra/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
# Counterpart at the nearest-.git root (pkgA), not at $outer — if the
|
|
# arithmetic were trusted here, validate.sh would look for a counterpart
|
|
# at $outer/.github/agents/my-agent.agent.md, which doesn't exist, and
|
|
# false-FAIL.
|
|
cat > "$pkg/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$pkg/extra/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "counterpart file not found"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Failing cases — project/user scope: CC/Copilot pair checks
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "fails when a CC-only field ('maxTurns') is present in a project-scope Copilot file" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
maxTurns: 10
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "CC-only field"
|
|
assert_output --partial "maxTurns"
|
|
}
|
|
|
|
@test "fails when a Copilot-only field ('target') is present in a project-scope CC file" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
target: cli
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "Copilot-only field"
|
|
assert_output --partial "target"
|
|
}
|
|
|
|
@test "fails when the Copilot counterpart is missing at project scope" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "counterpart file not found"
|
|
}
|
|
|
|
@test "--help exits 0 and shows Usage:" {
|
|
run bash "$SCRIPT" --help
|
|
assert_success
|
|
assert_output --partial "Usage:"
|
|
}
|
|
|
|
@test "fails when no arguments are given" {
|
|
run bash "$SCRIPT"
|
|
assert_failure
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Passing cases — plugin/APM scope
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "passes on a clean plugin/APM-scope agent file (name/description only)" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "passes on a clean plugin/APM-scope agent file with optional model field" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "model: claude-opus-4"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Failing cases — plugin/APM scope: allowlist violations
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "fails when 'tools' field is present in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "tools: Read Edit"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "tools"
|
|
}
|
|
|
|
@test "fails when a Claude-only field ('maxTurns') is present in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "maxTurns: 10"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "maxTurns"
|
|
}
|
|
|
|
@test "fails when a Copilot-only field ('target') is present in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "target: cli"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "target"
|
|
}
|
|
|
|
@test "fails when 'hooks' is present in a plugin/APM-scope agent file (outside allowlist)" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "hooks: {}"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "hooks"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Failing cases — plugin/APM scope: structural checks
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "fails when name is not kebab-case in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: MyAgent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "kebab"
|
|
}
|
|
|
|
@test "fails when name does not match filename stem in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: wrong-name
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "does not match filename stem"
|
|
}
|
|
|
|
@test "fails when 'name' field is missing from a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when 'description' field is missing from a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when a template HTML comment is left in plugin/APM-scope frontmatter" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
<!-- model: sonnet
|
|
Optional. Omit to inherit the runtime default. -->
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "template HTML comments"
|
|
}
|
|
|
|
@test "fails when body contains unfilled FILL IN: placeholder in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
FILL IN: replace this with your system prompt.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Plugin/APM scope: no pair, no counterpart concept
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "never raises a 'counterpart' FAIL on a clean plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "never raises a 'counterpart' FAIL on a failing plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "tools: Read"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Scope-detection walk-up
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "walk-up skips a type:-less apm.yml and finds a real package root further up" {
|
|
local root="$TMPDIR/case"
|
|
mkdir -p "$root/.apm/agents/nested/deeper"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: real-package
|
|
version: 1.0.0
|
|
type: skill
|
|
EOF
|
|
# Closer to the agent file than the real package root, but has no type:
|
|
# line — marketplace-only per monorepo-and-repo-shapes.md, must be skipped.
|
|
cat > "$root/.apm/agents/nested/apm.yml" <<EOF
|
|
name: not-a-package-manifest
|
|
version: 1.0.0
|
|
EOF
|
|
cat > "$root/.apm/agents/nested/deeper/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/nested/deeper/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "type:-less apm.yml is not treated as plugin scope — falls through to project scope" {
|
|
local root="$TMPDIR/proj-marketplace"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: marketplace-root
|
|
version: 1.0.0
|
|
marketplace:
|
|
packages: []
|
|
EOF
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — description budget (250 SUGGESTION / 400 FAIL)
|
|
#
|
|
# Agents take the SAME description gates as skills: name + description is
|
|
# preloaded into every session identically. Agents take NO body word gate — see
|
|
# the final test in this block, which pins that asymmetry.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "ADR-0020: agent description of exactly 250 chars raises no suggestion" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 250)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "SUGGESTION"
|
|
}
|
|
|
|
@test "ADR-0020: agent description of 251 chars raises a SUGGESTION and still exits 0" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 251)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
assert_output --partial "SUGGESTION"
|
|
assert_output --partial "description is 251 chars"
|
|
}
|
|
|
|
@test "ADR-0020: agent description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 400)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
assert_output --partial "SUGGESTION"
|
|
}
|
|
|
|
@test "ADR-0020: agent description of 401 chars FAILs and exits non-zero" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 401)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "description is 401 chars"
|
|
assert_output --partial "400-character ADR-0020 ceiling"
|
|
}
|
|
|
|
@test "ADR-0020: agent description length is measured after YAML folding is resolved" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
# 11 folded lines of 40 chars + 10 joining spaces = 450 characters. Read off
|
|
# the raw `description: >` line it is 1 character and passes.
|
|
{
|
|
echo "---"
|
|
echo "name: my-agent"
|
|
echo "description: >"
|
|
python3 -c "print('\n'.join([' ' + 'x' * 40] * 11))"
|
|
echo "---"
|
|
echo ""
|
|
echo "You are a test agent."
|
|
} > "$root/.apm/agents/my-agent.agent.md"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "description is 450 chars"
|
|
}
|
|
|
|
@test "ADR-0020: the description gate applies at project scope too" {
|
|
local root="$TMPDIR/project"
|
|
local desc
|
|
desc="$(python3 -c "print('x' * 401)")"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: $desc
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "400-character ADR-0020 ceiling"
|
|
}
|
|
|
|
@test "ADR-0020: agents take NO body word gate — a body far over the 900-word skill ceiling passes" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
# Deliberate asymmetry, not an oversight: a skill body is loaded into the
|
|
# caller's context and competes with the live conversation, while an agent
|
|
# body becomes the system prompt of a fresh context. ADR-0020 gates the
|
|
# former at 900 words and explicitly declines to gate the latter. If a body
|
|
# word gate is ever added here, it contradicts the ADR.
|
|
#
|
|
# The description carries a boundary clause so the ONLY thing this test can
|
|
# go red on is a body finding. Without one, the missing-boundary-clause
|
|
# SUGGESTION fires and the blanket `refute_output --partial "SUGGESTION"`
|
|
# below trips for a reason that has nothing to do with body length — which
|
|
# would look like the invariant breaking while proving nothing about it.
|
|
# AGENTS.md cites this test as the pin for that invariant, so it has to fail
|
|
# for one reason and one reason only.
|
|
{
|
|
echo "---"
|
|
echo "name: my-agent"
|
|
echo "description: A valid agent description. Do not use for anything else."
|
|
echo "---"
|
|
echo ""
|
|
python3 -c "print(' '.join(['word'] * 1500))"
|
|
} > "$root/.apm/agents/my-agent.agent.md"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
# A 1,500-word body is 667% of the skill ceiling. Nothing may be said about
|
|
# it at any tier: not a FAIL, not a SUGGESTION, and not the word-count
|
|
# wording either tier would use if a gate were quietly added later.
|
|
refute_output --partial "SUGGESTION"
|
|
refute_output --partial "1500 words"
|
|
refute_output --partial "900-word"
|
|
refute_output --partial "body is"
|
|
}
|
|
|
|
@test "a bare plugin.json with no apm.yml is no longer plugin scope — falls through to project scope" {
|
|
local root="$TMPDIR/proj-legacy-plugin-json"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
echo '{}' > "$root/plugin.json"
|
|
# 'hooks' is plugin-silently-ignored only at (old) plugin scope; at
|
|
# project scope it's a legitimate CC field. If this directory were
|
|
# mis-detected as plugin scope (old plugin.json-based logic), this would
|
|
# FAIL with a plugin-silently-ignored-fields finding on 'hooks'.
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
hooks:
|
|
PostToolUse:
|
|
- match: ".*"
|
|
command: "echo done"
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "hooks"
|
|
}
|