Why The two audit skills carried 1,724 lines of byte-identical duplication: the ADR-0020 boundary resolver (1,061), vale-wrap.sh (526), the Vale style rules (44) and the Contributing-files parser (93). Nothing shared them — they were held in sync by a 413-line pre-push gate and its 797-line test suite. Sync-by-gate had already failed once: at484357athe two parser copies drifted into different spellings of the bullet loop while a docstring asserted they were identical. That drift was behaviour-neutral and was re-unified by hand at598a7c3, so the copies were identical at merge time — but nothing had caught it, and the next drift need not be neutral. Implementation Notes Self-containment binds BETWEEN skills, not within one. The agentskills.io spec forbids reaching across skill directories, which is why two separate skills needed embedded copies; two files inside ONE skill may source a third. That is the whole reason the merge removes duplication rather than relocating it. The union of both bodies measured 1,532 words against BODY_MAX_WORDS=900, and only 211 of those words were shared, so SKILL.md is a dispatch body. Step 0 resolves the flow from the target path before any validation, and its table mirrors validate.sh's detection exactly: a directory holding SKILL.md or a SKILL.md file (skill); a *.agent.md, or a .md directly under an agents/ directory (agent); anything else stops without running a validator. Steps 1-3 live in references/skill-flow.md and references/agent-flow.md, and gotchas that apply to one flow live in that flow's file, since it is loaded on every invocation anyway. If validate.sh reports on the other artifact type, the body restarts at Step 0. Named factory-audit rather than forge-audit because forge is a live skill, and a family prefix that matches a live sibling reads as ownership rather than membership. The description carries one arrow per boundary target, because ADR-0020 resolves only the first target after an arrow. It drops the quoted "audit this skill"-style phrases, which restated "audited" in a second register (ADR-0020's duplicate-register rule). 241 characters, Gotchas 16% of the body: no size SUGGESTIONs. The boundary resolver stays embedded in two files rather than imported: a cache-installed plugin cannot read outside its own directory, and the repo-root hook resolves via .pre-commit-hooks.yaml where entry[0] is the only token pre-commit rewrites, so no single file is reachable by both. tests/test-adr0020-contract.sh hashes both copies for byte-identity, and asserts validate.sh sources the resolver and that no third copy exists. The entry scripts classify the target from its resolved parent directory, so a bare agent filename typed inside agents/ works; resolve SCRIPT_DIR CDPATH-safely; and exit 2 when a lib-*.sh is missing, rather than dying with exit 1, the tier the flows relay as real findings. The provenance run functions stash their findings code in KYBERFORGE_PROV_RC and return 0, so validate-provenance.sh calls them UNTESTED. Testing a function's status (`f || RC=$?`) disables errexit for its entire body, and no subshell or `set -e` inside can re-arm it once the call sits in a condition context (measured, both spellings). Their error paths use `exit`, which is unaffected either way; this keeps errexit armed for anything added later. Case 0's readability guard reads the file instead of asking `[[ -r ]]`. `-r` is access(2), which answers yes for uid 0 even on a mode-000 file, and this repo's dev environment is root -- so the guard could never fire where it exists to fire. A read attempt is also the stricter question, catching EIO. This is the reasoning scripts/check-vale-style-sync.sh carried before this commit deleted it; the hazard did not go with it. All three entry scripts are CDPATH-safe, vale-wrap.sh included: both of its cd sites are cleared, the --config resolution and the directory-mirror walk, where an exported CDPATH would otherwise print a decoy path into the -print0 stream and build the mirror from the decoy's files. The two remaining bare cd calls take absolute paths, which CDPATH is never consulted for. Impact BREAKING: skill-audit and agent-audit no longer exist as invocable skills. kyberforge goes to 2.0.0 (catalog 0.4.7). Check logic is unchanged: differential runs of the old and new validators across every skill and agent produced byte-identical stdout, stderr and exit codes, and the reconstructed Python payloads differ only in comments and the references/field-inventory.md -> agent-field-inventory.md rename. One doctrine governs the tiers: exit 0 is audited and clean, exit 1 is audited with findings OR a target present but unreadable, exit 2 is that nothing was audited at all. Edge paths DID change, deliberately (full table in ADR-0025): - a missing target exits 2 (never ran), not 1, under its own "does not exist" message; detection is by path shape, so a shape-matching path that is simply absent used to reach the validator and come back as a FAIL against a file that never existed; - an unshaped target exits 2 under the generic "matches neither" message, and a directory with no SKILL.md under a third, distinct one -- three exit-2 messages, not one; - a dangling symlink or a symlink loop stays exit 1: it is present but broken, which is a finding about the artifact rather than a usage error; - a SKILL.md file path is audited as its skill directory instead of refused; - a .md agent outside an agents/ directory is refused rather than audited; - a missing script library, a missing python3, a missing PyYAML, and no argument at all each exit 2. validate-provenance.sh already exited 2 for the last two; validate.sh now matches it. .pre-commit-hooks.yaml is a published contract consumed by external repos. Both hook IDs and both files: regexes are unchanged; only entry: and description: moved. scripts/check-vale-style-sync.sh (413), scripts/sync-vale-styles.sh (21), tests/test-check-vale-style-sync.sh (797) and agent-audit/scripts/README.md (47) are deleted. The checker made 17 assertions: 6 compared the two Vale copies and are moot; 10 are rehomed into tests/test-vale-wrap.sh (case 0, cases 28-31, and the suite's Vale-absent skip); and the cross-manifest files: agreement check, which selected hooks by entry: and so could not survive both hooks sharing one, is ported as case 33 pairing hooks by id:. Cases 28, 30 and 33 carry mutation self-tests; narrowing the local skill prefilter to 6 of 38 SKILL.md files now fails the suite. Skills go 39 to 38. Pre-push goes 9 repo-authored hooks to 8. ADR: 0025 BREAKING-CHANGE: the skill-audit and agent-audit skills are removed. Both flows are served by factory-audit, which auto-detects whether it was handed a skill directory or an agent file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YR2CjVumUbEGWcMikcoXBD
1220 lines
46 KiB
Bash
1220 lines
46 KiB
Bash
#!/usr/bin/env bats
|
|
|
|
setup() {
|
|
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../../" && pwd)"
|
|
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
|
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
|
|
|
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
|
TMPDIR="$(mktemp -d)"
|
|
|
|
# Helper: create an APM package root at <root> (apm.yml with a top-level
|
|
# type: line, marking it a real package manifest — not marketplace-only)
|
|
# plus a single vendor-neutral agent file at
|
|
# <root>/.apm/agents/<name>.agent.md. <extra_frontmatter>, if given, is
|
|
# inserted as additional raw frontmatter lines (used to inject fields
|
|
# under test).
|
|
make_apm_agent() {
|
|
local root="$1"
|
|
local name="$2"
|
|
local extra_frontmatter="${3:-}"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
|
|
---
|
|
name: ${name}
|
|
description: A valid agent description.
|
|
${extra_frontmatter}
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
}
|
|
|
|
# Helper: a description of EXACTLY <n> characters that carries a boundary
|
|
# clause and names no routing target. ADR-0020's missing-boundary-clause
|
|
# SUGGESTION fires on any description without one, so a fixture that omits it
|
|
# is never "otherwise clean" and a test refuting SUGGESTION would be asserting
|
|
# the boundary check's absence instead of the thing it names. The clause is
|
|
# paid for out of the measured budget rather than appended to it, because
|
|
# these tests measure the description LENGTH. "anything else" is not
|
|
# hyphenated, so no routing target comes with it.
|
|
desc_of_length() {
|
|
python3 - "$1" <<'PY'
|
|
import sys
|
|
n = int(sys.argv[1])
|
|
prefix = 'Use when doing the thing. Do not use for anything else. '
|
|
assert n >= len(prefix), 'requested description shorter than the boundary clause'
|
|
print(prefix + 'x' * (n - len(prefix)))
|
|
PY
|
|
}
|
|
|
|
# Helper: same shape as make_apm_agent, but the description is supplied
|
|
# verbatim — used by the ADR-0020 description-budget tests.
|
|
make_apm_agent_with_desc() {
|
|
local root="$1" name="$2" desc="$3"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
|
|
---
|
|
name: ${name}
|
|
description: ${desc}
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
}
|
|
}
|
|
|
|
teardown() {
|
|
rm -rf "$TMPDIR"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Passing cases — project/user scope (unchanged)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "passes on a clean project-scope pair (CC file as input)" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "user scope: \$HOME being a dotfiles .git repo does not shadow user scope" {
|
|
local fake_home="$TMPDIR/fakehome"
|
|
mkdir -p "$fake_home/.git" "$fake_home/.claude/agents" "$fake_home/.copilot/agents"
|
|
cat > "$fake_home/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$fake_home/.copilot/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "user scope: agent file directly in \$HOME (start dir IS exactly \$HOME, no walk-up) resolves to user scope" {
|
|
# The fake $HOME is named `agents` on purpose, and it is a fixture detail
|
|
# forced by the merged entry point rather than anything the scope resolver
|
|
# cares about. The case under test needs the agent file's own directory to BE
|
|
# $HOME (detect_scope's `original_start == home` branch), AND the input has
|
|
# to be the Claude Code half — a plain .md — because that is the only half
|
|
# whose counterpart path differs between user scope ($HOME/.copilot/agents)
|
|
# and project scope ($HOME/.github/agents). With a *.agent.md input the two
|
|
# scopes derive the identical counterpart and the test would pass without
|
|
# discriminating. Post-merge, a plain .md only classifies as an agent when
|
|
# its parent directory is named `agents`, so the two requirements can only be
|
|
# satisfied together by a $HOME whose basename is `agents`.
|
|
#
|
|
# detect_scope's conventional_shape stays FALSE here (the grandparent is not
|
|
# one of .claude/.github/.copilot/.apm), so this still exercises the
|
|
# no-walk-up branch and not the two-segments-up arithmetic.
|
|
local fake_home="$TMPDIR/fakehome-direct/agents"
|
|
mkdir -p "$fake_home" "$fake_home/.copilot/agents"
|
|
cat > "$fake_home/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$fake_home/.copilot/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: a nested marker-less directory walked up into \$HOME resolves to project scope, not user scope (live repro of new-agent.sh's stray-directory case)" {
|
|
local fake_home="$TMPDIR/fakehome-nested"
|
|
local nested="$fake_home/scratch/testdir"
|
|
mkdir -p "$nested/.claude/agents" "$nested/.github/agents"
|
|
cat > "$nested/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$nested/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$nested/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: nested marker-less dir under \$HOME does NOT look for a counterpart under the shared \$HOME/.copilot or \$HOME/.github dirs" {
|
|
local fake_home="$TMPDIR/fakehome-nested2"
|
|
local nested="$fake_home/scratch/testdir"
|
|
mkdir -p "$nested/.claude/agents" "$fake_home/.copilot/agents"
|
|
cat > "$nested/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
# Decoy counterpart at the *user*-scope location — if scope were
|
|
# misclassified as 'user' (the pre-fix bug), validate.sh would find this
|
|
# unrelated file and (wrongly) pass.
|
|
cat > "$fake_home/.copilot/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$fake_home" bash "$SCRIPT" "$nested/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "counterpart file not found"
|
|
}
|
|
|
|
@test "project scope: filesystem-root fallback (no \$HOME in path, no markers found) resolves to project scope, not user scope" {
|
|
local unrelated_home="$TMPDIR/unrelated-home-never-reached"
|
|
local root="$TMPDIR/no-home-relation/deep/proj"
|
|
mkdir -p "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run env HOME="$unrelated_home" bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: <root> one level below a .git ancestor resolves scope to <root>, not to wherever .git was found (subdirectory of a larger git-tracked tree)" {
|
|
local repo="$TMPDIR/repo-with-subdir"
|
|
local root="$repo/subdir"
|
|
mkdir -p "$repo/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
# new-agent.sh, invoked with <root> as its root argument, would place the
|
|
# counterpart at <root>/.github/agents — not at the repo root's
|
|
# .github/agents, even though .git lives at the repo root one level up.
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "project scope: a non-conventional path (agent file not directly under a literal 'agents' dir) falls back to the nearest .git boundary instead of two-segments-up arithmetic" {
|
|
local outer="$TMPDIR/outer-repo"
|
|
local pkg="$outer/pkgA"
|
|
mkdir -p "$pkg/.git" "$pkg/.claude/agents" "$pkg/extra"
|
|
# Misplaced file: sits two path segments below $outer (pkgA/extra), which
|
|
# matches the conventional_root arithmetic by coincidence, but its
|
|
# immediate parent dir is "extra", not "agents" — conventional_shape is
|
|
# false, so the fix must fall back to the nearest .git boundary (pkgA),
|
|
# not trust $outer.
|
|
#
|
|
# The *.agent.md suffix is a post-merge fixture requirement, not part of what
|
|
# is being measured: the merged entry point classifies a plain .md as an
|
|
# agent only when its parent directory is named `agents`, and the whole point
|
|
# of this fixture is that the parent is `extra`. The suffix is what keeps the
|
|
# file classifiable while leaving conventional_shape false. It flips which
|
|
# half of the pair is the input, so the counterpart below is the Claude Code
|
|
# half rather than the Copilot one; the scope arithmetic under test is
|
|
# untouched by that.
|
|
cat > "$pkg/extra/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
# Counterpart at the nearest-.git root (pkgA), not at $outer — if the
|
|
# arithmetic were trusted here, validate.sh would look for a counterpart
|
|
# at $outer/.claude/agents/my-agent.md, which doesn't exist, and
|
|
# false-FAIL.
|
|
cat > "$pkg/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$pkg/extra/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "counterpart file not found"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Failing cases — project/user scope: CC/Copilot pair checks
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "fails when a CC-only field ('maxTurns') is present in a project-scope Copilot file" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
maxTurns: 10
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "CC-only field"
|
|
assert_output --partial "maxTurns"
|
|
}
|
|
|
|
@test "fails when a Copilot-only field ('target') is present in a project-scope CC file" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
target: cli
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "Copilot-only field"
|
|
assert_output --partial "target"
|
|
}
|
|
|
|
@test "fails when the Copilot counterpart is missing at project scope" {
|
|
local root="$TMPDIR/project"
|
|
mkdir -p "$root/.git" "$root/.claude/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "counterpart file not found"
|
|
}
|
|
|
|
@test "--help exits 0 and shows Usage:" {
|
|
run bash "$SCRIPT" --help
|
|
assert_success
|
|
assert_output --partial "Usage:"
|
|
}
|
|
|
|
@test "fails when no arguments are given" {
|
|
run bash "$SCRIPT"
|
|
assert_failure
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Passing cases — plugin/APM scope
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "passes on a clean plugin/APM-scope agent file (name/description only)" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "passes on a clean plugin/APM-scope agent file with optional model field" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "model: claude-opus-4"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Failing cases — plugin/APM scope: allowlist violations
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "fails when 'tools' field is present in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "tools: Read Edit"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "tools"
|
|
}
|
|
|
|
@test "fails when a Claude-only field ('maxTurns') is present in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "maxTurns: 10"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "maxTurns"
|
|
}
|
|
|
|
@test "fails when a Copilot-only field ('target') is present in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "target: cli"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "target"
|
|
}
|
|
|
|
@test "fails when 'hooks' is present in a plugin/APM-scope agent file (outside allowlist)" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "hooks: {}"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "hooks"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Failing cases — plugin/APM scope: structural checks
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "fails when name is not kebab-case in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: MyAgent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "kebab"
|
|
}
|
|
|
|
@test "fails when name does not match filename stem in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: wrong-name
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "does not match filename stem"
|
|
}
|
|
|
|
@test "fails when 'name' field is missing from a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when 'description' field is missing from a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
}
|
|
|
|
@test "fails when a template HTML comment is left in plugin/APM-scope frontmatter" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
<!-- model: sonnet
|
|
Optional. Omit to inherit the runtime default. -->
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "template HTML comments"
|
|
}
|
|
|
|
@test "fails when body contains unfilled FILL IN: placeholder in a plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
FILL IN: replace this with your system prompt.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Plugin/APM scope: no pair, no counterpart concept
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "never raises a 'counterpart' FAIL on a clean plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
@test "never raises a 'counterpart' FAIL on a failing plugin/APM-scope agent file" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent "$root" "my-agent" "tools: Read"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
refute_output --partial "counterpart"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Scope-detection walk-up
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "walk-up skips a type:-less apm.yml and finds a real package root further up" {
|
|
local root="$TMPDIR/case"
|
|
mkdir -p "$root/.apm/agents/nested/deeper"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: real-package
|
|
version: 1.0.0
|
|
type: skill
|
|
EOF
|
|
# Closer to the agent file than the real package root, but has no type:
|
|
# line — marketplace-only per monorepo-and-repo-shapes.md, must be skipped.
|
|
cat > "$root/.apm/agents/nested/apm.yml" <<EOF
|
|
name: not-a-package-manifest
|
|
version: 1.0.0
|
|
EOF
|
|
cat > "$root/.apm/agents/nested/deeper/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/nested/deeper/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
@test "type:-less apm.yml is not treated as plugin scope — falls through to project scope" {
|
|
local root="$TMPDIR/proj-marketplace"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: marketplace-root
|
|
version: 1.0.0
|
|
marketplace:
|
|
packages: []
|
|
EOF
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ADR-0020 — description budget (250 SUGGESTION / 400 FAIL)
|
|
#
|
|
# Agents take the SAME description gates as skills: name + description is
|
|
# preloaded into every session identically. Agents take NO body word gate — see
|
|
# the final test in this block, which pins that asymmetry.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "ADR-0020: agent description of exactly 250 chars raises no suggestion" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 250)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "SUGGESTION"
|
|
}
|
|
|
|
@test "ADR-0020: agent description of 251 chars raises a SUGGESTION and still exits 0" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 251)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
assert_output --partial "SUGGESTION"
|
|
assert_output --partial "description is 251 chars"
|
|
}
|
|
|
|
@test "ADR-0020: agent description of exactly 400 chars is a SUGGESTION, not a FAIL" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 400)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
assert_output --partial "SUGGESTION"
|
|
}
|
|
|
|
@test "ADR-0020: agent description of 401 chars FAILs and exits non-zero" {
|
|
local root="$TMPDIR/pkg"
|
|
make_apm_agent_with_desc "$root" "my-agent" "$(desc_of_length 401)"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "description is 401 chars"
|
|
assert_output --partial "400-character ADR-0020 ceiling"
|
|
}
|
|
|
|
@test "ADR-0020: agent description length is measured after YAML folding is resolved" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
# 11 folded lines of 40 chars + 10 joining spaces = 450 characters. Read off
|
|
# the raw `description: >` line it is 1 character and passes.
|
|
{
|
|
echo "---"
|
|
echo "name: my-agent"
|
|
echo "description: >"
|
|
python3 -c "print('\n'.join([' ' + 'x' * 40] * 11))"
|
|
echo "---"
|
|
echo ""
|
|
echo "You are a test agent."
|
|
} > "$root/.apm/agents/my-agent.agent.md"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_failure
|
|
assert_output --partial "description is 450 chars"
|
|
}
|
|
|
|
@test "ADR-0020: the description gate applies at project scope too" {
|
|
local root="$TMPDIR/project"
|
|
local desc
|
|
desc="$(python3 -c "print('x' * 401)")"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: $desc
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_failure
|
|
assert_output --partial "400-character ADR-0020 ceiling"
|
|
}
|
|
|
|
@test "ADR-0020: agents take NO body word gate — a body far over the 900-word skill ceiling passes" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
# Deliberate asymmetry, not an oversight: a skill body is loaded into the
|
|
# caller's context and competes with the live conversation, while an agent
|
|
# body becomes the system prompt of a fresh context. ADR-0020 gates the
|
|
# former at 900 words and explicitly declines to gate the latter. If a body
|
|
# word gate is ever added here, it contradicts the ADR.
|
|
#
|
|
# The description carries a boundary clause so the ONLY thing this test can
|
|
# go red on is a body finding. Without one, the missing-boundary-clause
|
|
# SUGGESTION fires and the blanket `refute_output --partial "SUGGESTION"`
|
|
# below trips for a reason that has nothing to do with body length — which
|
|
# would look like the invariant breaking while proving nothing about it.
|
|
# AGENTS.md cites this test as the pin for that invariant, so it has to fail
|
|
# for one reason and one reason only.
|
|
{
|
|
echo "---"
|
|
echo "name: my-agent"
|
|
echo "description: A valid agent description. Do not use for anything else."
|
|
echo "---"
|
|
echo ""
|
|
python3 -c "print(' '.join(['word'] * 1500))"
|
|
} > "$root/.apm/agents/my-agent.agent.md"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "FAIL"
|
|
# A 1,500-word body is 667% of the skill ceiling. Nothing may be said about
|
|
# it at any tier: not a FAIL, not a SUGGESTION, and not the word-count
|
|
# wording either tier would use if a gate were quietly added later.
|
|
refute_output --partial "SUGGESTION"
|
|
refute_output --partial "1500 words"
|
|
refute_output --partial "900-word"
|
|
refute_output --partial "body is"
|
|
}
|
|
|
|
@test "a bare plugin.json with no apm.yml is no longer plugin scope — falls through to project scope" {
|
|
local root="$TMPDIR/proj-legacy-plugin-json"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
echo '{}' > "$root/plugin.json"
|
|
# 'hooks' is plugin-silently-ignored only at (old) plugin scope; at
|
|
# project scope it's a legitimate CC field. If this directory were
|
|
# mis-detected as plugin scope (old plugin.json-based logic), this would
|
|
# FAIL with a plugin-silently-ignored-fields finding on 'hooks'.
|
|
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
hooks:
|
|
PostToolUse:
|
|
- match: ".*"
|
|
command: "echo done"
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "hooks"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# tools: — both YAML spellings
|
|
# ---------------------------------------------------------------------------
|
|
# The subagent-unavailable-tool SUGGESTION is read off the `tools` field, and
|
|
# `tools` has two legal spellings: an inline scalar and a block sequence. The
|
|
# field used to be pulled out with a line regex whose capture is newline-bounded
|
|
# on purpose, so a block sequence captured NOTHING and the check silently
|
|
# stopped firing — on the shape Copilot agent files actually use, which is to say
|
|
# on the files it was written for. Both spellings are pinned, and they are pinned
|
|
# together: the inline case alone was green throughout.
|
|
|
|
# make_pair <root> <tools-frontmatter> — a project-scope CC + Copilot pair
|
|
# carrying the same `tools` value in both files. `tools` is on neither the
|
|
# claude-code-only nor the copilot-only list, so it is legal in both and the pair
|
|
# stays otherwise clean; the description carries a boundary clause so the only
|
|
# SUGGESTION that can fire is the one under test.
|
|
make_tools_pair() {
|
|
local root="$1" tools="$2"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
local f
|
|
for f in "$root/.claude/agents/my-agent.md" "$root/.github/agents/my-agent.agent.md"; do
|
|
{
|
|
echo "---"
|
|
echo "name: my-agent"
|
|
echo "description: A valid agent description. Do not use for anything else."
|
|
echo "$tools"
|
|
echo "---"
|
|
echo ""
|
|
echo "You are a test agent. When invoked, do the thing."
|
|
} > "$f"
|
|
done
|
|
}
|
|
|
|
@test "a subagent-unavailable tool in an INLINE tools scalar raises a SUGGESTION" {
|
|
make_tools_pair "$TMPDIR/inline" "tools: Read ExitPlanMode"
|
|
run bash "$SCRIPT" "$TMPDIR/inline/.claude/agents/my-agent.md"
|
|
assert_success
|
|
assert_output --partial "'ExitPlanMode' is listed in tools but is never available to subagents"
|
|
}
|
|
|
|
@test "a subagent-unavailable tool in a BLOCK SEQUENCE tools field raises the same SUGGESTION" {
|
|
make_tools_pair "$TMPDIR/block" "$(printf 'tools:\n - Read\n - ExitPlanMode')"
|
|
run bash "$SCRIPT" "$TMPDIR/block/.claude/agents/my-agent.md"
|
|
assert_success
|
|
assert_output --partial "'ExitPlanMode' is listed in tools but is never available to subagents"
|
|
}
|
|
|
|
@test "a tools list with no subagent-unavailable tool stays silent in both spellings" {
|
|
# The control. Without it both cases above are satisfied by a check that
|
|
# fires on every tools field it can see, which would be the opposite defect.
|
|
make_tools_pair "$TMPDIR/inline-clean" "tools: Read Edit"
|
|
run bash "$SCRIPT" "$TMPDIR/inline-clean/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "never available to subagents"
|
|
|
|
make_tools_pair "$TMPDIR/block-clean" "$(printf 'tools:\n - Read\n - Edit')"
|
|
run bash "$SCRIPT" "$TMPDIR/block-clean/.claude/agents/my-agent.md"
|
|
assert_success
|
|
refute_output --partial "never available to subagents"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# A file that is not there, and a file that cannot be read
|
|
# ---------------------------------------------------------------------------
|
|
# These are two different tiers and the split is the point.
|
|
#
|
|
# NOT THERE is exit 2: nothing is at the path, so no check ran. The entry
|
|
# point's existence guard catches it before python3 is invoked, because the
|
|
# *.agent.md and agents/-parent branches classify on NAME alone and a typo'd
|
|
# path used to sail through them into python and come back as a FAIL at exit 1
|
|
# — the FINDINGS tier, for a target that was never there to have findings
|
|
# about. (Before that it was worse still: a bare FileNotFoundError traceback
|
|
# and no FAIL line at all.) Both scope paths are covered because they were
|
|
# separate call sites, check_apm_agent_file and check_file, and each needed its
|
|
# own handler; the guard now short-circuits both, and the traceback assertions
|
|
# below are kept to pin that the diagnostic is still a sentence and not a stack.
|
|
#
|
|
# CANNOT BE READ is exit 1: something IS at the path and opening it fails, which
|
|
# is a real finding, and it must still reach check_file's `except OSError` arm.
|
|
# That is why the guard tests `! -e && ! -L` rather than `! -e` alone — a
|
|
# dangling symlink and a symlink loop are both false to -e and true to -L. The
|
|
# unreadable/nonexistent pairs below sit together so that distinction cannot be
|
|
# collapsed by accident.
|
|
#
|
|
# The unreadable fixture is a SYMLINK LOOP rather than a chmod 000 file: these
|
|
# tests run as root in CI, where mode bits do not deny anything and a permissions
|
|
# fixture would be silently readable and prove nothing.
|
|
#
|
|
# It used to be a DIRECTORY named `is-a-dir.agent.md`, which served the same
|
|
# purpose pre-merge. That no longer reaches the handler: the merged entry point
|
|
# classifies on shape first, and a directory is a skill-shaped target whatever it
|
|
# is named, so it is rejected at exit 2 before python3 is invoked. A symlink loop
|
|
# is a FILE by the detector's reckoning, classifies as an agent on its name, and
|
|
# still raises OSError on open — so it exercises the same `except OSError` arm.
|
|
# The directory-named-like-an-agent case did not go away with the fixture; it is
|
|
# pinned in its own test immediately below.
|
|
|
|
@test "a nonexistent plugin/APM-scope agent file exits 2 naming the path, never the findings tier" {
|
|
local root="$TMPDIR/pkg"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/absent.agent.md"
|
|
# The literal 2, not assert_failure: 1 would pass a bare assert_failure, and
|
|
# 1 is exactly the wrong answer here — it is the tier that says the file was
|
|
# audited and has findings.
|
|
[ "$status" -eq 2 ]
|
|
assert_output --partial "Error: '$root/.apm/agents/absent.agent.md' does not exist."
|
|
# Kept from when this path reached python3 and returned a stack. Trivially
|
|
# true now that the guard short-circuits before the interpreter, and kept
|
|
# anyway: the assertion is "a reader gets a sentence, not a traceback", and
|
|
# that intent outlives the mechanism that currently satisfies it.
|
|
refute_output --partial "Traceback"
|
|
refute_output --partial "FileNotFoundError"
|
|
}
|
|
|
|
@test "an unreadable plugin/APM-scope agent file gets a FAIL naming the path, not a traceback" {
|
|
local root="$TMPDIR/pkg-loop"
|
|
mkdir -p "$root/.apm/agents"
|
|
ln -s "unreadable.agent.md" "$root/.apm/agents/unreadable.agent.md"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/unreadable.agent.md"
|
|
assert_failure
|
|
assert_output --partial "FAIL"
|
|
assert_output --partial "could not be read"
|
|
assert_output --partial "unreadable.agent.md"
|
|
refute_output --partial "Traceback"
|
|
refute_output --partial "OSError"
|
|
}
|
|
|
|
@test "a DIRECTORY named like an agent file is rejected by shape, not audited as one" {
|
|
# The other half of the case the fixture above used to cover, kept as its own
|
|
# test so nothing is lost in the swap. A directory cannot be an agent target
|
|
# under ADR-0025's rule at all, so it never reaches check_apm_agent_file's
|
|
# OSError arm — it is refused at classification time. Exit 2 (not auditable)
|
|
# rather than 1 (audited and failing), and still a message naming the path:
|
|
# the failure mode this guards against is a silent exit 0 or a bare non-zero
|
|
# with an interpreter stack for a diagnostic.
|
|
local root="$TMPDIR/pkg-dir"
|
|
mkdir -p "$root/.apm/agents/is-a-dir.agent.md"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/is-a-dir.agent.md"
|
|
[ "$status" -eq 2 ]
|
|
assert_output --partial "Error: '$root/.apm/agents/is-a-dir.agent.md' is a directory with no SKILL.md in it."
|
|
refute_output --partial "Traceback"
|
|
refute_output --partial "IsADirectoryError"
|
|
}
|
|
|
|
@test "a nonexistent project-scope agent file exits 2 naming the path, never the findings tier" {
|
|
# The counterpart exists and the named file does not. Before the guard this
|
|
# exercised the OTHER call site — check_file, reached after the counterpart
|
|
# pre-check — and the fact that the counterpart is present is what made it a
|
|
# distinct case. The guard is upstream of scope detection entirely, so both
|
|
# scopes now stop at the same place; the fixture is kept so a guard that ever
|
|
# regresses to a scope-specific check is caught on both paths.
|
|
local root="$TMPDIR/proj-missing"
|
|
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
|
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description. Do not use for anything else.
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
|
# The literal 2, not assert_failure — see the plugin-scope twin above.
|
|
[ "$status" -eq 2 ]
|
|
assert_output --partial "Error: '$root/.claude/agents/my-agent.md' does not exist."
|
|
# Trivially true now that the guard short-circuits before python3; kept
|
|
# because the intent — a sentence, not an interpreter stack — is unchanged.
|
|
refute_output --partial "Traceback"
|
|
refute_output --partial "FileNotFoundError"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Encoding, write side: sys.stdout/stderr.reconfigure(encoding='utf-8')
|
|
#
|
|
# read_text() in the shared resolver block pins the READS to UTF-8. That moved
|
|
# the LC_ALL=C crash to the WRITE: this script's own message text carries em
|
|
# dashes (the ADR-0020 boundary SUGGESTION is one), so the streams' ASCII
|
|
# default raised UnicodeEncodeError while PRINTING — after every check had
|
|
# already run. Here it also flipped a clean exit 0 into a traceback and exit 1.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "under LC_ALL=C the report is printed, not lost to a UnicodeEncodeError" {
|
|
local root="$TMPDIR/locale-pkg"
|
|
make_apm_agent "$root" "locale-agent"
|
|
run env LC_ALL=C PYTHONUTF8=0 bash "$SCRIPT" "$root/.apm/agents/locale-agent.agent.md"
|
|
assert_success
|
|
assert_output --partial "description has no boundary clause"
|
|
refute_output --partial "UnicodeEncodeError"
|
|
refute_output --partial "Traceback"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Auto-detection — the merged entry point classifies its own target
|
|
#
|
|
# NEW with the factory-audit merge, and new behaviour rather than a ported
|
|
# case: scripts/validate.sh is now ONE entry point for both artifact types and
|
|
# works out from the target which rubric to run. This file is the AGENT half of
|
|
# that contract; validate-skill.bats holds the skill half and the
|
|
# neither-shape rejection. Same script in all four, different fixtures.
|
|
#
|
|
# ADR-0025 states the rule: "A directory containing SKILL.md takes the skill
|
|
# flow; an .agent.md file or a file under agents/ takes the agent flow." That is
|
|
# TWO independent rules on the agent side, and each is pinned on its own below,
|
|
# because either one alone would make the other look like it worked:
|
|
#
|
|
# 1. the filename ends in .agent.md, wherever it sits
|
|
# 2. the file sits under a directory named agents/, whatever it is called
|
|
# (.apm/agents/, .claude/agents/, .github/agents/ — the same shape the
|
|
# exported Vale hook matches with `(^|/)agents/[^/]+\.md$`)
|
|
#
|
|
# A misclassification here is silent and total — the skill rubric would run
|
|
# against an agent file, so ADR-0022's metadata.version and ADR-0020's 900-word
|
|
# body ceiling would both be applied to an artifact that ADR-0020 explicitly
|
|
# exempts from the latter (see the 1,500-word body test above, which AGENTS.md
|
|
# cites as the pin for that asymmetry).
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "auto-detect: a *.agent.md file is audited in AGENT mode wherever it sits" {
|
|
# Rule 1 in isolation. The directory is deliberately named `defs/`, not
|
|
# `agents/`, so rule 2 cannot reach this fixture and the extension is the
|
|
# only thing that can classify it. Put it under agents/ and both rules match,
|
|
# and a detector implementing only rule 2 would still pass this test.
|
|
local root="$TMPDIR/pkg-by-extension"
|
|
mkdir -p "$root/defs"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/defs/my-agent.agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description. Do not use for anything else.
|
|
target: cli
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/defs/my-agent.agent.md"
|
|
# 'target' is a Copilot-only field, and the vendor-field allowlist that
|
|
# rejects it exists only in the agent rubric. The finding is therefore proof
|
|
# of WHICH rubric ran, not just that something went wrong. Exit 1 exactly:
|
|
# a bare assert_failure also accepts exit 2, which is a refusal to audit, and
|
|
# "target" is too common a word to tell a findings report from a refusal.
|
|
assert_failure 1
|
|
assert_output --partial "target"
|
|
}
|
|
|
|
@test "auto-detect: a file under .apm/agents/ is audited in AGENT mode even without the .agent.md suffix" {
|
|
# Rule 2 in isolation. `.apm/agents/` is the vendor-neutral authoring
|
|
# directory, so anything sitting in it is an agent definition whatever it is
|
|
# called — and the plain `.md` suffix here is deliberate, so that rule 1
|
|
# cannot rescue the fixture and rule 2 is the only thing being measured.
|
|
local root="$TMPDIR/pkg-by-location"
|
|
mkdir -p "$root/.apm/agents"
|
|
cat > "$root/apm.yml" <<EOF
|
|
name: test-package
|
|
version: 0.1.0
|
|
type: skill
|
|
EOF
|
|
cat > "$root/.apm/agents/my-agent.md" <<EOF
|
|
---
|
|
name: my-agent
|
|
description: A valid agent description. Do not use for anything else.
|
|
tools: Read Edit
|
|
---
|
|
|
|
You are a test agent. When invoked, do the thing.
|
|
EOF
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.md"
|
|
# 'tools' sits outside the plugin/APM allowlist — again agent-rubric-only
|
|
# vocabulary, so the finding names the rubric that produced it. Exit 1
|
|
# exactly, for the same reason as the case above: exit 2 is a refusal, not a
|
|
# findings report, and "tools" alone cannot tell the two apart.
|
|
assert_failure 1
|
|
assert_output --partial "tools"
|
|
}
|
|
|
|
@test "auto-detect: an agent file never picks up a skill-only gate" {
|
|
# The control for both cases above. Without it, either is satisfied by a
|
|
# detector that runs BOTH rubrics and unions the findings — which would
|
|
# report every agent file in the corpus as missing a metadata.version it is
|
|
# not required to have.
|
|
local root="$TMPDIR/pkg-clean-detect"
|
|
make_apm_agent "$root" "my-agent"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
|
assert_success
|
|
refute_output --partial "metadata.version"
|
|
refute_output --partial "900-word"
|
|
refute_output --partial "SKILL.md"
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Entry-point robustness: how the target and the libraries are located
|
|
# ---------------------------------------------------------------------------
|
|
# Three defects in the merged entry point's own plumbing, none of which a valid
|
|
# absolute-path target exercises:
|
|
#
|
|
# - The agents/-parent rule read the parent's NAME off the argument text, so a
|
|
# bare `my-agent.md` typed from inside the agents/ directory had a parent of
|
|
# "." and was refused as matching neither shape — by a message naming that
|
|
# exact shape as valid.
|
|
# - A missing lib-*.sh killed the script under `set -e` with exit 1, the tier
|
|
# the flow references relay verbatim as real findings.
|
|
# - SCRIPT_DIR was resolved with a bare `cd`, which under an exported CDPATH
|
|
# prints the directory it chose and can choose a DIFFERENT one — so a
|
|
# relative invocation could source same-named libraries from elsewhere.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
@test "entry point: a bare filename typed from inside an agents/ directory is audited, not refused" {
|
|
# The plain .md suffix is deliberate: *.agent.md is accepted on its name
|
|
# alone, so only the agents/-parent rule is measured here.
|
|
local root="$TMPDIR/pkg-bare"
|
|
make_apm_agent "$root" "my-agent"
|
|
mv "$root/.apm/agents/my-agent.agent.md" "$root/.apm/agents/my-agent.md"
|
|
run bash "$SCRIPT" "$root/.apm/agents/my-agent.md"
|
|
local abs_status="$status"
|
|
# The literal 0, not `-ne 2`. The three invocations below are only compared
|
|
# against this one, so an anchor of "anything but 2" left the whole test
|
|
# green under a regression that made all three exit 1 — every verdict here
|
|
# was self-referential. make_apm_agent builds a CLEAN agent, so the
|
|
# absolute-path baseline is 0 and nothing weaker belongs here. Its provenance
|
|
# twin in validate-provenance-agent.bats pins the same literal.
|
|
[ "$abs_status" -eq 0 ]
|
|
|
|
cd "$root/.apm/agents"
|
|
run bash "$SCRIPT" my-agent.md
|
|
refute_output --partial "matches neither"
|
|
[ "$status" -eq "$abs_status" ]
|
|
|
|
run bash "$SCRIPT" ./my-agent.md
|
|
refute_output --partial "matches neither"
|
|
[ "$status" -eq "$abs_status" ]
|
|
}
|
|
|
|
@test "entry point: a missing mode library exits 2 naming it, never exit 1" {
|
|
# Exit 1 is the findings tier. A validate.sh separated from its libraries is
|
|
# an install problem, and has to land in the "never ran" tier instead.
|
|
local root="$TMPDIR/pkg-lone"
|
|
make_apm_agent "$root" "my-agent"
|
|
local lone="$TMPDIR/lone-scripts"
|
|
cp -R "$(dirname "$SCRIPT")" "$lone"
|
|
rm "$lone/lib-checks-agent.sh"
|
|
run bash "$lone/validate.sh" "$root/.apm/agents/my-agent.agent.md"
|
|
[ "$status" -eq 2 ]
|
|
assert_output --partial "required library '$lone/lib-checks-agent.sh' is missing or unreadable"
|
|
refute_output --partial "No such file or directory"
|
|
}
|
|
|
|
@test "entry point: an exported CDPATH cannot redirect which libraries are sourced" {
|
|
# A decoy scripts/ directory reachable through CDPATH, holding libraries that
|
|
# announce themselves and abort. The relative invocation `bash
|
|
# scripts/validate.sh` is the one the flow references prescribe, and it is the
|
|
# one CDPATH applies to.
|
|
local root="$TMPDIR/pkg-cdpath"
|
|
make_apm_agent "$root" "my-agent"
|
|
local decoy="$TMPDIR/decoy"
|
|
mkdir -p "$decoy/scripts"
|
|
local lib
|
|
for lib in lib-boundary-resolver.sh lib-checks-skill.sh lib-checks-agent.sh; do
|
|
printf 'echo DECOY-SOURCED; exit 99\n' > "$decoy/scripts/$lib"
|
|
done
|
|
cd "$(dirname "$SCRIPT")/.."
|
|
run env CDPATH="$decoy" bash scripts/validate.sh "$root/.apm/agents/my-agent.agent.md"
|
|
refute_output --partial "DECOY-SOURCED"
|
|
assert_success
|
|
}
|