Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity
Files
6f6b70781d0142360c0bd17bcc4986ccabfcb1ea
holocron/plugins/kyberforge/skills/agent-audit/tests/validate.bats
Defame1297 6f6b70781d fix(kyberforge): fix scope walk-up and manifest-parsing bugs from PR #93 review 
A fresh /code-review of the APM-native authoring retarget (PR #93) found
several correctness bugs beyond the ones already fixed on this branch:

- new-agent.sh silently walked a marker-less subdirectory under $HOME up
  to user scope, contradicting its own usage text ("user scope is checked
  directly, no walk-up") and risking scaffolding into shared global
  ~/.claude or ~/.copilot directories instead of the intended local path.
- The hand-copied apm.yml type: manifest detector in new-agent.sh and
  new-skill.sh accepted mismatched quotes (e.g. `type: "skill'`) that
  validate.sh's regex correctly rejects, and silently dropped a final
  apm.yml line lacking a trailing newline — causing the scaffolder and
  validator to disagree on scope for identical input.
- Plugin-scope agent frontmatter could still contain the apm-agent.md
  template's HTML comments at ship time with no audit signal, yet
  apm compile copies frontmatter verbatim and <!-- --> breaks YAML
  parsing on both downstream harnesses.
- ADR-0016 asserted agent-audit already implements a SUGGESTION heuristic
  for tool-restriction-needing plugin-scope agents; it doesn't.
- agent-audit/README.md still described the old plugin-pair model this
  PR replaced with a single-file allowlist model.
- validate.sh's project/user-scope CC-only/Copilot-only field checks and
  counterpart-missing check lost their only test coverage when the old
  plugin-pair fixture was deleted.

Also replaces an echo-into-sed two-value parse (4 forks per call) with a
single space-separated echo + read in both scaffolders.

Regression tests added for every fix above, including one for a bug this
pass introduced and the test suite caught: an initial two-line
echo + `read` attempt silently dropped the second value, since `read`
consumes only one line regardless of embedded newlines.

Full suite: 158 bats tests, 39 shell-script tests, 12/12 summary
categories, 0 failures.

Refs: #89, #93
2026-08-11 21:49:38 +00:00

474 lines
14 KiB
Bash
Raw Blame History

#!/usr/bin/env bats
setup() {
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)"
load "$REPO_ROOT/tests/test_helper/bats-support/load"
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
TMPDIR="$(mktemp -d)"
# Helper: create an APM package root at <root> (apm.yml with a top-level
# type: line, marking it a real package manifest — not marketplace-only)
# plus a single vendor-neutral agent file at
# <root>/.apm/agents/<name>.agent.md. <extra_frontmatter>, if given, is
# inserted as additional raw frontmatter lines (used to inject fields
# under test).
make_apm_agent() {
local root="$1"
local name="$2"
local extra_frontmatter="${3:-}"
mkdir -p "$root/.apm/agents"
cat > "$root/apm.yml" <<EOF
name: test-package
version: 0.1.0
type: skill
EOF
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
---
name: ${name}
description: A valid agent description.
${extra_frontmatter}
---
You are a test agent. When invoked, do the thing.
EOF
}
}
teardown() {
rm -rf "$TMPDIR"
}
# ---------------------------------------------------------------------------
# Passing cases — project/user scope (unchanged)
# ---------------------------------------------------------------------------
@test "passes on a clean project-scope pair (CC file as input)" {
local root="$TMPDIR/project"
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
cat > "$root/.claude/agents/my-agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
assert_success
refute_output --partial "FAIL"
}
@test "user scope: \$HOME being a dotfiles .git repo does not shadow user scope" {
local fake_home="$TMPDIR/fakehome"
mkdir -p "$fake_home/.git" "$fake_home/.claude/agents" "$fake_home/.copilot/agents"
cat > "$fake_home/.claude/agents/my-agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
cat > "$fake_home/.copilot/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/.claude/agents/my-agent.md"
assert_success
refute_output --partial "FAIL"
}
# ---------------------------------------------------------------------------
# Failing cases — project/user scope: CC/Copilot pair checks
# ---------------------------------------------------------------------------
@test "fails when a CC-only field ('maxTurns') is present in a project-scope Copilot file" {
local root="$TMPDIR/project"
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
cat > "$root/.claude/agents/my-agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
maxTurns: 10
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
assert_failure
assert_output --partial "CC-only field"
assert_output --partial "maxTurns"
}
@test "fails when a Copilot-only field ('target') is present in a project-scope CC file" {
local root="$TMPDIR/project"
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
cat > "$root/.claude/agents/my-agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
target: cli
---
You are a test agent. When invoked, do the thing.
EOF
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
assert_failure
assert_output --partial "Copilot-only field"
assert_output --partial "target"
}
@test "fails when the Copilot counterpart is missing at project scope" {
local root="$TMPDIR/project"
mkdir -p "$root/.git" "$root/.claude/agents"
cat > "$root/.claude/agents/my-agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
assert_failure
assert_output --partial "counterpart file not found"
}
@test "--help exits 0 and shows Usage:" {
run bash "$SCRIPT" --help
assert_success
assert_output --partial "Usage:"
}
@test "fails when no arguments are given" {
run bash "$SCRIPT"
assert_failure
}
# ---------------------------------------------------------------------------
# Passing cases — plugin/APM scope
# ---------------------------------------------------------------------------
@test "passes on a clean plugin/APM-scope agent file (name/description only)" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_success
refute_output --partial "FAIL"
}
@test "passes on a clean plugin/APM-scope agent file with optional model field" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent" "model: claude-opus-4"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_success
refute_output --partial "FAIL"
}
# ---------------------------------------------------------------------------
# Failing cases — plugin/APM scope: allowlist violations
# ---------------------------------------------------------------------------
@test "fails when 'tools' field is present in a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent" "tools: Read Edit"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "tools"
}
@test "fails when a Claude-only field ('maxTurns') is present in a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent" "maxTurns: 10"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "maxTurns"
}
@test "fails when a Copilot-only field ('target') is present in a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent" "target: cli"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "target"
}
@test "fails when 'hooks' is present in a plugin/APM-scope agent file (outside allowlist)" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent" "hooks: {}"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "hooks"
}
# ---------------------------------------------------------------------------
# Failing cases — plugin/APM scope: structural checks
# ---------------------------------------------------------------------------
@test "fails when name is not kebab-case in a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
mkdir -p "$root/.apm/agents"
cat > "$root/apm.yml" <<EOF
name: test-package
version: 0.1.0
type: skill
EOF
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
---
name: MyAgent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "kebab"
}
@test "fails when name does not match filename stem in a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
mkdir -p "$root/.apm/agents"
cat > "$root/apm.yml" <<EOF
name: test-package
version: 0.1.0
type: skill
EOF
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
---
name: wrong-name
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "does not match filename stem"
}
@test "fails when 'name' field is missing from a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
mkdir -p "$root/.apm/agents"
cat > "$root/apm.yml" <<EOF
name: test-package
version: 0.1.0
type: skill
EOF
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
---
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
}
@test "fails when 'description' field is missing from a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
mkdir -p "$root/.apm/agents"
cat > "$root/apm.yml" <<EOF
name: test-package
version: 0.1.0
type: skill
EOF
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
---
name: my-agent
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
}
@test "fails when a template HTML comment is left in plugin/APM-scope frontmatter" {
local root="$TMPDIR/pkg"
mkdir -p "$root/.apm/agents"
cat > "$root/apm.yml" <<EOF
name: test-package
version: 0.1.0
type: skill
EOF
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
<!-- model: sonnet
Optional. Omit to inherit the runtime default. -->
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "template HTML comments"
}
@test "fails when body contains unfilled FILL IN: placeholder in a plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
mkdir -p "$root/.apm/agents"
cat > "$root/apm.yml" <<EOF
name: test-package
version: 0.1.0
type: skill
EOF
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
FILL IN: replace this with your system prompt.
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
}
# ---------------------------------------------------------------------------
# Plugin/APM scope: no pair, no counterpart concept
# ---------------------------------------------------------------------------
@test "never raises a 'counterpart' FAIL on a clean plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_success
refute_output --partial "counterpart"
}
@test "never raises a 'counterpart' FAIL on a failing plugin/APM-scope agent file" {
local root="$TMPDIR/pkg"
make_apm_agent "$root" "my-agent" "tools: Read"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
refute_output --partial "counterpart"
}
# ---------------------------------------------------------------------------
# Scope-detection walk-up
# ---------------------------------------------------------------------------
@test "walk-up skips a type:-less apm.yml and finds a real package root further up" {
local root="$TMPDIR/case"
mkdir -p "$root/.apm/agents/nested/deeper"
cat > "$root/apm.yml" <<EOF
name: real-package
version: 1.0.0
type: skill
EOF
# Closer to the agent file than the real package root, but has no type:
# line — marketplace-only per monorepo-and-repo-shapes.md, must be skipped.
cat > "$root/.apm/agents/nested/apm.yml" <<EOF
name: not-a-package-manifest
version: 1.0.0
EOF
cat > "$root/.apm/agents/nested/deeper/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.apm/agents/nested/deeper/my-agent.agent.md"
assert_success
refute_output --partial "FAIL"
}
@test "type:-less apm.yml is not treated as plugin scope — falls through to project scope" {
local root="$TMPDIR/proj-marketplace"
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
cat > "$root/apm.yml" <<EOF
name: marketplace-root
version: 1.0.0
marketplace:
packages: []
EOF
cat > "$root/.claude/agents/my-agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
assert_success
refute_output --partial "FAIL"
}
@test "a bare plugin.json with no apm.yml is no longer plugin scope — falls through to project scope" {
local root="$TMPDIR/proj-legacy-plugin-json"
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
echo '{}' > "$root/plugin.json"
# 'hooks' is plugin-silently-ignored only at (old) plugin scope; at
# project scope it's a legitimate CC field. If this directory were
# mis-detected as plugin scope (old plugin.json-based logic), this would
# FAIL with a plugin-silently-ignored-fields finding on 'hooks'.
cat > "$root/.claude/agents/my-agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
hooks:
PostToolUse:
- match: ".*"
command: "echo done"
---
You are a test agent. When invoked, do the thing.
EOF
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
---
You are a test agent. When invoked, do the thing.
EOF
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
assert_success
refute_output --partial "hooks"
}
Reference in New Issue View Git Blame Copy Permalink