fix(kyberforge): fix scope walk-up and manifest-parsing bugs from PR #93 review
A fresh /code-review of the APM-native authoring retarget (PR #93) found several correctness bugs beyond the ones already fixed on this branch: - new-agent.sh silently walked a marker-less subdirectory under $HOME up to user scope, contradicting its own usage text ("user scope is checked directly, no walk-up") and risking scaffolding into shared global ~/.claude or ~/.copilot directories instead of the intended local path. - The hand-copied apm.yml type: manifest detector in new-agent.sh and new-skill.sh accepted mismatched quotes (e.g. `type: "skill'`) that validate.sh's regex correctly rejects, and silently dropped a final apm.yml line lacking a trailing newline — causing the scaffolder and validator to disagree on scope for identical input. - Plugin-scope agent frontmatter could still contain the apm-agent.md template's HTML comments at ship time with no audit signal, yet apm compile copies frontmatter verbatim and <!-- --> breaks YAML parsing on both downstream harnesses. - ADR-0016 asserted agent-audit already implements a SUGGESTION heuristic for tool-restriction-needing plugin-scope agents; it doesn't. - agent-audit/README.md still described the old plugin-pair model this PR replaced with a single-file allowlist model. - validate.sh's project/user-scope CC-only/Copilot-only field checks and counterpart-missing check lost their only test coverage when the old plugin-pair fixture was deleted. Also replaces an echo-into-sed two-value parse (4 forks per call) with a single space-separated echo + read in both scaffolders. Regression tests added for every fix above, including one for a bug this pass introduced and the test suite caught: an initial two-line echo + `read` attempt silently dropped the second value, since `read` consumes only one line regardless of embedded newlines. Full suite: 158 bats tests, 39 shell-script tests, 12/12 summary categories, 0 failures. Refs: #89, #93
This commit is contained in:
@@ -1,10 +1,27 @@
|
||||
# agent-audit
|
||||
|
||||
Audits a Claude Code and Copilot agent definition file pair for correctness and quality.
|
||||
Audits an agent definition for correctness and quality — a single vendor-neutral file at
|
||||
plugin/APM scope, or a Claude Code and Copilot file pair at project/user scope.
|
||||
|
||||
## What it does
|
||||
|
||||
Accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives the counterpart automatically, and validates both. Runs structural checks via `validate.sh` (required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, silently-ignored fields at plugin scope), provenance chain validation via `validate-provenance.sh` (checks `source_keys` against `sources.md` at the plugin root), then qualitative checks on description phrasing and system prompt quality. Step 1 also runs a Vale-based prose sub-check via `vale-wrap.sh` against both files of the pair, using the `Kyberforge` style (both files) and `KyberforgeCopilot` style (Copilot file only) — every alert is a `FAIL`, cited by rule ID — falling back to Step 2 judgment when the `vale` binary is unavailable or reports `0 files` scanned. Produces a compact findings report in the same format as `skill-audit`.
|
||||
At **plugin/APM scope**, accepts the single `.apm/agents/<name>.agent.md` file — there is no
|
||||
counterpart. Structural checks via `validate.sh` hard-`FAIL` any frontmatter field outside the
|
||||
vendor-neutral allowlist (`name`, `description`, `model` — see ADR-0016), since `apm compile`
|
||||
copies frontmatter verbatim to both harnesses and an unsafe field can't be silently dropped for
|
||||
just one of them.
|
||||
|
||||
At **project/user scope**, accepts either file in a CC `.md` / Copilot `.agent.md` pair, derives
|
||||
the counterpart automatically, and validates both. Runs structural checks via `validate.sh`
|
||||
(required fields, kebab-case name, no placeholders, no CC-only fields in the Copilot file, no
|
||||
Copilot-only fields in the CC file), provenance chain validation via `validate-provenance.sh`
|
||||
(checks `source_keys` against `sources.md` at the plugin root — plugin/APM scope only), then
|
||||
qualitative checks on description phrasing and system prompt quality. Step 1 also runs a
|
||||
Vale-based prose sub-check via `vale-wrap.sh` against both files of the pair, using the
|
||||
`Kyberforge` style (both files) and `KyberforgeCopilot` style (Copilot file only) — every alert
|
||||
is a `FAIL`, cited by rule ID — falling back to Step 2 judgment when the `vale` binary is
|
||||
unavailable or reports `0 files` scanned. Produces a compact findings report in the same format
|
||||
as `skill-audit`.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
@@ -185,6 +185,17 @@ def check_apm_agent_file(fpath, allowlist, stem):
|
||||
fail(f"no valid YAML frontmatter (---...---) — {local_fname}")
|
||||
return
|
||||
|
||||
# The apm-agent.md template embeds its authoring guidance as HTML
|
||||
# comments inside the frontmatter block (so they render invisible in a
|
||||
# Markdown preview but stay visible in the raw file). get_frontmatter_keys
|
||||
# silently ignores any line that isn't a `key:` match, so a comment left
|
||||
# behind at ship time would otherwise pass unnoticed — yet apm compile
|
||||
# copies this frontmatter verbatim to both harnesses, and `<!-- -->` is
|
||||
# not valid YAML, so yaml.safe_load breaks on both downstream (ADR-0016).
|
||||
if re.search(r'<!--|-->', fm):
|
||||
fail(f"frontmatter still contains template HTML comments (<!-- ... -->) "
|
||||
f"— delete them before shipping — {local_fname}")
|
||||
|
||||
# Allowlist: only name/description/model may appear — no tools, no
|
||||
# Claude-only or Copilot-only fields. apm compile verbatim-copies
|
||||
# frontmatter to every target, so anything else is unsafe on at least
|
||||
|
||||
@@ -92,6 +92,78 @@ EOF
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Failing cases — project/user scope: CC/Copilot pair checks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "fails when a CC-only field ('maxTurns') is present in a project-scope Copilot file" {
|
||||
local root="$TMPDIR/project"
|
||||
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
||||
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
maxTurns: 10
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "CC-only field"
|
||||
assert_output --partial "maxTurns"
|
||||
}
|
||||
|
||||
@test "fails when a Copilot-only field ('target') is present in a project-scope CC file" {
|
||||
local root="$TMPDIR/project"
|
||||
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
||||
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
target: cli
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "Copilot-only field"
|
||||
assert_output --partial "target"
|
||||
}
|
||||
|
||||
@test "fails when the Copilot counterpart is missing at project scope" {
|
||||
local root="$TMPDIR/project"
|
||||
mkdir -p "$root/.git" "$root/.claude/agents"
|
||||
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "counterpart file not found"
|
||||
}
|
||||
|
||||
@test "--help exits 0 and shows Usage:" {
|
||||
run bash "$SCRIPT" --help
|
||||
assert_success
|
||||
@@ -243,6 +315,29 @@ EOF
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when a template HTML comment is left in plugin/APM-scope frontmatter" {
|
||||
local root="$TMPDIR/pkg"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
<!-- model: sonnet
|
||||
Optional. Omit to inherit the runtime default. -->
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "template HTML comments"
|
||||
}
|
||||
|
||||
@test "fails when body contains unfilled FILL IN: placeholder in a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
|
||||
Reference in New Issue
Block a user