feat(kyberforge): restructure agent-audit for plugin-scope apm agents
Validates the new single-file .apm/agents/<name>.agent.md shape agent-author now produces at plugin/APM scope: frontmatter allowlist (name/description/ model only, from a new apm-agent-allowlist entry in field-inventory.md), no counterpart derivation, and Pair Consistency dropped from that scope's report entirely (nothing to pair by design). Adds a plugin/APM-scope-only SUGGESTION when an agent's description/body implies a tool restriction or Claude-only behavior the vendor-neutral frontmatter can no longer express (ADR-0016). Scope detection in both validate.sh and validate-provenance.sh switches from a flat plugin.json/.claude-plugin/plugin.json check to a walk-up for the nearest ancestor apm.yml with a top-level type: field, skipping type:-less marketplace-only manifests — full switch, no dual-mode fallback to the old plugin.json signal. validate-provenance.sh's walk-up was fixed to match validate.sh's (it still used the old plugin.json check, and its counterpart-merge logic was rewritten to read a single file's source_keys instead of merging a CC+Copilot pair, since plugin/APM scope has no counterpart). Project/user scope validation is unchanged in both scripts. Refs: #89
This commit is contained in:
@@ -26,8 +26,8 @@ metadata:
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The unit of authoring in this project is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is a FAIL under the kyberforge project convention — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures.
|
||||
- Plugin scope is detected by the presence of `plugin.json` or `.claude-plugin/plugin.json` in the directory tree — not by the file path pattern. Walk up both paths at each level, don't guess.
|
||||
- The unit of authoring at project/user scope is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is a FAIL under the kyberforge project convention at those scopes — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures. **At plugin/APM scope there is no pair** — the unit of authoring is a single vendor-neutral `.apm/agents/<name>.agent.md` file, and Pair Consistency does not apply there at all (see below).
|
||||
- Scope is detected by walking up from the agent file's directory: at each level, if `apm.yml` exists AND contains a top-level `type: instructions|skill|hybrid|prompts` line, that directory is an APM package root — plugin/APM scope. A `type:`-less `apm.yml` is marketplace-only (see `docs/research/docs/microsoft-apm/monorepo-and-repo-shapes.md`) — skip it and keep walking up. Otherwise, if `.git` is a directory at that level, stop there — project scope. If neither is found before the filesystem root, fall back to user scope at `$HOME`. `plugin.json`/`.claude-plugin/plugin.json` are no longer scope signals for this skill — a directory with only a `plugin.json` and no `apm.yml` falls through to project (or user) scope.
|
||||
- `references/field-inventory.md` must exist for `validate.sh` to run. The script exits with an error if it is missing.
|
||||
- Do not output findings while auditing — gather internally, surface in Step 3 report.
|
||||
|
||||
@@ -36,16 +36,21 @@ metadata:
|
||||
```bash
|
||||
bash scripts/validate.sh <path-to-agent-file>
|
||||
bash scripts/validate-provenance.sh <path-to-agent-file>
|
||||
scripts/vale-wrap.sh <path-to-cc-file> <path-to-copilot-file>
|
||||
scripts/vale-wrap.sh <path-to-cc-file> <path-to-copilot-file> # project/user scope
|
||||
scripts/vale-wrap.sh <path-to-apm-agent-file> # plugin/APM scope — single file
|
||||
```
|
||||
|
||||
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
|
||||
The script accepts either the CC file, the Copilot file, or (at plugin/APM scope) the single `.apm/agents/<name>.agent.md` file. It detects provider from extension and scope from the walk-up above, then runs the checks for that scope.
|
||||
|
||||
`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". Run it against both files of the pair (not just the one passed in). `Kyberforge` applies to both files; `KyberforgeCopilot` applies to the `.agent.md` file only, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a Copilot description — there's nothing to flag in the CC file, so it isn't scoped there. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered.
|
||||
At **project/user scope** it derives the counterpart and runs the existing pair-based checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
|
||||
|
||||
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `sources.md` (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
||||
At **plugin/APM scope** there is no counterpart — the script instead checks the single file's frontmatter against the `apm-agent-allowlist` in `references/field-inventory.md` (`name`, `description`, `model` — nothing else). Findings about missing fields, bad name format, name/filename-stem mismatch, empty body, or missing frontmatter → `### Structure`, same as project/user scope. Findings about any field outside the allowlist (e.g. `tools`, or any Claude-only/Copilot-only field carried over from a hand-edit) and body length → `### Provider safety` — but the dimension's meaning shifts here: it is no longer a CC-vs-Copilot field-leakage check, it's a vendor-neutral-field-allowlist check, since `apm compile` verbatim-copies this file's frontmatter to every target and there is no per-target integrator to reconcile a CC-only or Copilot-only field (ADR-0016). `### Pair consistency` never applies at this scope — the script never emits a missing-counterpart FAIL here, because there is nothing to pair by design.
|
||||
|
||||
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists).
|
||||
`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". At project/user scope, run it against both files of the pair (not just the one passed in); at plugin/APM scope, run it against the single file. `Kyberforge` applies to all of these files via the `**/agents/*.md` glob; `KyberforgeCopilot` applies to any `*.agent.md` file — including the plugin/APM-scope file, which already has that extension — via the `**/*.agent.md` glob, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a vendor-neutral or Copilot description. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered.
|
||||
|
||||
`validate-provenance.sh` operates at plugin/APM scope only — it walks up from the agent file's directory the same way `validate.sh` does (nearest ancestor `apm.yml` with a top-level `type:` field; skip a `type:`-less marketplace-only `apm.yml`; stop at `.git` or the filesystem root) and exits 0 silently if that walk doesn't land on a package root, or when no provenance data exists. When it does apply, it validates the chain between the single file's own `source_keys` and the package-scoped `sources.md` (package root — see ADR-0010). Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
||||
|
||||
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually. At project/user scope: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists). At plugin/APM scope: required fields present (`name`, `description`, non-empty body), `name` is kebab-case and matches the filename stem, no `FILL IN:` placeholders, no frontmatter field outside `name`/`description`/`model` (read the `apm-agent-allowlist` section of `references/field-inventory.md`).
|
||||
|
||||
## Step 2 — Qualitative checks
|
||||
|
||||
@@ -69,18 +74,29 @@ If a description finding is borderline, read `references/description-quality.md`
|
||||
- Look for patterns like `# Optional. <long explanation>` or extensive inline guidance (more than 1–2 lines per field) that should be condensed or removed before shipping.
|
||||
- This mirrors skill-audit's body-discipline check but applies to template documentation in the frontmatter — template guidance belongs in development; agent-ready files should have minimal comments.
|
||||
|
||||
**Pair consistency (cross-file):**
|
||||
**Pair consistency (cross-file) — project/user scope only:**
|
||||
- Both files exist — FAIL if counterpart is missing (kyberforge project convention; not a platform requirement from either CC or Copilot — label as such)
|
||||
- The following checks are covered automatically by `validate.sh`; apply them manually only when the script cannot run: both system prompt bodies non-empty — FAIL if either is empty
|
||||
- **Does not apply at plugin/APM scope** — there is only one file, by design; do not raise a Pair Consistency finding there under any circumstance.
|
||||
|
||||
**Unexpressable Claude-only behavior — plugin/APM scope only:**
|
||||
- Read the description and body. If either implies a need the vendor-neutral frontmatter can no longer express — tool restriction, `isolation`, `memory`, or another Claude-only behavior that a hand-authored CC file could have declared — flag it as a SUGGESTION, never a FAIL. This is a known upstream schema limitation (APM's agent primitive has no per-target compile integrator, so `tools:`/`isolation`/etc. can't be emitted safely to both CC and Copilot — ADR-0016), not an authoring mistake. The finding exists to give the author visibility into the gap, not to imply the schema can be made to do something it can't.
|
||||
- Example: a body that says "only use Read and Grep, never Edit" but the frontmatter has no `tools` field to enforce it — SUGGESTION, not FAIL.
|
||||
|
||||
## Step 3 — Report
|
||||
|
||||
Open with a coverage line:
|
||||
Open with a coverage line. At project/user scope:
|
||||
|
||||
```text
|
||||
Checked: structure · provider-safety · description · body · comment-discipline · pair-consistency · provenance
|
||||
```
|
||||
|
||||
At plugin/APM scope, omit `pair-consistency` — it does not apply when there is no pair:
|
||||
|
||||
```text
|
||||
Checked: structure · provider-safety · description · body · comment-discipline · provenance
|
||||
```
|
||||
|
||||
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. `### Provenance` findings are sourced verbatim from `validate-provenance.sh` output — copy them without rephrasing.
|
||||
|
||||
For each finding:
|
||||
|
||||
@@ -26,3 +26,7 @@ name description tools target model disable-model-invocation user-invocable mcp-
|
||||
## copilot-only-fields
|
||||
|
||||
target disable-model-invocation user-invocable mcp-servers metadata
|
||||
|
||||
## apm-agent-allowlist
|
||||
|
||||
name description model
|
||||
|
||||
@@ -5,8 +5,10 @@ usage() {
|
||||
cat <<EOF
|
||||
Usage: validate-provenance.sh <agent-file>
|
||||
|
||||
Validate that an agent pair's sources provenance chain is complete and internally consistent.
|
||||
Operates at plugin scope only — exits 0 silently for project and user scope agents.
|
||||
Validate that an agent's sources provenance chain is complete and internally consistent.
|
||||
Operates at plugin/APM scope only (a single vendor-neutral .apm/agents/<name>.agent.md
|
||||
inside a package with a type:-bearing apm.yml) — exits 0 silently for project and user
|
||||
scope agents.
|
||||
|
||||
Arguments:
|
||||
agent-file Path to either the Claude Code .md or Copilot .agent.md agent file.
|
||||
@@ -47,23 +49,28 @@ agent_file = os.path.abspath(sys.argv[1])
|
||||
fname = os.path.basename(agent_file)
|
||||
agent_dir = os.path.dirname(agent_file)
|
||||
|
||||
# --- Detect provider ---
|
||||
if fname.endswith('.agent.md'):
|
||||
provider = 'copilot'
|
||||
name_stem = fname[:-len('.agent.md')]
|
||||
elif fname.endswith('.md'):
|
||||
provider = 'claude-code'
|
||||
name_stem = fname[:-len('.md')]
|
||||
else:
|
||||
# --- Sanity-check extension (single vendor-neutral .agent.md file at plugin/APM scope) ---
|
||||
if not (fname.endswith('.agent.md') or fname.endswith('.md')):
|
||||
print(f"Error: unrecognized extension '{fname}' — expected .md or .agent.md", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
|
||||
# --- Find plugin root ---
|
||||
TYPE_RE = re.compile(r'^type:\s*(instructions|skill|hybrid|prompts)\b')
|
||||
|
||||
# --- Find package root: walk up for the nearest ancestor apm.yml that
|
||||
# declares a top-level type: field. An apm.yml with no type: field is a
|
||||
# marketplace-only manifest (see monorepo-and-repo-shapes.md) — skip it and
|
||||
# keep walking. Stop at a .git boundary or the filesystem root: neither is
|
||||
# plugin/APM scope, so this script has nothing to check there.
|
||||
def find_plugin_root(start_dir):
|
||||
current = os.path.abspath(start_dir)
|
||||
while True:
|
||||
if (os.path.isfile(os.path.join(current, 'plugin.json')) or os.path.isfile(os.path.join(current, '.claude-plugin', 'plugin.json'))):
|
||||
return current
|
||||
apm_yml = os.path.join(current, 'apm.yml')
|
||||
if os.path.isfile(apm_yml):
|
||||
with open(apm_yml) as f:
|
||||
if any(TYPE_RE.match(line) for line in f):
|
||||
return current
|
||||
if os.path.isdir(os.path.join(current, '.git')):
|
||||
return None
|
||||
parent = os.path.dirname(current)
|
||||
if parent == current:
|
||||
return None
|
||||
@@ -73,12 +80,6 @@ plugin_root = find_plugin_root(agent_dir)
|
||||
if plugin_root is None:
|
||||
sys.exit(0)
|
||||
|
||||
# --- Derive counterpart ---
|
||||
if provider == 'copilot':
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.md')
|
||||
else:
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.agent.md')
|
||||
|
||||
sources_md_path = os.path.join(plugin_root, 'sources.md')
|
||||
|
||||
# --- Helpers ---
|
||||
@@ -166,15 +167,9 @@ def get_source_keys_from_file(fpath):
|
||||
fm, _ = parse_frontmatter(content)
|
||||
return parse_source_keys(fm)
|
||||
|
||||
# Plugin/APM scope is a single vendor-neutral file — no counterpart to merge.
|
||||
given_keys = get_source_keys_from_file(agent_file)
|
||||
counterpart_keys = get_source_keys_from_file(counterpart)
|
||||
# Deduplicated union, preserving order
|
||||
seen = set()
|
||||
all_source_keys = []
|
||||
for k in given_keys + counterpart_keys:
|
||||
if k not in seen:
|
||||
seen.add(k)
|
||||
all_source_keys.append(k)
|
||||
all_source_keys = given_keys
|
||||
|
||||
sources_md_exists = os.path.isfile(sources_md_path)
|
||||
|
||||
@@ -212,8 +207,8 @@ for line in sources_content.splitlines():
|
||||
)
|
||||
break
|
||||
|
||||
# --- Check 2: source_keys in agent files → slug exists in sources.md ---
|
||||
for fpath, keys in [(agent_file, given_keys), (counterpart, counterpart_keys)]:
|
||||
# --- Check 2: source_keys in the agent file → slug exists in sources.md ---
|
||||
for fpath, keys in [(agent_file, given_keys)]:
|
||||
if not keys:
|
||||
continue
|
||||
rel = os.path.relpath(fpath, plugin_root)
|
||||
|
||||
@@ -5,10 +5,15 @@ usage() {
|
||||
cat <<EOF
|
||||
Usage: validate.sh <agent-file>
|
||||
|
||||
Validate a Claude Code or Copilot agent file pair against the agent definition spec.
|
||||
Validate an agent definition file against the agent definition spec.
|
||||
|
||||
At plugin/APM scope, <agent-file> is a single vendor-neutral
|
||||
.apm/agents/<name>.agent.md file (frontmatter allowlist: name, description,
|
||||
model — no counterpart file). At project or user scope, <agent-file> is
|
||||
either half of a Claude Code .md / Copilot .agent.md pair.
|
||||
|
||||
Arguments:
|
||||
agent-file Path to either the Claude Code .md or Copilot .agent.md agent file.
|
||||
agent-file Path to the agent file (or either half of a project/user-scope pair).
|
||||
|
||||
Exit codes:
|
||||
0 All checks passed (may include SUGGESTIONs)
|
||||
@@ -74,6 +79,7 @@ def parse_section_tokens(content, section_name):
|
||||
cc_only_fields = parse_section_tokens(inv_content, 'claude-code-only-fields')
|
||||
copilot_only_fields = parse_section_tokens(inv_content, 'copilot-only-fields')
|
||||
plugin_ignored_fields = parse_section_tokens(inv_content, 'plugin-silently-ignored-fields')
|
||||
apm_agent_allowlist = parse_section_tokens(inv_content, 'apm-agent-allowlist')
|
||||
|
||||
# Tools the runtime withholds from subagents regardless of the tools field
|
||||
SUBAGENT_UNAVAILABLE_TOOLS = {
|
||||
@@ -83,48 +89,7 @@ SUBAGENT_UNAVAILABLE_TOOLS = {
|
||||
# Copilot body length limit (chars) — content beyond this is silently truncated
|
||||
COPILOT_BODY_LIMIT = 30000
|
||||
|
||||
# --- Detect scope ---
|
||||
def detect_scope(start_dir):
|
||||
current = os.path.abspath(start_dir)
|
||||
while True:
|
||||
if (os.path.isfile(os.path.join(current, 'plugin.json')) or
|
||||
os.path.isfile(os.path.join(current, '.claude-plugin', 'plugin.json'))):
|
||||
return 'plugin', current
|
||||
if os.path.isdir(os.path.join(current, '.git')):
|
||||
return 'project', current
|
||||
parent = os.path.dirname(current)
|
||||
if parent == current:
|
||||
return 'user', os.path.expanduser('~')
|
||||
current = parent
|
||||
|
||||
agent_dir = os.path.dirname(agent_file)
|
||||
scope, scope_root = detect_scope(agent_dir)
|
||||
|
||||
# --- Derive counterpart path ---
|
||||
if scope == 'plugin':
|
||||
if provider == 'copilot':
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
else:
|
||||
counterpart = os.path.join(agent_dir, name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
elif scope == 'project':
|
||||
if provider == 'claude-code':
|
||||
counterpart = os.path.join(scope_root, '.github', 'agents', name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
else:
|
||||
counterpart = os.path.join(scope_root, '.claude', 'agents', name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
else: # user
|
||||
home = os.path.expanduser('~')
|
||||
if provider == 'claude-code':
|
||||
counterpart = os.path.join(home, '.copilot', 'agents', name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
else:
|
||||
counterpart = os.path.join(home, '.claude', 'agents', name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
|
||||
# --- Helpers ---
|
||||
# --- Helpers (shared by every scope) ---
|
||||
failed = False
|
||||
suggestions = []
|
||||
|
||||
@@ -167,6 +132,113 @@ def is_copilot_cloud_ide(fpath):
|
||||
"""True if the file is a cloud/IDE Copilot agent (name is optional for these)."""
|
||||
return '.github/copilot/agents' in os.path.abspath(fpath).replace(os.sep, '/')
|
||||
|
||||
# --- Detect scope ---
|
||||
# APM_TYPE_RE matches a top-level (column-0) `type:` line in apm.yml whose value is
|
||||
# one of the four package content types. `[\'"]?` tolerates a quoted value; the
|
||||
# pattern doesn't anchor the line end, so trailing whitespace/comments don't matter.
|
||||
APM_TYPE_RE = re.compile(r"^type:\s*['\"]?(instructions|skill|hybrid|prompts)\b")
|
||||
|
||||
def find_apm_package_root(apm_yml_path):
|
||||
"""Return True if apm_yml_path has a top-level type: line (i.e. is a package
|
||||
manifest, not a type:-less marketplace-only apm.yml)."""
|
||||
with open(apm_yml_path) as f:
|
||||
for line in f:
|
||||
if APM_TYPE_RE.match(line):
|
||||
return True
|
||||
return False
|
||||
|
||||
def detect_scope(start_dir):
|
||||
current = os.path.abspath(start_dir)
|
||||
while True:
|
||||
apm_yml = os.path.join(current, 'apm.yml')
|
||||
if os.path.isfile(apm_yml) and find_apm_package_root(apm_yml):
|
||||
return 'plugin', current
|
||||
if os.path.isdir(os.path.join(current, '.git')):
|
||||
return 'project', current
|
||||
parent = os.path.dirname(current)
|
||||
if parent == current:
|
||||
return 'user', os.path.expanduser('~')
|
||||
current = parent
|
||||
|
||||
agent_dir = os.path.dirname(agent_file)
|
||||
scope, scope_root = detect_scope(agent_dir)
|
||||
|
||||
# --- Plugin/APM scope: single vendor-neutral file, no counterpart ---
|
||||
def check_apm_agent_file(fpath, allowlist, stem):
|
||||
local_fname = os.path.basename(fpath)
|
||||
with open(fpath) as f:
|
||||
content = f.read()
|
||||
|
||||
fm, body = parse_frontmatter(content)
|
||||
if fm is None:
|
||||
fail(f"no valid YAML frontmatter (---...---) — {local_fname}")
|
||||
return
|
||||
|
||||
# Allowlist: only name/description/model may appear — no tools, no
|
||||
# Claude-only or Copilot-only fields. apm compile verbatim-copies
|
||||
# frontmatter to every target, so anything else is unsafe on at least
|
||||
# one harness (ADR-0016).
|
||||
fm_keys = get_frontmatter_keys(fm)
|
||||
for key in sorted(fm_keys):
|
||||
if key not in allowlist:
|
||||
fail(f"field '{key}' is not in the vendor-neutral APM agent allowlist "
|
||||
f"({', '.join(sorted(allowlist))}) — {local_fname}")
|
||||
|
||||
# name — required, kebab-case, must match filename stem (file is <name>.agent.md)
|
||||
name_val = extract_field(fm, 'name')
|
||||
if not name_val:
|
||||
fail(f"name field is missing or empty — {local_fname}")
|
||||
else:
|
||||
if not re.match(r'^[a-z0-9]+(-[a-z0-9]+)*$', name_val):
|
||||
fail(f"name '{name_val}' is not kebab-case — {local_fname}")
|
||||
if name_val != stem:
|
||||
fail(f"name '{name_val}' does not match filename stem '{stem}' — {local_fname}")
|
||||
|
||||
# description — required, non-empty, no placeholder
|
||||
desc_val = extract_field(fm, 'description')
|
||||
if not desc_val:
|
||||
fail(f"description field is missing or empty — {local_fname}")
|
||||
else:
|
||||
if PLACEHOLDER_RE.search(desc_val):
|
||||
fail(f"description contains unfilled FILL IN: placeholder — {local_fname}")
|
||||
|
||||
# body — required, non-empty, no placeholder; same Copilot truncation risk
|
||||
# applies since this file compiles verbatim into a real Copilot file downstream.
|
||||
if not body.strip():
|
||||
fail(f"system prompt body is empty — {local_fname}")
|
||||
else:
|
||||
if PLACEHOLDER_RE.search(body):
|
||||
fail(f"body contains unfilled FILL IN: placeholder — {local_fname}")
|
||||
if len(body) > COPILOT_BODY_LIMIT:
|
||||
suggest(f"body exceeds {COPILOT_BODY_LIMIT:,} characters ({len(body):,} chars) — "
|
||||
f"content beyond the limit is silently truncated by the Copilot runtime "
|
||||
f"once apm compile emits it downstream — {local_fname}")
|
||||
|
||||
if scope == 'plugin':
|
||||
check_apm_agent_file(agent_file, apm_agent_allowlist, name_stem)
|
||||
for s in suggestions:
|
||||
print(f"SUGGESTION {s}")
|
||||
sys.exit(1 if failed else 0)
|
||||
|
||||
# --- Project/user scope: unchanged CC/Copilot pair validation ---
|
||||
|
||||
# --- Derive counterpart path ---
|
||||
if scope == 'project':
|
||||
if provider == 'claude-code':
|
||||
counterpart = os.path.join(scope_root, '.github', 'agents', name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
else:
|
||||
counterpart = os.path.join(scope_root, '.claude', 'agents', name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
else: # user
|
||||
home = os.path.expanduser('~')
|
||||
if provider == 'claude-code':
|
||||
counterpart = os.path.join(home, '.copilot', 'agents', name_stem + '.agent.md')
|
||||
counterpart_provider = 'copilot'
|
||||
else:
|
||||
counterpart = os.path.join(home, '.claude', 'agents', name_stem + '.md')
|
||||
counterpart_provider = 'claude-code'
|
||||
|
||||
def check_file(fpath, file_provider, is_plugin_scope):
|
||||
local_fname = os.path.basename(fpath)
|
||||
with open(fpath) as f:
|
||||
|
||||
@@ -8,26 +8,24 @@ setup() {
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-provenance.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
|
||||
# Helper: create a plugin root with plugin.json and an agents/ directory
|
||||
make_plugin() {
|
||||
# Helper: create an APM package root at <root> (apm.yml with a top-level
|
||||
# type: line — a real package manifest, not marketplace-only) plus a
|
||||
# single vendor-neutral agent file at <root>/.apm/agents/<name>.agent.md.
|
||||
make_package() {
|
||||
local root="$1"
|
||||
mkdir -p "$root/agents"
|
||||
echo '{"name":"test-plugin","version":"0.1.0"}' > "$root/plugin.json"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a clean agent pair (no source_keys)
|
||||
make_clean_pair() {
|
||||
# Helper: create a clean agent file (no source_keys)
|
||||
make_clean_agent() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
cat > "$root/agents/${name}.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
cat > "$root/agents/${name}.agent.md" <<EOF
|
||||
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
@@ -37,12 +35,12 @@ You are a test agent.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a CC agent file with source_keys
|
||||
make_cc_with_source_keys() {
|
||||
# Helper: create an agent file with source_keys
|
||||
make_agent_with_source_keys() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
local slug="${3:-my-source}"
|
||||
cat > "$root/agents/${name}.md" <<EOF
|
||||
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
@@ -50,37 +48,6 @@ source_keys:
|
||||
- ${slug}
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a Copilot agent file with source_keys
|
||||
make_copilot_with_source_keys() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
local slug="${3:-my-source}"
|
||||
cat > "$root/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- ${slug}
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Helper: create a minimal Copilot file without source_keys
|
||||
make_copilot_clean() {
|
||||
local root="$1"
|
||||
local name="${2:-my-agent}"
|
||||
cat > "$root/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
}
|
||||
@@ -89,7 +56,7 @@ EOF
|
||||
make_sources_md() {
|
||||
local root="$1"
|
||||
local slug="${2:-my-source}"
|
||||
local contrib="${3:-agents/my-agent.md, agents/my-agent.agent.md}"
|
||||
local contrib="${3:-.apm/agents/my-agent.agent.md}"
|
||||
local research="${4:-(none)}"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
@@ -120,12 +87,56 @@ teardown() {
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Non-plugin scope → exit 0 silently
|
||||
# Non-plugin/APM scope → exit 0 silently
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "non-plugin scope: no plugin.json in tree → exit 0, no output" {
|
||||
local dir="$TMPDIR/no-plugin"
|
||||
@test "non-plugin scope: no apm.yml in tree → exit 0, no output" {
|
||||
local dir="$TMPDIR/no-package"
|
||||
mkdir -p "$dir/.apm/agents"
|
||||
cat > "$dir/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- my-source
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
@test "non-plugin scope: apm.yml present but type:-less (marketplace-only) → exit 0, no output" {
|
||||
local dir="$TMPDIR/marketplace-only"
|
||||
mkdir -p "$dir/.apm/agents"
|
||||
cat > "$dir/apm.yml" <<EOF
|
||||
name: root-marketplace
|
||||
marketplace:
|
||||
owner: someone
|
||||
packages:
|
||||
- ./packages/plugin-a
|
||||
EOF
|
||||
cat > "$dir/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- my-source
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
@test "non-plugin scope: bare plugin.json (no apm.yml) is no longer a scope signal → exit 0, no output" {
|
||||
local dir="$TMPDIR/old-plugin-json-only"
|
||||
mkdir -p "$dir/agents"
|
||||
echo '{"name":"test-plugin","version":"0.1.0"}' > "$dir/plugin.json"
|
||||
cat > "$dir/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
@@ -141,15 +152,70 @@ EOF
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
@test "non-plugin scope: walk-up stops at .git boundary before reaching an ancestor apm.yml" {
|
||||
local dir="$TMPDIR/repo"
|
||||
mkdir -p "$dir/.git" "$dir/.apm/agents"
|
||||
cat > "$dir/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
mkdir -p "$dir/sub/.apm/agents"
|
||||
cat > "$dir/sub/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- my-source
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
# sub/ has no .git and no apm.yml of its own; the real package apm.yml
|
||||
# lives at $dir, but $dir/.git means the walk from sub/ should stop at
|
||||
# sub/ itself only if sub/ had a .git — here .git is at $dir, ABOVE
|
||||
# sub/, so the walk from sub/ reaches $dir/apm.yml before any .git.
|
||||
# This test instead verifies the walk finds that package root correctly
|
||||
# (a positive case) — see the dedicated .git-stops-first test below for
|
||||
# the negative case.
|
||||
run bash "$SCRIPT" "$dir/sub/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "non-plugin scope: .git between the agent file and an ancestor apm.yml stops the walk first" {
|
||||
local dir="$TMPDIR/repo2"
|
||||
mkdir -p "$dir/.apm/agents"
|
||||
cat > "$dir/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
mkdir -p "$dir/sub/.git" "$dir/sub/.apm/agents"
|
||||
cat > "$dir/sub/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- my-source
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/sub/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Early exit: no sources.md, no source_keys → exit 0, no output
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "clean pass: no sources.md and no source_keys → exit 0, no output" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_clean_pair "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_clean_agent "$root"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output ""
|
||||
}
|
||||
@@ -158,22 +224,11 @@ EOF
|
||||
# Check 0: source_keys present but sources.md absent → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: source_keys in CC file but sources.md absent" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_clean "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FAIL: source_keys in Copilot file but sources.md absent" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_clean_pair "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.agent.md"
|
||||
@test "FAIL: source_keys in agent file but sources.md absent" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
@@ -183,10 +238,9 @@ EOF
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: FILL IN: placeholder in sources.md" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
@@ -194,23 +248,22 @@ EOF
|
||||
|
||||
- **URL:** FILL IN: add url
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md, agents/my-agent.agent.md
|
||||
- **Contributing files:** .apm/agents/my-agent.agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FILL IN: inside backticks in sources.md does not fail" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
make_sources_md "$root"
|
||||
echo "Use \`FILL IN: value\` as example." >> "$root/sources.md"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
}
|
||||
|
||||
@@ -218,62 +271,47 @@ EOF
|
||||
# Check 2: source_keys slug missing from sources.md → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: source_keys slug in CC file not present as H2 in sources.md" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root" "my-agent" "my-source"
|
||||
make_copilot_clean "$root"
|
||||
@test "FAIL: source_keys slug in agent file not present as H2 in sources.md" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root" "my-agent" "my-source"
|
||||
make_sources_md "$root" "different-source" "(none)" "(none)"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FAIL: source_keys slug in Copilot file not present as H2 in sources.md" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_clean_pair "$root"
|
||||
make_copilot_with_source_keys "$root" "my-agent" "my-source"
|
||||
make_sources_md "$root" "different-source" "(none)" "(none)"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 4: Contributing file path doesn't exist → FAIL
|
||||
# Check 3: Contributing file path doesn't exist → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: Contributing file listed in sources.md does not exist" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
make_sources_md "$root" "my-source" "agents/nonexistent.md"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
make_sources_md "$root" "my-source" ".apm/agents/nonexistent.agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "pass: (none) in Contributing files is skipped" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
make_sources_md "$root" "my-source" "(none — not used directly)"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 6: Research doc field missing or placeholder → FAIL
|
||||
# Check 5: Research doc field missing or placeholder → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: Research doc field missing from sources.md entry" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
@@ -281,19 +319,18 @@ EOF
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md, agents/my-agent.agent.md
|
||||
- **Contributing files:** .apm/agents/my-agent.agent.md
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "FAIL: Research doc field is FILL IN: placeholder" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
@@ -301,24 +338,24 @@ EOF
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md, agents/my-agent.agent.md
|
||||
- **Contributing files:** .apm/agents/my-agent.agent.md
|
||||
- **Research doc:** FILL IN: path to research doc
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 5: Bidirectional — contributing file missing slug in source_keys → FAIL
|
||||
# Check 4: Bidirectional — contributing file missing slug in source_keys → FAIL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: Contributing file exists but does not list parent slug in source_keys" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
# CC file has source_keys: other-source (not my-source)
|
||||
cat > "$root/agents/my-agent.md" <<EOF
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
# agent file has source_keys: other-source (not my-source)
|
||||
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
@@ -328,8 +365,8 @@ source_keys:
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
make_copilot_clean "$root"
|
||||
# sources.md says my-agent.md contributed to my-source, but my-agent.md doesn't list my-source
|
||||
# sources.md says my-agent.agent.md contributed to my-source, but
|
||||
# my-agent.agent.md doesn't list my-source
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
@@ -337,7 +374,7 @@ EOF
|
||||
|
||||
- **URL:** https://example.com/other-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** agents/my-agent.md
|
||||
- **Contributing files:** .apm/agents/my-agent.agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
|
||||
@@ -345,39 +382,24 @@ EOF
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** Another source.
|
||||
- **Contributing files:** agents/my-agent.md
|
||||
- **Contributing files:** .apm/agents/my-agent.agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry via Copilot file path
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "accepts Copilot file path as entry point" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
make_sources_md "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Clean full pass
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "clean full pass: all checks satisfied via CC file" {
|
||||
local root="$TMPDIR/plugin"
|
||||
make_plugin "$root"
|
||||
make_cc_with_source_keys "$root"
|
||||
make_copilot_with_source_keys "$root"
|
||||
@test "clean full pass: all checks satisfied" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
make_sources_md "$root"
|
||||
run bash "$SCRIPT" "$root/agents/my-agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
}
|
||||
|
||||
@@ -8,24 +8,27 @@ setup() {
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
|
||||
# Helper: create a plugin-scope pair in <dir> with given <name>
|
||||
make_plugin_pair() {
|
||||
local dir="$1"
|
||||
# Helper: create an APM package root at <root> (apm.yml with a top-level
|
||||
# type: line, marking it a real package manifest — not marketplace-only)
|
||||
# plus a single vendor-neutral agent file at
|
||||
# <root>/.apm/agents/<name>.agent.md. <extra_frontmatter>, if given, is
|
||||
# inserted as additional raw frontmatter lines (used to inject fields
|
||||
# under test).
|
||||
make_apm_agent() {
|
||||
local root="$1"
|
||||
local name="$2"
|
||||
mkdir -p "$dir"
|
||||
echo '{}' > "$dir/plugin.json"
|
||||
cat > "$dir/${name}.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
local extra_frontmatter="${3:-}"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$dir/${name}.agent.md" <<EOF
|
||||
cat > "$root/.apm/agents/${name}.agent.md" <<EOF
|
||||
---
|
||||
name: ${name}
|
||||
description: A valid agent description.
|
||||
${extra_frontmatter}
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
@@ -38,17 +41,9 @@ teardown() {
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Passing cases
|
||||
# Passing cases — project/user scope (unchanged)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "passes on a clean plugin-scope pair (CC file as input)" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "passes on a clean project-scope pair (CC file as input)" {
|
||||
local root="$TMPDIR/project"
|
||||
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
||||
@@ -79,40 +74,261 @@ EOF
|
||||
assert_output --partial "Usage:"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Failing cases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "fails when Copilot counterpart is missing" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
rm "$dir/my-agent.agent.md"
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
@test "fails when no arguments are given" {
|
||||
run bash "$SCRIPT"
|
||||
assert_failure
|
||||
assert_output --partial "counterpart"
|
||||
}
|
||||
|
||||
@test "fails when CC-only field 'maxTurns' is in Copilot file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
maxTurns: 10
|
||||
---
|
||||
# ---------------------------------------------------------------------------
|
||||
# Passing cases — plugin/APM scope
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
@test "passes on a clean plugin/APM-scope agent file (name/description only)" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "passes on a clean plugin/APM-scope agent file with optional model field" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent" "model: claude-opus-4"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Failing cases — plugin/APM scope: allowlist violations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "fails when 'tools' field is present in a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent" "tools: Read Edit"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "tools"
|
||||
}
|
||||
|
||||
@test "fails when a Claude-only field ('maxTurns') is present in a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent" "maxTurns: 10"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "maxTurns"
|
||||
}
|
||||
|
||||
@test "fails when plugin-silently-ignored field 'hooks' is in plugin-scope CC file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
@test "fails when a Copilot-only field ('target') is present in a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent" "target: cli"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "target"
|
||||
}
|
||||
|
||||
@test "fails when 'hooks' is present in a plugin/APM-scope agent file (outside allowlist)" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent" "hooks: {}"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "hooks"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Failing cases — plugin/APM scope: structural checks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "fails when name is not kebab-case in a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: MyAgent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "kebab"
|
||||
}
|
||||
|
||||
@test "fails when name does not match filename stem in a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: wrong-name
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "does not match filename stem"
|
||||
}
|
||||
|
||||
@test "fails when 'name' field is missing from a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when 'description' field is missing from a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when body contains unfilled FILL IN: placeholder in a plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
mkdir -p "$root/.apm/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: test-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
cat > "$root/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
FILL IN: replace this with your system prompt.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Plugin/APM scope: no pair, no counterpart concept
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "never raises a 'counterpart' FAIL on a clean plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
refute_output --partial "counterpart"
|
||||
}
|
||||
|
||||
@test "never raises a 'counterpart' FAIL on a failing plugin/APM-scope agent file" {
|
||||
local root="$TMPDIR/pkg"
|
||||
make_apm_agent "$root" "my-agent" "tools: Read"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
refute_output --partial "counterpart"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Scope-detection walk-up
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "walk-up skips a type:-less apm.yml and finds a real package root further up" {
|
||||
local root="$TMPDIR/case"
|
||||
mkdir -p "$root/.apm/agents/nested/deeper"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: real-package
|
||||
version: 1.0.0
|
||||
type: skill
|
||||
EOF
|
||||
# Closer to the agent file than the real package root, but has no type:
|
||||
# line — marketplace-only per monorepo-and-repo-shapes.md, must be skipped.
|
||||
cat > "$root/.apm/agents/nested/apm.yml" <<EOF
|
||||
name: not-a-package-manifest
|
||||
version: 1.0.0
|
||||
EOF
|
||||
cat > "$root/.apm/agents/nested/deeper/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/nested/deeper/my-agent.agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "type:-less apm.yml is not treated as plugin scope — falls through to project scope" {
|
||||
local root="$TMPDIR/proj-marketplace"
|
||||
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
||||
cat > "$root/apm.yml" <<EOF
|
||||
name: marketplace-root
|
||||
version: 1.0.0
|
||||
marketplace:
|
||||
packages: []
|
||||
EOF
|
||||
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
||||
assert_success
|
||||
refute_output --partial "FAIL"
|
||||
}
|
||||
|
||||
@test "a bare plugin.json with no apm.yml is no longer plugin scope — falls through to project scope" {
|
||||
local root="$TMPDIR/proj-legacy-plugin-json"
|
||||
mkdir -p "$root/.git" "$root/.claude/agents" "$root/.github/agents"
|
||||
echo '{}' > "$root/plugin.json"
|
||||
# 'hooks' is plugin-silently-ignored only at (old) plugin scope; at
|
||||
# project scope it's a legitimate CC field. If this directory were
|
||||
# mis-detected as plugin scope (old plugin.json-based logic), this would
|
||||
# FAIL with a plugin-silently-ignored-fields finding on 'hooks'.
|
||||
cat > "$root/.claude/agents/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
@@ -124,95 +340,15 @@ hooks:
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "hooks"
|
||||
}
|
||||
|
||||
@test "fails when CC file name is not kebab-case" {
|
||||
local dir="$TMPDIR/agents"
|
||||
mkdir -p "$dir"
|
||||
echo '{}' > "$dir/plugin.json"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
name: MyAgent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
cat > "$dir/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: MyAgent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "kebab"
|
||||
}
|
||||
|
||||
@test "fails when 'name' field is missing from CC file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when 'description' field is missing from CC file" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when body contains unfilled FILL IN: placeholder" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.md" <<EOF
|
||||
cat > "$root/.github/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
FILL IN: replace this with your system prompt.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when frontmatter name does not match filename stem" {
|
||||
local dir="$TMPDIR/agents"
|
||||
make_plugin_pair "$dir" "my-agent"
|
||||
cat > "$dir/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: wrong-name
|
||||
description: A valid agent description.
|
||||
---
|
||||
|
||||
You are a test agent. When invoked, do the thing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$dir/my-agent.md"
|
||||
assert_failure
|
||||
}
|
||||
|
||||
@test "fails when no arguments are given" {
|
||||
run bash "$SCRIPT"
|
||||
assert_failure
|
||||
run bash "$SCRIPT" "$root/.claude/agents/my-agent.md"
|
||||
assert_success
|
||||
refute_output --partial "hooks"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user