From 675ba4023802a03bc2a83bf9c45984e0b42f5bda Mon Sep 17 00:00:00 2001 From: Defame1297 Date: Tue, 11 Aug 2026 18:05:33 +0000 Subject: [PATCH] feat(kyberforge): restructure agent-audit for plugin-scope apm agents MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Validates the new single-file .apm/agents/.agent.md shape agent-author now produces at plugin/APM scope: frontmatter allowlist (name/description/ model only, from a new apm-agent-allowlist entry in field-inventory.md), no counterpart derivation, and Pair Consistency dropped from that scope's report entirely (nothing to pair by design). Adds a plugin/APM-scope-only SUGGESTION when an agent's description/body implies a tool restriction or Claude-only behavior the vendor-neutral frontmatter can no longer express (ADR-0016). Scope detection in both validate.sh and validate-provenance.sh switches from a flat plugin.json/.claude-plugin/plugin.json check to a walk-up for the nearest ancestor apm.yml with a top-level type: field, skipping type:-less marketplace-only manifests — full switch, no dual-mode fallback to the old plugin.json signal. validate-provenance.sh's walk-up was fixed to match validate.sh's (it still used the old plugin.json check, and its counterpart-merge logic was rewritten to read a single file's source_keys instead of merging a CC+Copilot pair, since plugin/APM scope has no counterpart). Project/user scope validation is unchanged in both scripts. Refs: #89 --- .../kyberforge/skills/agent-audit/SKILL.md | 34 +- .../agent-audit/references/field-inventory.md | 4 + .../scripts/validate-provenance.sh | 53 ++- .../skills/agent-audit/scripts/validate.sh | 160 +++++-- .../tests/validate-provenance.bats | 334 ++++++++------- .../skills/agent-audit/tests/validate.bats | 400 ++++++++++++------ 6 files changed, 615 insertions(+), 370 deletions(-) diff --git a/plugins/kyberforge/skills/agent-audit/SKILL.md b/plugins/kyberforge/skills/agent-audit/SKILL.md index 08cc88d..ee02739 100644 --- a/plugins/kyberforge/skills/agent-audit/SKILL.md +++ b/plugins/kyberforge/skills/agent-audit/SKILL.md @@ -26,8 +26,8 @@ metadata: ## Gotchas -- The unit of authoring in this project is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is a FAIL under the kyberforge project convention — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures. -- Plugin scope is detected by the presence of `plugin.json` or `.claude-plugin/plugin.json` in the directory tree — not by the file path pattern. Walk up both paths at each level, don't guess. +- The unit of authoring at project/user scope is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is a FAIL under the kyberforge project convention at those scopes — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures. **At plugin/APM scope there is no pair** — the unit of authoring is a single vendor-neutral `.apm/agents/.agent.md` file, and Pair Consistency does not apply there at all (see below). +- Scope is detected by walking up from the agent file's directory: at each level, if `apm.yml` exists AND contains a top-level `type: instructions|skill|hybrid|prompts` line, that directory is an APM package root — plugin/APM scope. A `type:`-less `apm.yml` is marketplace-only (see `docs/research/docs/microsoft-apm/monorepo-and-repo-shapes.md`) — skip it and keep walking up. Otherwise, if `.git` is a directory at that level, stop there — project scope. If neither is found before the filesystem root, fall back to user scope at `$HOME`. `plugin.json`/`.claude-plugin/plugin.json` are no longer scope signals for this skill — a directory with only a `plugin.json` and no `apm.yml` falls through to project (or user) scope. - `references/field-inventory.md` must exist for `validate.sh` to run. The script exits with an error if it is missing. - Do not output findings while auditing — gather internally, surface in Step 3 report. @@ -36,16 +36,21 @@ metadata: ```bash bash scripts/validate.sh bash scripts/validate-provenance.sh -scripts/vale-wrap.sh +scripts/vale-wrap.sh # project/user scope +scripts/vale-wrap.sh # plugin/APM scope — single file ``` -The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`. +The script accepts either the CC file, the Copilot file, or (at plugin/APM scope) the single `.apm/agents/.agent.md` file. It detects provider from extension and scope from the walk-up above, then runs the checks for that scope. -`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". Run it against both files of the pair (not just the one passed in). `Kyberforge` applies to both files; `KyberforgeCopilot` applies to the `.agent.md` file only, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a Copilot description — there's nothing to flag in the CC file, so it isn't scoped there. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered. +At **project/user scope** it derives the counterpart and runs the existing pair-based checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`. -`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `sources.md` (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix. +At **plugin/APM scope** there is no counterpart — the script instead checks the single file's frontmatter against the `apm-agent-allowlist` in `references/field-inventory.md` (`name`, `description`, `model` — nothing else). Findings about missing fields, bad name format, name/filename-stem mismatch, empty body, or missing frontmatter → `### Structure`, same as project/user scope. Findings about any field outside the allowlist (e.g. `tools`, or any Claude-only/Copilot-only field carried over from a hand-edit) and body length → `### Provider safety` — but the dimension's meaning shifts here: it is no longer a CC-vs-Copilot field-leakage check, it's a vendor-neutral-field-allowlist check, since `apm compile` verbatim-copies this file's frontmatter to every target and there is no per-target integrator to reconcile a CC-only or Copilot-only field (ADR-0016). `### Pair consistency` never applies at this scope — the script never emits a missing-counterpart FAIL here, because there is nothing to pair by design. -If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists). +`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". At project/user scope, run it against both files of the pair (not just the one passed in); at plugin/APM scope, run it against the single file. `Kyberforge` applies to all of these files via the `**/agents/*.md` glob; `KyberforgeCopilot` applies to any `*.agent.md` file — including the plugin/APM-scope file, which already has that extension — via the `**/*.agent.md` glob, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a vendor-neutral or Copilot description. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered. + +`validate-provenance.sh` operates at plugin/APM scope only — it walks up from the agent file's directory the same way `validate.sh` does (nearest ancestor `apm.yml` with a top-level `type:` field; skip a `type:`-less marketplace-only `apm.yml`; stop at `.git` or the filesystem root) and exits 0 silently if that walk doesn't land on a package root, or when no provenance data exists. When it does apply, it validates the chain between the single file's own `source_keys` and the package-scoped `sources.md` (package root — see ADR-0010). Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix. + +If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually. At project/user scope: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists). At plugin/APM scope: required fields present (`name`, `description`, non-empty body), `name` is kebab-case and matches the filename stem, no `FILL IN:` placeholders, no frontmatter field outside `name`/`description`/`model` (read the `apm-agent-allowlist` section of `references/field-inventory.md`). ## Step 2 — Qualitative checks @@ -69,18 +74,29 @@ If a description finding is borderline, read `references/description-quality.md` - Look for patterns like `# Optional. ` or extensive inline guidance (more than 1–2 lines per field) that should be condensed or removed before shipping. - This mirrors skill-audit's body-discipline check but applies to template documentation in the frontmatter — template guidance belongs in development; agent-ready files should have minimal comments. -**Pair consistency (cross-file):** +**Pair consistency (cross-file) — project/user scope only:** - Both files exist — FAIL if counterpart is missing (kyberforge project convention; not a platform requirement from either CC or Copilot — label as such) - The following checks are covered automatically by `validate.sh`; apply them manually only when the script cannot run: both system prompt bodies non-empty — FAIL if either is empty +- **Does not apply at plugin/APM scope** — there is only one file, by design; do not raise a Pair Consistency finding there under any circumstance. + +**Unexpressable Claude-only behavior — plugin/APM scope only:** +- Read the description and body. If either implies a need the vendor-neutral frontmatter can no longer express — tool restriction, `isolation`, `memory`, or another Claude-only behavior that a hand-authored CC file could have declared — flag it as a SUGGESTION, never a FAIL. This is a known upstream schema limitation (APM's agent primitive has no per-target compile integrator, so `tools:`/`isolation`/etc. can't be emitted safely to both CC and Copilot — ADR-0016), not an authoring mistake. The finding exists to give the author visibility into the gap, not to imply the schema can be made to do something it can't. + - Example: a body that says "only use Read and Grep, never Edit" but the frontmatter has no `tools` field to enforce it — SUGGESTION, not FAIL. ## Step 3 — Report -Open with a coverage line: +Open with a coverage line. At project/user scope: ```text Checked: structure · provider-safety · description · body · comment-discipline · pair-consistency · provenance ``` +At plugin/APM scope, omit `pair-consistency` — it does not apply when there is no pair: + +```text +Checked: structure · provider-safety · description · body · comment-discipline · provenance +``` + Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. `### Provenance` findings are sourced verbatim from `validate-provenance.sh` output — copy them without rephrasing. For each finding: diff --git a/plugins/kyberforge/skills/agent-audit/references/field-inventory.md b/plugins/kyberforge/skills/agent-audit/references/field-inventory.md index 047995d..2e446fa 100644 --- a/plugins/kyberforge/skills/agent-audit/references/field-inventory.md +++ b/plugins/kyberforge/skills/agent-audit/references/field-inventory.md @@ -26,3 +26,7 @@ name description tools target model disable-model-invocation user-invocable mcp- ## copilot-only-fields target disable-model-invocation user-invocable mcp-servers metadata + +## apm-agent-allowlist + +name description model diff --git a/plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh b/plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh index 2b6e2fc..e14f1bc 100755 --- a/plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh +++ b/plugins/kyberforge/skills/agent-audit/scripts/validate-provenance.sh @@ -5,8 +5,10 @@ usage() { cat < -Validate that an agent pair's sources provenance chain is complete and internally consistent. -Operates at plugin scope only — exits 0 silently for project and user scope agents. +Validate that an agent's sources provenance chain is complete and internally consistent. +Operates at plugin/APM scope only (a single vendor-neutral .apm/agents/.agent.md +inside a package with a type:-bearing apm.yml) — exits 0 silently for project and user +scope agents. Arguments: agent-file Path to either the Claude Code .md or Copilot .agent.md agent file. @@ -47,23 +49,28 @@ agent_file = os.path.abspath(sys.argv[1]) fname = os.path.basename(agent_file) agent_dir = os.path.dirname(agent_file) -# --- Detect provider --- -if fname.endswith('.agent.md'): - provider = 'copilot' - name_stem = fname[:-len('.agent.md')] -elif fname.endswith('.md'): - provider = 'claude-code' - name_stem = fname[:-len('.md')] -else: +# --- Sanity-check extension (single vendor-neutral .agent.md file at plugin/APM scope) --- +if not (fname.endswith('.agent.md') or fname.endswith('.md')): print(f"Error: unrecognized extension '{fname}' — expected .md or .agent.md", file=sys.stderr) sys.exit(2) -# --- Find plugin root --- +TYPE_RE = re.compile(r'^type:\s*(instructions|skill|hybrid|prompts)\b') + +# --- Find package root: walk up for the nearest ancestor apm.yml that +# declares a top-level type: field. An apm.yml with no type: field is a +# marketplace-only manifest (see monorepo-and-repo-shapes.md) — skip it and +# keep walking. Stop at a .git boundary or the filesystem root: neither is +# plugin/APM scope, so this script has nothing to check there. def find_plugin_root(start_dir): current = os.path.abspath(start_dir) while True: - if (os.path.isfile(os.path.join(current, 'plugin.json')) or os.path.isfile(os.path.join(current, '.claude-plugin', 'plugin.json'))): - return current + apm_yml = os.path.join(current, 'apm.yml') + if os.path.isfile(apm_yml): + with open(apm_yml) as f: + if any(TYPE_RE.match(line) for line in f): + return current + if os.path.isdir(os.path.join(current, '.git')): + return None parent = os.path.dirname(current) if parent == current: return None @@ -73,12 +80,6 @@ plugin_root = find_plugin_root(agent_dir) if plugin_root is None: sys.exit(0) -# --- Derive counterpart --- -if provider == 'copilot': - counterpart = os.path.join(agent_dir, name_stem + '.md') -else: - counterpart = os.path.join(agent_dir, name_stem + '.agent.md') - sources_md_path = os.path.join(plugin_root, 'sources.md') # --- Helpers --- @@ -166,15 +167,9 @@ def get_source_keys_from_file(fpath): fm, _ = parse_frontmatter(content) return parse_source_keys(fm) +# Plugin/APM scope is a single vendor-neutral file — no counterpart to merge. given_keys = get_source_keys_from_file(agent_file) -counterpart_keys = get_source_keys_from_file(counterpart) -# Deduplicated union, preserving order -seen = set() -all_source_keys = [] -for k in given_keys + counterpart_keys: - if k not in seen: - seen.add(k) - all_source_keys.append(k) +all_source_keys = given_keys sources_md_exists = os.path.isfile(sources_md_path) @@ -212,8 +207,8 @@ for line in sources_content.splitlines(): ) break -# --- Check 2: source_keys in agent files → slug exists in sources.md --- -for fpath, keys in [(agent_file, given_keys), (counterpart, counterpart_keys)]: +# --- Check 2: source_keys in the agent file → slug exists in sources.md --- +for fpath, keys in [(agent_file, given_keys)]: if not keys: continue rel = os.path.relpath(fpath, plugin_root) diff --git a/plugins/kyberforge/skills/agent-audit/scripts/validate.sh b/plugins/kyberforge/skills/agent-audit/scripts/validate.sh index a2205df..7ed3f09 100755 --- a/plugins/kyberforge/skills/agent-audit/scripts/validate.sh +++ b/plugins/kyberforge/skills/agent-audit/scripts/validate.sh @@ -5,10 +5,15 @@ usage() { cat < -Validate a Claude Code or Copilot agent file pair against the agent definition spec. +Validate an agent definition file against the agent definition spec. + +At plugin/APM scope, is a single vendor-neutral +.apm/agents/.agent.md file (frontmatter allowlist: name, description, +model — no counterpart file). At project or user scope, is +either half of a Claude Code .md / Copilot .agent.md pair. Arguments: - agent-file Path to either the Claude Code .md or Copilot .agent.md agent file. + agent-file Path to the agent file (or either half of a project/user-scope pair). Exit codes: 0 All checks passed (may include SUGGESTIONs) @@ -74,6 +79,7 @@ def parse_section_tokens(content, section_name): cc_only_fields = parse_section_tokens(inv_content, 'claude-code-only-fields') copilot_only_fields = parse_section_tokens(inv_content, 'copilot-only-fields') plugin_ignored_fields = parse_section_tokens(inv_content, 'plugin-silently-ignored-fields') +apm_agent_allowlist = parse_section_tokens(inv_content, 'apm-agent-allowlist') # Tools the runtime withholds from subagents regardless of the tools field SUBAGENT_UNAVAILABLE_TOOLS = { @@ -83,48 +89,7 @@ SUBAGENT_UNAVAILABLE_TOOLS = { # Copilot body length limit (chars) — content beyond this is silently truncated COPILOT_BODY_LIMIT = 30000 -# --- Detect scope --- -def detect_scope(start_dir): - current = os.path.abspath(start_dir) - while True: - if (os.path.isfile(os.path.join(current, 'plugin.json')) or - os.path.isfile(os.path.join(current, '.claude-plugin', 'plugin.json'))): - return 'plugin', current - if os.path.isdir(os.path.join(current, '.git')): - return 'project', current - parent = os.path.dirname(current) - if parent == current: - return 'user', os.path.expanduser('~') - current = parent - -agent_dir = os.path.dirname(agent_file) -scope, scope_root = detect_scope(agent_dir) - -# --- Derive counterpart path --- -if scope == 'plugin': - if provider == 'copilot': - counterpart = os.path.join(agent_dir, name_stem + '.md') - counterpart_provider = 'claude-code' - else: - counterpart = os.path.join(agent_dir, name_stem + '.agent.md') - counterpart_provider = 'copilot' -elif scope == 'project': - if provider == 'claude-code': - counterpart = os.path.join(scope_root, '.github', 'agents', name_stem + '.agent.md') - counterpart_provider = 'copilot' - else: - counterpart = os.path.join(scope_root, '.claude', 'agents', name_stem + '.md') - counterpart_provider = 'claude-code' -else: # user - home = os.path.expanduser('~') - if provider == 'claude-code': - counterpart = os.path.join(home, '.copilot', 'agents', name_stem + '.agent.md') - counterpart_provider = 'copilot' - else: - counterpart = os.path.join(home, '.claude', 'agents', name_stem + '.md') - counterpart_provider = 'claude-code' - -# --- Helpers --- +# --- Helpers (shared by every scope) --- failed = False suggestions = [] @@ -167,6 +132,113 @@ def is_copilot_cloud_ide(fpath): """True if the file is a cloud/IDE Copilot agent (name is optional for these).""" return '.github/copilot/agents' in os.path.abspath(fpath).replace(os.sep, '/') +# --- Detect scope --- +# APM_TYPE_RE matches a top-level (column-0) `type:` line in apm.yml whose value is +# one of the four package content types. `[\'"]?` tolerates a quoted value; the +# pattern doesn't anchor the line end, so trailing whitespace/comments don't matter. +APM_TYPE_RE = re.compile(r"^type:\s*['\"]?(instructions|skill|hybrid|prompts)\b") + +def find_apm_package_root(apm_yml_path): + """Return True if apm_yml_path has a top-level type: line (i.e. is a package + manifest, not a type:-less marketplace-only apm.yml).""" + with open(apm_yml_path) as f: + for line in f: + if APM_TYPE_RE.match(line): + return True + return False + +def detect_scope(start_dir): + current = os.path.abspath(start_dir) + while True: + apm_yml = os.path.join(current, 'apm.yml') + if os.path.isfile(apm_yml) and find_apm_package_root(apm_yml): + return 'plugin', current + if os.path.isdir(os.path.join(current, '.git')): + return 'project', current + parent = os.path.dirname(current) + if parent == current: + return 'user', os.path.expanduser('~') + current = parent + +agent_dir = os.path.dirname(agent_file) +scope, scope_root = detect_scope(agent_dir) + +# --- Plugin/APM scope: single vendor-neutral file, no counterpart --- +def check_apm_agent_file(fpath, allowlist, stem): + local_fname = os.path.basename(fpath) + with open(fpath) as f: + content = f.read() + + fm, body = parse_frontmatter(content) + if fm is None: + fail(f"no valid YAML frontmatter (---...---) — {local_fname}") + return + + # Allowlist: only name/description/model may appear — no tools, no + # Claude-only or Copilot-only fields. apm compile verbatim-copies + # frontmatter to every target, so anything else is unsafe on at least + # one harness (ADR-0016). + fm_keys = get_frontmatter_keys(fm) + for key in sorted(fm_keys): + if key not in allowlist: + fail(f"field '{key}' is not in the vendor-neutral APM agent allowlist " + f"({', '.join(sorted(allowlist))}) — {local_fname}") + + # name — required, kebab-case, must match filename stem (file is .agent.md) + name_val = extract_field(fm, 'name') + if not name_val: + fail(f"name field is missing or empty — {local_fname}") + else: + if not re.match(r'^[a-z0-9]+(-[a-z0-9]+)*$', name_val): + fail(f"name '{name_val}' is not kebab-case — {local_fname}") + if name_val != stem: + fail(f"name '{name_val}' does not match filename stem '{stem}' — {local_fname}") + + # description — required, non-empty, no placeholder + desc_val = extract_field(fm, 'description') + if not desc_val: + fail(f"description field is missing or empty — {local_fname}") + else: + if PLACEHOLDER_RE.search(desc_val): + fail(f"description contains unfilled FILL IN: placeholder — {local_fname}") + + # body — required, non-empty, no placeholder; same Copilot truncation risk + # applies since this file compiles verbatim into a real Copilot file downstream. + if not body.strip(): + fail(f"system prompt body is empty — {local_fname}") + else: + if PLACEHOLDER_RE.search(body): + fail(f"body contains unfilled FILL IN: placeholder — {local_fname}") + if len(body) > COPILOT_BODY_LIMIT: + suggest(f"body exceeds {COPILOT_BODY_LIMIT:,} characters ({len(body):,} chars) — " + f"content beyond the limit is silently truncated by the Copilot runtime " + f"once apm compile emits it downstream — {local_fname}") + +if scope == 'plugin': + check_apm_agent_file(agent_file, apm_agent_allowlist, name_stem) + for s in suggestions: + print(f"SUGGESTION {s}") + sys.exit(1 if failed else 0) + +# --- Project/user scope: unchanged CC/Copilot pair validation --- + +# --- Derive counterpart path --- +if scope == 'project': + if provider == 'claude-code': + counterpart = os.path.join(scope_root, '.github', 'agents', name_stem + '.agent.md') + counterpart_provider = 'copilot' + else: + counterpart = os.path.join(scope_root, '.claude', 'agents', name_stem + '.md') + counterpart_provider = 'claude-code' +else: # user + home = os.path.expanduser('~') + if provider == 'claude-code': + counterpart = os.path.join(home, '.copilot', 'agents', name_stem + '.agent.md') + counterpart_provider = 'copilot' + else: + counterpart = os.path.join(home, '.claude', 'agents', name_stem + '.md') + counterpart_provider = 'claude-code' + def check_file(fpath, file_provider, is_plugin_scope): local_fname = os.path.basename(fpath) with open(fpath) as f: diff --git a/plugins/kyberforge/skills/agent-audit/tests/validate-provenance.bats b/plugins/kyberforge/skills/agent-audit/tests/validate-provenance.bats index f5627f4..12c57e4 100644 --- a/plugins/kyberforge/skills/agent-audit/tests/validate-provenance.bats +++ b/plugins/kyberforge/skills/agent-audit/tests/validate-provenance.bats @@ -8,26 +8,24 @@ setup() { SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-provenance.sh" TMPDIR="$(mktemp -d)" - # Helper: create a plugin root with plugin.json and an agents/ directory - make_plugin() { + # Helper: create an APM package root at (apm.yml with a top-level + # type: line — a real package manifest, not marketplace-only) plus a + # single vendor-neutral agent file at /.apm/agents/.agent.md. + make_package() { local root="$1" - mkdir -p "$root/agents" - echo '{"name":"test-plugin","version":"0.1.0"}' > "$root/plugin.json" + mkdir -p "$root/.apm/agents" + cat > "$root/apm.yml" < "$root/agents/${name}.md" < "$root/agents/${name}.agent.md" < "$root/.apm/agents/${name}.agent.md" < "$root/agents/${name}.md" < "$root/.apm/agents/${name}.agent.md" < "$root/agents/${name}.agent.md" < "$root/agents/${name}.agent.md" < "$root/sources.md" < "$dir/.apm/agents/my-agent.agent.md" < "$dir/apm.yml" < "$dir/.apm/agents/my-agent.agent.md" < "$dir/plugin.json" cat > "$dir/agents/my-agent.md" < "$dir/apm.yml" < "$dir/sub/.apm/agents/my-agent.agent.md" < "$dir/apm.yml" < "$dir/sub/.apm/agents/my-agent.agent.md" < "$root/sources.md" <> "$root/sources.md" - run bash "$SCRIPT" "$root/agents/my-agent.md" + run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md" assert_success } @@ -218,62 +271,47 @@ EOF # Check 2: source_keys slug missing from sources.md → FAIL # --------------------------------------------------------------------------- -@test "FAIL: source_keys slug in CC file not present as H2 in sources.md" { - local root="$TMPDIR/plugin" - make_plugin "$root" - make_cc_with_source_keys "$root" "my-agent" "my-source" - make_copilot_clean "$root" +@test "FAIL: source_keys slug in agent file not present as H2 in sources.md" { + local root="$TMPDIR/package" + make_package "$root" + make_agent_with_source_keys "$root" "my-agent" "my-source" make_sources_md "$root" "different-source" "(none)" "(none)" - run bash "$SCRIPT" "$root/agents/my-agent.md" - assert_failure - assert_output --partial "FAIL" -} - -@test "FAIL: source_keys slug in Copilot file not present as H2 in sources.md" { - local root="$TMPDIR/plugin" - make_plugin "$root" - make_clean_pair "$root" - make_copilot_with_source_keys "$root" "my-agent" "my-source" - make_sources_md "$root" "different-source" "(none)" "(none)" - run bash "$SCRIPT" "$root/agents/my-agent.agent.md" + run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md" assert_failure assert_output --partial "FAIL" } # --------------------------------------------------------------------------- -# Check 4: Contributing file path doesn't exist → FAIL +# Check 3: Contributing file path doesn't exist → FAIL # --------------------------------------------------------------------------- @test "FAIL: Contributing file listed in sources.md does not exist" { - local root="$TMPDIR/plugin" - make_plugin "$root" - make_cc_with_source_keys "$root" - make_copilot_with_source_keys "$root" - make_sources_md "$root" "my-source" "agents/nonexistent.md" - run bash "$SCRIPT" "$root/agents/my-agent.md" + local root="$TMPDIR/package" + make_package "$root" + make_agent_with_source_keys "$root" + make_sources_md "$root" "my-source" ".apm/agents/nonexistent.agent.md" + run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md" assert_failure assert_output --partial "FAIL" } @test "pass: (none) in Contributing files is skipped" { - local root="$TMPDIR/plugin" - make_plugin "$root" - make_cc_with_source_keys "$root" - make_copilot_with_source_keys "$root" + local root="$TMPDIR/package" + make_package "$root" + make_agent_with_source_keys "$root" make_sources_md "$root" "my-source" "(none — not used directly)" - run bash "$SCRIPT" "$root/agents/my-agent.md" + run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md" assert_success } # --------------------------------------------------------------------------- -# Check 6: Research doc field missing or placeholder → FAIL +# Check 5: Research doc field missing or placeholder → FAIL # --------------------------------------------------------------------------- @test "FAIL: Research doc field missing from sources.md entry" { - local root="$TMPDIR/plugin" - make_plugin "$root" - make_cc_with_source_keys "$root" - make_copilot_with_source_keys "$root" + local root="$TMPDIR/package" + make_package "$root" + make_agent_with_source_keys "$root" cat > "$root/sources.md" < "$root/sources.md" < "$root/agents/my-agent.md" < "$root/.apm/agents/my-agent.agent.md" < "$root/sources.md" < with given - make_plugin_pair() { - local dir="$1" + # Helper: create an APM package root at (apm.yml with a top-level + # type: line, marking it a real package manifest — not marketplace-only) + # plus a single vendor-neutral agent file at + # /.apm/agents/.agent.md. , if given, is + # inserted as additional raw frontmatter lines (used to inject fields + # under test). + make_apm_agent() { + local root="$1" local name="$2" - mkdir -p "$dir" - echo '{}' > "$dir/plugin.json" - cat > "$dir/${name}.md" < "$root/apm.yml" < "$dir/${name}.agent.md" < "$root/.apm/agents/${name}.agent.md" < "$dir/my-agent.agent.md" < "$dir/my-agent.md" < "$root/apm.yml" < "$root/.apm/agents/my-agent.agent.md" < "$root/apm.yml" < "$root/.apm/agents/my-agent.agent.md" < "$root/apm.yml" < "$root/.apm/agents/my-agent.agent.md" < "$root/apm.yml" < "$root/.apm/agents/my-agent.agent.md" < "$root/apm.yml" < "$root/.apm/agents/my-agent.agent.md" < "$root/apm.yml" < "$root/.apm/agents/nested/apm.yml" < "$root/.apm/agents/nested/deeper/my-agent.agent.md" < "$root/apm.yml" < "$root/.claude/agents/my-agent.md" < "$root/.github/agents/my-agent.agent.md" < "$root/plugin.json" + # 'hooks' is plugin-silently-ignored only at (old) plugin scope; at + # project scope it's a legitimate CC field. If this directory were + # mis-detected as plugin scope (old plugin.json-based logic), this would + # FAIL with a plugin-silently-ignored-fields finding on 'hooks'. + cat > "$root/.claude/agents/my-agent.md" < "$dir/plugin.json" - cat > "$dir/my-agent.md" < "$dir/my-agent.agent.md" < "$dir/my-agent.md" < "$dir/my-agent.md" < "$dir/my-agent.md" < "$root/.github/agents/my-agent.agent.md" < "$dir/my-agent.agent.md" <