Defame1297
eada85db99
field-inventory.md's apm-agent-allowlist and validate.sh's runtime check already included source_keys as a 4th allowed field, and the apm-agent.md template already instructed authors to add it for provenance tracking — but SKILL.md (x2), README.md, ADR-0016, and deployment-modes.md still described the allowlist as name/description/ model, "nothing else". The template itself even contradicted its own source_keys guidance with a header claiming "ONLY the three fields below — full stop" directly above it. Updates all six locations to document source_keys as the intentional 4th field, resolving the contradiction.
125 lines
13 KiB
Markdown
125 lines
13 KiB
Markdown
---
|
||
name: agent-audit
|
||
description: >
|
||
Use when the user wants to review an agent definition they wrote, says "audit this
|
||
agent", "check if my agent follows best practices", "review my agent file", or wants
|
||
to know if an agent pair is ready to ship — even if they don't use the word "audit".
|
||
Also invoke proactively after directly hand-editing an agent file pair outside
|
||
agent-author — an unaudited hand-edit is the same risk as unreviewed code.
|
||
Audits a Claude Code .md and Copilot .agent.md agent file pair across six dimensions:
|
||
structural validation, provider safety, description quality, body quality, comment
|
||
discipline, and pair consistency — plus provenance chain validation. Produces a
|
||
compact findings report
|
||
(findings only, no PASS noise) with Why and Fix per finding. Do not use to fix agent
|
||
files — use /agent-author instead. Do not use to audit SKILL.md files — use
|
||
/skill-audit instead.
|
||
allowed-tools: Bash Read
|
||
metadata:
|
||
category: factory
|
||
source_keys:
|
||
- context7-websites-code-claude
|
||
- claude-code-plugins-docs
|
||
- claude-code-subagents-docs
|
||
- context7-github-en-copilot
|
||
- github-custom-agents-configuration
|
||
---
|
||
|
||
## Gotchas
|
||
|
||
- The unit of authoring at project/user scope is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is a FAIL under the kyberforge project convention at those scopes — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures. **At plugin/APM scope there is no pair** — the unit of authoring is a single vendor-neutral `.apm/agents/<name>.agent.md` file, and Pair Consistency does not apply there at all (see below).
|
||
- Scope is detected by walking up from the agent file's directory: at each level, if `apm.yml` exists AND contains a top-level `type: instructions|skill|hybrid|prompts` line, that directory is an APM package root — plugin/APM scope. A `type:`-less `apm.yml` is marketplace-only (see `docs/research/docs/microsoft-apm/monorepo-and-repo-shapes.md`) — skip it and keep walking up. Otherwise, if `.git` is a directory at that level, stop there — project scope. If neither is found before the filesystem root, fall back to user scope at `$HOME`. `plugin.json`/`.claude-plugin/plugin.json` are no longer scope signals for this skill — a directory with only a `plugin.json` and no `apm.yml` falls through to project (or user) scope.
|
||
- `references/field-inventory.md` must exist for `validate.sh` to run. The script exits with an error if it is missing.
|
||
- Do not output findings while auditing — gather internally, surface in Step 3 report.
|
||
|
||
## Step 1 — Run structural validation
|
||
|
||
```bash
|
||
bash scripts/validate.sh <path-to-agent-file>
|
||
bash scripts/validate-provenance.sh <path-to-agent-file>
|
||
scripts/vale-wrap.sh <path-to-cc-file> <path-to-copilot-file> # project/user scope
|
||
scripts/vale-wrap.sh <path-to-apm-agent-file> # plugin/APM scope — single file
|
||
```
|
||
|
||
The script accepts either the CC file, the Copilot file, or (at plugin/APM scope) the single `.apm/agents/<name>.agent.md` file. It detects provider from extension and scope from the walk-up above, then runs the checks for that scope.
|
||
|
||
At **project/user scope** it derives the counterpart and runs the existing pair-based checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
|
||
|
||
At **plugin/APM scope** there is no counterpart — the script instead checks the single file's frontmatter against the `apm-agent-allowlist` in `references/field-inventory.md` (`name`, `description`, `model`, `source_keys` — nothing else; `source_keys` is provenance metadata, not a provider-specific field, and is validated separately by `validate-provenance.sh` against `sources.md`). Findings about missing fields, bad name format, name/filename-stem mismatch, empty body, or missing frontmatter → `### Structure`, same as project/user scope. Findings about any field outside the allowlist (e.g. `tools`, or any Claude-only/Copilot-only field carried over from a hand-edit) and body length → `### Provider safety` — but the dimension's meaning shifts here: it is no longer a CC-vs-Copilot field-leakage check, it's a vendor-neutral-field-allowlist check, since `apm compile` verbatim-copies this file's frontmatter to every target and there is no per-target integrator to reconcile a CC-only or Copilot-only field (ADR-0016). `### Pair consistency` never applies at this scope — the script never emits a missing-counterpart FAIL here, because there is nothing to pair by design.
|
||
|
||
`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". At project/user scope, run it against both files of the pair (not just the one passed in); at plugin/APM scope, run it against the single file. `Kyberforge` applies to all of these files via the `**/agents/*.md` glob; `KyberforgeCopilot` applies to any `*.agent.md` file — including the plugin/APM-scope file, which already has that extension — via the `**/*.agent.md` glob, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a vendor-neutral or Copilot description. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered.
|
||
|
||
`validate-provenance.sh` operates at plugin/APM scope only — it walks up from the agent file's directory the same way `validate.sh` does (nearest ancestor `apm.yml` with a top-level `type:` field; skip a `type:`-less marketplace-only `apm.yml`; stop at `.git` or the filesystem root) and exits 0 silently if that walk doesn't land on a package root, or when no provenance data exists. When it does apply, it validates the chain between the single file's own `source_keys` and the package-scoped `sources.md` (package root — see ADR-0010). Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
||
|
||
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually. At project/user scope: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists). At plugin/APM scope: required fields present (`name`, `description`, non-empty body), `name` is kebab-case and matches the filename stem, no `FILL IN:` placeholders, no frontmatter field outside `name`/`description`/`model`/`source_keys` (read the `apm-agent-allowlist` section of `references/field-inventory.md`; `source_keys` carries provenance metadata, checked separately by `validate-provenance.sh` against `sources.md`).
|
||
|
||
## Step 2 — Qualitative checks
|
||
|
||
Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3.
|
||
|
||
**Description (both files):**
|
||
- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent. Vale's `Kyberforge.DescriptionOpener` alert flags the specific known-bad "This agent..." opener directly; verifying an arbitrary opening word is genuinely a strong verb still requires judgment.
|
||
- Specificity: is the trigger condition stated precisely? — SUGGESTION if vague. Vale's `Kyberforge.VagueWording` alert covers known filler ("helps with", "utilize", ...) directly; report those as FAILs without re-deriving by judgment.
|
||
- `Use proactively` in a Copilot description: Vale's `KyberforgeCopilot.ProactivePhrase` alert (Copilot file only) flags this directly — report it without re-deriving by judgment.
|
||
|
||
If a description finding is borderline, read `references/description-quality.md`.
|
||
|
||
**Body:**
|
||
- Direct role instruction: system prompt opens with `You are a [role]. When invoked, [action].` — SUGGESTION if absent
|
||
- One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded
|
||
- Generic, non-specific reference pointers to the `references/` directory: Vale's `Kyberforge.PaddingPhrase` alert flags this directly — report it without re-deriving by judgment
|
||
- Sentences that open with "There is"/"There are": Vale's `Kyberforge.SentenceOpenerThereIs` alert flags this directly — report it without re-deriving by judgment
|
||
|
||
**Body/Frontmatter comments:**
|
||
- Inspect each comment block in the YAML frontmatter. For each comment, apply: *"Would the agent get this wrong without this comment?"* Flag any that answer "no" as padding.
|
||
- Look for patterns like `# Optional. <long explanation>` or extensive inline guidance (more than 1–2 lines per field) that should be condensed or removed before shipping.
|
||
- This mirrors skill-audit's body-discipline check but applies to template documentation in the frontmatter — template guidance belongs in development; agent-ready files should have minimal comments.
|
||
|
||
**Pair consistency (cross-file) — project/user scope only:**
|
||
- Both files exist — FAIL if counterpart is missing (kyberforge project convention; not a platform requirement from either CC or Copilot — label as such)
|
||
- The following checks are covered automatically by `validate.sh`; apply them manually only when the script cannot run: both system prompt bodies non-empty — FAIL if either is empty
|
||
- **Does not apply at plugin/APM scope** — there is only one file, by design; do not raise a Pair Consistency finding there under any circumstance.
|
||
|
||
**Unexpressable Claude-only behavior — plugin/APM scope only:**
|
||
- Read the description and body. If either implies a need the vendor-neutral frontmatter can no longer express — tool restriction, `isolation`, `memory`, or another Claude-only behavior that a hand-authored CC file could have declared — flag it as a SUGGESTION, never a FAIL. This is a known upstream schema limitation (APM's agent primitive has no per-target compile integrator, so `tools:`/`isolation`/etc. can't be emitted safely to both CC and Copilot — ADR-0016), not an authoring mistake. The finding exists to give the author visibility into the gap, not to imply the schema can be made to do something it can't.
|
||
- Example: a body that says "only use Read and Grep, never Edit" but the frontmatter has no `tools` field to enforce it — SUGGESTION, not FAIL.
|
||
|
||
## Step 3 — Report
|
||
|
||
Open with a coverage line. At project/user scope:
|
||
|
||
```text
|
||
Checked: structure · provider-safety · description · body · comment-discipline · pair-consistency · provenance
|
||
```
|
||
|
||
At plugin/APM scope, omit `pair-consistency` — it does not apply when there is no pair:
|
||
|
||
```text
|
||
Checked: structure · provider-safety · description · body · comment-discipline · provenance
|
||
```
|
||
|
||
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. `### Provenance` findings are sourced verbatim from `validate-provenance.sh` output — copy them without rephrasing.
|
||
|
||
For each finding:
|
||
|
||
```text
|
||
FAIL/SUGGESTION <finding> — file:line
|
||
Why: <why this is a problem>
|
||
Fix: <exact change — quote before/after where applicable>
|
||
```
|
||
|
||
Close with:
|
||
|
||
```text
|
||
## Result
|
||
|
||
PASS
|
||
PASS · P info
|
||
PASS (N suggestions)
|
||
PASS (N suggestions) · P info
|
||
FAIL (N fails · M suggestions)
|
||
FAIL (N fails · M suggestions) · P info
|
||
Run /agent-author to address findings.
|
||
```
|
||
|
||
Omit `Run /agent-author to address findings.` when there are no findings at all. Do not apply fixes — report and propose only.
|