Defame1297
792d3e1852
Addresses PR #85's outstanding review items after grilling the open
questions against ADR-0013/CONTEXT.md/ADR-0010:
Blocking fixes:
- vale-wrap.sh: replace json.dumps() escaping (which silently defeated
Vale's frontmatter scope on any description containing a quote,
backslash, or non-ASCII char — ~58% of the corpus) with a single-quoted
YAML scalar, substituting a Unicode right single quote for embedded
apostrophes rather than '' doubling (Vale's frontmatter scanner isn't a
full YAML parser and silently truncates on '' too).
- vale-wrap.sh: fix a blank-line-inside-a-folded-description truncation
bug via indentation-based, blank-line-tolerant body capture; narrow
flattening to `>`-style scalars only (`|` already works unflattened).
- skill-audit/agent-audit Step 1: make the vale-wrap.sh invocation
cwd-independent via git rev-parse --show-toplevel, fixing a bug where
no single cwd satisfied all three Step 1 commands.
- styles/Kyberforge/VagueQualifier.yml: prune 17 tokens verified
false-positive-dominated on this repo's own voice via a real corpus
sweep (obvious, clearly, usually, several, simple, easy, completely,
simply, tiny, etc.), keep 13 with real or unattested noise. Revert the
28 prose "fixes" those tokens drove across 14 skill files back to their
original, correct wording, including a functional regression to
caveman/SKILL.md's own filler-word list (a mention, not a use) — now
guarded with vale-off comments against recurrence.
Gaps:
- --minAlertLevel=warning on the pre-commit hook and Step 1 invocation
so warning-level rules actually surface, without collapsing the
FAIL/SUGGESTION severity mapping skill-audit/agent-audit rely on.
- vale-wrap.sh: fix --config=<path> equals-form, absolute-path silent
no-op, and a zero-file-argument stdin hang.
- Route vale-run and lint-runner through a documented wrapper script
when a target repo has one, instead of unconditionally recommending
bare `vale`.
- Wire Kyberforge.VagueQualifier/SentenceOpenerThereIs into skill-audit/
agent-audit's dimension-mapping prose (Body discipline).
- Add plugins/lint/sources.md provenance for lint-runner (ADR-0010).
- Sync both marketplace.json lint-entry descriptions with plugin.json.
- Retune skill-size-check.sh's MAX_WORDS 5000->2900 (measured ~1.6-1.7
tokens/word on this repo's corpus, the old value gated at ~8,500
tokens against a stated 5,000 ceiling); fix the >/>= line-count
boundary and wc -l undercount on files with no trailing newline.
- Document the vale binary as a Setup prerequisite in AGENTS.md.
- Fix SentenceOpenerThereIs's dead regex alternative and add a real
sentence-start anchor/scope.
- Fix a stale docs/research/docs/vale/ index pointer in kyberforge's
docs README (moved to plugins/lint/ in e1a5403).
- Rewrite ADR-0013's Consequences section past-tense to describe what
actually landed, and record the styles-portability limitation
(repo-root placement stays intentional; deferred to a separate
session per this PR's review).
Test coverage: 9 new vale-wrap.sh fixtures (quotes, backslash/unicode,
blank-line paragraphs, --config= form, zero-arg/absolute-path handling,
literal-block no-regression) and boundary-pair tests for
skill-size-check.sh's line/word ceilings.
bash tests/run-tests.sh: 9 scripts + 125 bats assertions, all passing.
scripts/check-manifests.sh and claude plugin validate --strict: clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MCQ648fLSFXPHGZdQ8gn58
109 lines
7.8 KiB
Markdown
109 lines
7.8 KiB
Markdown
---
|
||
name: agent-audit
|
||
description: >
|
||
Use when the user wants to review an agent definition they wrote, says "audit this
|
||
agent", "check if my agent follows best practices", "review my agent file", or wants
|
||
to know if an agent pair is ready to ship — even if they don't use the word "audit".
|
||
Also invoke proactively after directly hand-editing an agent file pair outside
|
||
agent-author — an unaudited hand-edit is the same risk as unreviewed code.
|
||
Audits a Claude Code .md and Copilot .agent.md agent file pair across six dimensions:
|
||
structural validation, provider safety, description quality, body quality, comment
|
||
discipline, and pair consistency — plus provenance chain validation. Produces a
|
||
compact findings report
|
||
(findings only, no PASS noise) with Why and Fix per finding. Do not use to fix agent
|
||
files — use /agent-author instead. Do not use to audit SKILL.md files — use
|
||
/skill-audit instead.
|
||
allowed-tools: Bash Read
|
||
metadata:
|
||
category: factory
|
||
source_keys:
|
||
- context7-websites-code-claude
|
||
- claude-code-plugins-docs
|
||
- claude-code-subagents-docs
|
||
- context7-github-en-copilot
|
||
- github-custom-agents-configuration
|
||
---
|
||
|
||
## Gotchas
|
||
|
||
- The unit of authoring in this project is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is a FAIL under the kyberforge project convention — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures.
|
||
- Plugin scope is detected by the presence of `plugin.json` or `.claude-plugin/plugin.json` in the directory tree — not by the file path pattern. Walk up both paths at each level, don't guess.
|
||
- `references/field-inventory.md` must exist for `validate.sh` to run. The script exits with an error if it is missing.
|
||
- Do not output findings while auditing — gather internally, surface in Step 3 report.
|
||
|
||
## Step 1 — Run structural validation
|
||
|
||
```bash
|
||
bash scripts/validate.sh <path-to-agent-file>
|
||
bash scripts/validate-provenance.sh <path-to-agent-file>
|
||
"$(git rev-parse --show-toplevel)/scripts/vale-wrap.sh" --config "$(git rev-parse --show-toplevel)/.vale.ini" <path-to-cc-file> <path-to-copilot-file>
|
||
```
|
||
|
||
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
|
||
|
||
`vale-wrap.sh` resolves its own path and `.vale.ini` via `git rev-parse --show-toplevel`, so it runs correctly regardless of the caller's cwd. Run it against both files of the pair (not just the one passed in). `Kyberforge` applies to both files; `KyberforgeCopilot` applies to the `.agent.md` file only, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a Copilot description — there's nothing to flag in the CC file, so it isn't scoped there. Map `error` → `FAIL` and `warning`/`suggestion` → `SUGGESTION` in the `### Description` / `### Body` dimensions, citing the rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if vale or `.vale.ini` is unavailable.
|
||
|
||
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `sources.md` (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
||
|
||
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists).
|
||
|
||
## Step 2 — Qualitative checks
|
||
|
||
Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3.
|
||
|
||
**Description (both files):**
|
||
- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent. Vale's `Kyberforge.DescriptionOpener` alert flags the specific known-bad "This agent..." opener directly; verifying an arbitrary opening word is genuinely a strong verb still requires judgment.
|
||
- Specificity: is the trigger condition stated precisely? — SUGGESTION if vague. Vale's `Kyberforge.VagueWording` alert covers known filler ("helps with", "utilize", ...) directly; report those without re-deriving by judgment.
|
||
- `Use proactively` in a Copilot description: Vale's `KyberforgeCopilot.ProactivePhrase` alert (Copilot file only) flags this directly — report it without re-deriving by judgment.
|
||
|
||
If a description finding is borderline, read `references/description-quality.md`.
|
||
|
||
**Body:**
|
||
- Direct role instruction: system prompt opens with `You are a [role]. When invoked, [action].` — SUGGESTION if absent
|
||
- One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded
|
||
- Generic, non-specific reference pointers to the `references/` directory: Vale's `Kyberforge.PaddingPhrase` alert flags this directly — report it without re-deriving by judgment
|
||
- Vague filler wording and sentences that open with "There is"/"There are": Vale's `Kyberforge.VagueQualifier` and `Kyberforge.SentenceOpenerThereIs` alerts flag this directly — report them without re-deriving by judgment
|
||
|
||
**Body/Frontmatter comments:**
|
||
- Inspect each comment block in the YAML frontmatter. For each comment, apply: *"Would the agent get this wrong without this comment?"* Flag any that answer "no" as padding.
|
||
- Look for patterns like `# Optional. <long explanation>` or extensive inline guidance (more than 1–2 lines per field) that should be condensed or removed before shipping.
|
||
- This mirrors skill-audit's body-discipline check but applies to template documentation in the frontmatter — template guidance belongs in development; agent-ready files should have minimal comments.
|
||
|
||
**Pair consistency (cross-file):**
|
||
- Both files exist — FAIL if counterpart is missing (kyberforge project convention; not a platform requirement from either CC or Copilot — label as such)
|
||
- The following checks are covered automatically by `validate.sh`; apply them manually only when the script cannot run: both system prompt bodies non-empty — FAIL if either is empty
|
||
|
||
## Step 3 — Report
|
||
|
||
Open with a coverage line:
|
||
|
||
```text
|
||
Checked: structure · provider-safety · description · body · comment-discipline · pair-consistency · provenance
|
||
```
|
||
|
||
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. `### Provenance` findings are sourced verbatim from `validate-provenance.sh` output — copy them without rephrasing.
|
||
|
||
For each finding:
|
||
|
||
```text
|
||
FAIL/SUGGESTION <finding> — file:line
|
||
Why: <why this is a problem>
|
||
Fix: <exact change — quote before/after where applicable>
|
||
```
|
||
|
||
Close with:
|
||
|
||
```text
|
||
## Result
|
||
|
||
PASS
|
||
PASS · P info
|
||
PASS (N suggestions)
|
||
PASS (N suggestions) · P info
|
||
FAIL (N fails · M suggestions)
|
||
FAIL (N fails · M suggestions) · P info
|
||
Run /agent-author to address findings.
|
||
```
|
||
|
||
Omit `Run /agent-author to address findings.` when there are no findings at all. Do not apply fixes — report and propose only.
|