Defame1297
4ae2429840
Three divergences between what the audit skills claim and what the hooks enforce, each of which fails silently rather than loudly: - `skill-size-check.sh` blocked at 2900 words while `validate.sh` checked only the 500-line ceiling, so `/skill-audit` could report a skill ready to ship that the commit hook then rejected. `validate.sh` now checks the same pair on the same inclusive terms; the constants are duplicated with a comment naming the other file, because a plugin skill's scripts cannot read outside the plugin directory once installed to the cache. - Both audit skills' Step 1 passed `--config assets/vale/.vale.ini`, which is redundant (the wrapper self-locates its sibling config) and fragile: an agent that resolves the script path against the skill directory but not the config path gets E100, exit 2, which the surrounding fallback clause misreads as "vale unavailable" and downgrades to full LLM judgment with no signal. - The external-consumer test registered only the two Vale hooks, never the third shipped hook, so a lost executable bit would have broken every consumer while the local suite stayed green. Verified by mutation: `chmod 644` on the copied script now turns three passes into two failures. Also corrects the size hook's calibration comment, which claimed ~5.7-6.5 characters per word against a corpus whose measured median is 6.79 — the stated upper bound sat below the median, so the "calibrated with margin" claim was inverted for prose-dense files. MAX_WORDS is unchanged pending a decision; the comment is now explicit that the gate holds under 5,000 tokens for typical prose density, not for any file. Refs: #85
8.4 KiB
name, description, allowed-tools, metadata
| name | description | allowed-tools | metadata | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| agent-audit | Use when the user wants to review an agent definition they wrote, says "audit this agent", "check if my agent follows best practices", "review my agent file", or wants to know if an agent pair is ready to ship — even if they don't use the word "audit". Also invoke proactively after directly hand-editing an agent file pair outside agent-author — an unaudited hand-edit is the same risk as unreviewed code. Audits a Claude Code .md and Copilot .agent.md agent file pair across six dimensions: structural validation, provider safety, description quality, body quality, comment discipline, and pair consistency — plus provenance chain validation. Produces a compact findings report (findings only, no PASS noise) with Why and Fix per finding. Do not use to fix agent files — use /agent-author instead. Do not use to audit SKILL.md files — use /skill-audit instead. | Bash Read |
|
Gotchas
- The unit of authoring in this project is always a pair (CC
.md+ Copilot.agent.md). A missing counterpart is a FAIL under the kyberforge project convention — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures. - Plugin scope is detected by the presence of
plugin.jsonor.claude-plugin/plugin.jsonin the directory tree — not by the file path pattern. Walk up both paths at each level, don't guess. references/field-inventory.mdmust exist forvalidate.shto run. The script exits with an error if it is missing.- Do not output findings while auditing — gather internally, surface in Step 3 report.
Step 1 — Run structural validation
bash scripts/validate.sh <path-to-agent-file>
bash scripts/validate-provenance.sh <path-to-agent-file>
scripts/vale-wrap.sh <path-to-cc-file> <path-to-copilot-file>
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the ### Structure and ### Provider safety report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → ### Structure. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → ### Provider safety. A missing counterpart file → ### Pair consistency.
vale-wrap.sh ships inside this skill's own scripts/ — resolve it relative to this skill's directory the same way scripts/validate.sh is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no --config: handed none, the wrapper loads its own sibling assets/vale/.vale.ini, located from the script's path rather than from the cwd. Adding an explicit relative --config breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields E100 Runtime error ... does not exist, exit 2, which the fallback below then misreads as "vale unavailable". Run it against both files of the pair (not just the one passed in). Kyberforge applies to both files; KyberforgeCopilot applies to the .agent.md file only, since its one rule (Use proactively) flags CC-specific phrasing that's meaningless in a Copilot description — there's nothing to flag in the CC file, so it isn't scoped there. Every Vale alert is a FAIL — all rules are graded error — so report each one in the ### Description / ### Body dimensions citing its rule ID (e.g. KyberforgeCopilot.ProactivePhrase). Skip and fall back to Step 2 judgment if the vale binary is unavailable. If Vale reports 0 files scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered.
validate-provenance.sh validates the provenance chain between the agent pair's source_keys and the plugin-scoped sources.md (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the ### Provenance dimension — surface them verbatim with Why and Fix.
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: counterpart file exists, required fields present (name, description, non-empty body), name is kebab-case, Copilot CLI .agent.md name must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no FILL IN: placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read references/field-inventory.md for the authoritative field lists).
Step 2 — Qualitative checks
Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3.
Description (both files):
- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent. Vale's
Kyberforge.DescriptionOpeneralert flags the specific known-bad "This agent..." opener directly; verifying an arbitrary opening word is genuinely a strong verb still requires judgment. - Specificity: is the trigger condition stated precisely? — SUGGESTION if vague. Vale's
Kyberforge.VagueWordingalert covers known filler ("helps with", "utilize", ...) directly; report those as FAILs without re-deriving by judgment. Use proactivelyin a Copilot description: Vale'sKyberforgeCopilot.ProactivePhrasealert (Copilot file only) flags this directly — report it without re-deriving by judgment.
If a description finding is borderline, read references/description-quality.md.
Body:
- Direct role instruction: system prompt opens with
You are a [role]. When invoked, [action].— SUGGESTION if absent - One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded
- Generic, non-specific reference pointers to the
references/directory: Vale'sKyberforge.PaddingPhrasealert flags this directly — report it without re-deriving by judgment - Sentences that open with "There is"/"There are": Vale's
Kyberforge.SentenceOpenerThereIsalert flags this directly — report it without re-deriving by judgment
Body/Frontmatter comments:
- Inspect each comment block in the YAML frontmatter. For each comment, apply: "Would the agent get this wrong without this comment?" Flag any that answer "no" as padding.
- Look for patterns like
# Optional. <long explanation>or extensive inline guidance (more than 1–2 lines per field) that should be condensed or removed before shipping. - This mirrors skill-audit's body-discipline check but applies to template documentation in the frontmatter — template guidance belongs in development; agent-ready files should have minimal comments.
Pair consistency (cross-file):
- Both files exist — FAIL if counterpart is missing (kyberforge project convention; not a platform requirement from either CC or Copilot — label as such)
- The following checks are covered automatically by
validate.sh; apply them manually only when the script cannot run: both system prompt bodies non-empty — FAIL if either is empty
Step 3 — Report
Open with a coverage line:
Checked: structure · provider-safety · description · body · comment-discipline · pair-consistency · provenance
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. ### Provenance findings are sourced verbatim from validate-provenance.sh output — copy them without rephrasing.
For each finding:
FAIL/SUGGESTION <finding> — file:line
Why: <why this is a problem>
Fix: <exact change — quote before/after where applicable>
Close with:
## Result
PASS
PASS · P info
PASS (N suggestions)
PASS (N suggestions) · P info
FAIL (N fails · M suggestions)
FAIL (N fails · M suggestions) · P info
Run /agent-author to address findings.
Omit Run /agent-author to address findings. when there are no findings at all. Do not apply fixes — report and propose only.