Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

feat(kyberforge): enforce the ADR-0020 context contract for skills and agents

Browse Source 
Skill name+description pairs are preloaded into every session, costing
~6,200 tokens across 39 skills before any skill is invoked. The authoring
rules mandated that growth: skill-author:104 and description-quality.md:21
both required padding, while skill-author:102 (the deflating rule) had no
FAIL condition behind it.

Gates (blocking, no baseline file):
- description 250 chars SUGGESTION / 400 FAIL, measured on the folded
  YAML value
- body-only 600 words SUGGESTION / 900 FAIL, independent of the unchanged
  whole-file 2770-word / 500-line spec backstop
- every boundary-clause routing target must resolve to a real skill or
  agent; catches skill-improve, neuledge-context and gitea-labels
- agents take the description gates but deliberately no body gate; a test
  pins that absence

Vale: DescriptionOpener widened to ^This\b, new CompositionNote rule
banning architecture notes from descriptions. 10 hits, 0 false positives.

Kyberforge's own four skills retrofitted: descriptions 3,364 -> 938 chars
(-72%), bodies 8,306 -> 2,487 words (-70%), all via the apm-workflow
dispatch pattern. Fixes the skill-improve dangling route and the
agent-author misroute to manual review.

Also fixes a pre-existing false positive where any line-initial 'read '
was flagged as interactive input, which had already caused two scripts to
be rewritten around it.

Refs: ADR-0020
This commit is contained in:
Defame1297
2026-08-14 21:13:13 +00:00
parent 1c6eababb0
commit 4a5c3c0cff
104 changed files with 6272 additions and 1880 deletions
Download Patch File Download Diff File Expand all files Collapse all files

119
plugins/kyberforge/skills/agent-audit/SKILL.md
View File

@@ -1,18 +1,10 @@
---
name: agent-audit
description: >
Use when the user wants to review an agent definition they wrote, says "audit this
agent", "check if my agent follows best practices", "review my agent file", or wants
to know if an agent pair is ready to ship — even if they don't use the word "audit".
Also invoke proactively after directly hand-editing an agent file pair outside
agent-author — an unaudited hand-edit is the same risk as unreviewed code.
Audits a Claude Code .md and Copilot .agent.md agent file pair across six dimensions:
structural validation, provider safety, description quality, body quality, comment
discipline, and pair consistency — plus provenance chain validation. Produces a
compact findings report
(findings only, no PASS noise) with Why and Fix per finding. Do not use to fix agent
files — use /agent-author instead. Do not use to audit SKILL.md files — use
/skill-audit instead.
Use when the user wants an agent definition audited — "audit this agent",
"review my agent file", "is this ready to ship" — or after hand-editing an
agent outside agent-author. Not applying fixes -> agent-author. Not a skill
directory -> skill-audit.
allowed-tools: Bash Read
metadata:
category: factory
@@ -26,80 +18,67 @@ metadata:
## Gotchas
- The unit of authoring at project/user scope is always a pair (CC `.md` + Copilot `.agent.md`). A missing counterpart is a FAIL under the kyberforge project convention at those scopes — neither the CC nor the Copilot platform itself requires a counterpart file. Label such findings as project convention violations, not platform spec failures. **At plugin/APM scope there is no pair** — the unit of authoring is a single vendor-neutral `.apm/agents/<name>.agent.md` file, and Pair Consistency does not apply there at all (see below).
- Scope is detected by walking up from the agent file's directory: at each level, if `apm.yml` exists AND contains a top-level `type: instructions|skill|hybrid|prompts` line, that directory is an APM package root — plugin/APM scope. A `type:`-less `apm.yml` is marketplace-only (see `docs/research/docs/microsoft-apm/monorepo-and-repo-shapes.md`) — skip it and keep walking up. Otherwise, if `.git` is a directory at that level, stop there — project scope. If neither is found before the filesystem root, fall back to user scope at `$HOME`. `plugin.json`/`.claude-plugin/plugin.json` are no longer scope signals for this skill — a directory with only a `plugin.json` and no `apm.yml` falls through to project (or user) scope.
- `references/field-inventory.md` must exist for `validate.sh` to run. The script exits with an error if it is missing.
- Do not output findings while auditing — gather internally, surface in Step 3 report.
- Do not narrate PASS/FAIL per check while auditing. Gather findings internally and surface them only in the Step 4 report. Narrating each check as you go is the default failure mode here.
- Agents take the same 250/400-character description gates as skills and **no body word gate at all** — an agent body becomes the system prompt of a fresh context, so the 900-word skill ceiling does not transfer. Judge an over-long agent body through the delegation check, never by word count.
- At plugin/APM scope the agent is a single vendor-neutral file by design: never raise a pair-consistency finding there, and provider safety stops meaning Claude-Code-versus-Copilot field leakage.
- Vale reporting `0 files` scanned means NOT RUN, not clean. Fall back to full Step 3 judgment for every dimension it would have covered.
## Step 1 — Run structural validation
## Step 1 — Deterministic checks
Resolve all three paths against this skill's own directory so they work from a repo checkout and an installed plugin cache alike. Run exactly:
```bash
bash scripts/validate.sh <path-to-agent-file>
bash scripts/validate-provenance.sh <path-to-agent-file>
scripts/vale-wrap.sh <path-to-cc-file> <path-to-copilot-file> # project/user scope
scripts/vale-wrap.sh <path-to-apm-agent-file> # plugin/APM scope — single file
bash scripts/validate.sh <agent-file>
bash scripts/validate-provenance.sh <agent-file>
scripts/vale-wrap.sh <agent-file> [<counterpart-file>]
```
The script accepts either the CC file, the Copilot file, or (at plugin/APM scope) the single `.apm/agents/<name>.agent.md` file. It detects provider from extension and scope from the walk-up above, then runs the checks for that scope.
`validate.sh` takes either half of a project/user-scope pair or the single plugin/APM-scope file, detects the provider from the extension and the scope by walking up, then checks required fields, kebab-case `name`, `FILL IN:` placeholders, template HTML comments left in frontmatter, the ADR-0020 description budget (250 chars SUGGESTION, 400 FAIL, measured on the folded YAML value) and the fields that scope permits. Its findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both — except the ones the Step 2 scope contract re-routes.
At **project/user scope** it derives the counterpart and runs the existing pair-based checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
If a validation script fails or cannot run — Bash denied, `python3` or `vale` absent, `references/field-inventory.md` missing — read `references/validation-scripts.md`; what these scripts measure is not reproducible by reading.
At **plugin/APM scope** there is no counterpart — the script instead checks the single file's frontmatter against the `apm-agent-allowlist` in `references/field-inventory.md`. Read that section for the current list rather than reciting one here; it is the authoritative source and it changes. As of 2026-08-14 it is `name`, `description`, `model`, `source_keys`, `disallowedTools` — `source_keys` is provenance metadata, not a provider-specific field, and is validated separately by `validate-provenance.sh` against `sources.md`; `disallowedTools` is a denylist, admitted because denying a tool by name is safe under `apm compile`'s verbatim copy in a way the `tools` allowlist is not (ADR-0016's 2026-08-14 amendment, and the rationale recorded alongside the list itself). Findings about missing fields, bad name format, name/filename-stem mismatch, empty body, or missing frontmatter → `### Structure`, same as project/user scope. Findings about any field outside the allowlist (e.g. `tools`, or any Claude-only/Copilot-only field carried over from a hand-edit) and body length → `### Provider safety` — but the dimension's meaning shifts here: it is no longer a CC-vs-Copilot field-leakage check, it's a vendor-neutral-field-allowlist check, since `apm compile` verbatim-copies this file's frontmatter to every target and there is no per-target integrator to reconcile a CC-only or Copilot-only field (ADR-0016). `### Pair consistency` never applies at this scope — the script never emits a missing-counterpart FAIL here, because there is nothing to pair by design.
`validate-provenance.sh` prints nothing on success and runs at plugin/APM scope only, exiting 0 silently elsewhere. Its FAIL findings become a separate `### Provenance` dimension, and it emits Why and Fix itself — surface those verbatim.
`vale-wrap.sh` ships inside this skill's own `scripts/` — resolve it relative to this skill's directory the same way `scripts/validate.sh` is resolved above, so the invocation works whether this skill is running from this repo or from an installed plugin cache. Pass no `--config`: handed none, the wrapper loads its own sibling `assets/vale/.vale.ini`, located from the script's path rather than from the cwd. Adding an explicit relative `--config` breaks exactly the case the self-location covers — a resolved script path plus an unresolved config path yields `E100 Runtime error ... does not exist`, exit 2, which the fallback below then misreads as "vale unavailable". At project/user scope, run it against both files of the pair (not just the one passed in); at plugin/APM scope, run it against the single file. `Kyberforge` applies to all of these files via the `**/agents/*.md` glob; `KyberforgeCopilot` applies to any `*.agent.md` file — including the plugin/APM-scope file, which already has that extension — via the `**/*.agent.md` glob, since its one rule (`Use proactively`) flags CC-specific phrasing that's meaningless in a vendor-neutral or Copilot description. Every Vale alert is a `FAIL` — all rules are graded `error` — so report each one in the `### Description` / `### Body` dimensions citing its rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). Skip and fall back to Step 2 judgment if the `vale` binary is unavailable. If Vale reports `0 files` scanned, treat the pass as NOT RUN — not as clean — and fall back to full Step 2 judgment for the dimensions it would have covered.
`vale-wrap.sh` applies the bundled `Kyberforge` style as a prefilter. Pass no `--config`; the wrapper locates its own. At project/user scope pass both files of the pair, not only the one you were handed. Every rule is graded `error`, so every alert is a FAIL. Report each one citing its rule ID, filed under the dimension it belongs to, and do not re-derive it by judgment:
`validate-provenance.sh` operates at plugin/APM scope only — it walks up from the agent file's directory the same way `validate.sh` does (nearest ancestor `apm.yml` with a top-level `type:` field; skip a `type:`-less marketplace-only `apm.yml`; stop at `.git` or the filesystem root) and exits 0 silently if that walk doesn't land on a package root, or when no provenance data exists. When it does apply, it validates the chain between the single file's own `source_keys` and the package-scoped `sources.md` (package root — see ADR-0010). Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
| Rule | Dimension |
|---|---|
| `Kyberforge.DescriptionOpener`, `Kyberforge.CompositionNote`, `Kyberforge.VagueWording`, `KyberforgeCopilot.ProactivePhrase` | description |
| `Kyberforge.SentenceOpenerThereIs`, `Kyberforge.PaddingPhrase` | body |
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually. At project/user scope: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists). At plugin/APM scope: required fields present (`name`, `description`, non-empty body), `name` is kebab-case and matches the filename stem, no `FILL IN:` placeholders, no frontmatter field outside the allowlist — read the `apm-agent-allowlist` section of `references/field-inventory.md` for it, do not work from memory (`source_keys` carries provenance metadata, checked separately by `validate-provenance.sh` against `sources.md`).
## Step 2 — Read the agent and load its scope contract
## Step 2 — Qualitative checks
Read the agent file end to end, and at project/user scope its counterpart too. A path containing `.apm/agents/` is plugin/APM scope; anything else is project or user scope. Each contract names the dimensions that apply there and where `validate.sh` findings other than Structure belong:
Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3.
| Scope | Read |
|---|---|
| plugin/APM | `references/scope-plugin-apm.md` |
| project, user | `references/scope-project-user.md` |
**Description (both files):**
- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent. Vale's `Kyberforge.DescriptionOpener` alert flags the specific known-bad "This agent..." opener directly; verifying an arbitrary opening word is genuinely a strong verb still requires judgment.
- Specificity: is the trigger condition stated precisely? — SUGGESTION if vague. Vale's `Kyberforge.VagueWording` alert covers known filler ("helps with", "utilize", ...) directly; report those as FAILs without re-deriving by judgment.
- `Use proactively` in a Copilot description: Vale's `KyberforgeCopilot.ProactivePhrase` alert (Copilot file only) flags this directly — report it without re-deriving by judgment.
## Step 3 — Qualitative audit
If a description finding is borderline, read `references/description-quality.md`.
Load a dimension's rubric before judging that dimension.
**Body:**
- Direct role instruction: system prompt opens with `You are a [role]. When invoked, [action].` — SUGGESTION if absent
- One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded
- Generic, non-specific reference pointers to the `references/` directory: Vale's `Kyberforge.PaddingPhrase` alert flags this directly — report it without re-deriving by judgment
- Sentences that open with "There is"/"There are": Vale's `Kyberforge.SentenceOpenerThereIs` alert flags this directly — report it without re-deriving by judgment
| Dimension | Read |
|---|---|
| description | `references/description-quality.md` |
| body, delegation, comment-discipline | `references/body-and-delegation.md` |
**Body/Frontmatter comments:**
- Inspect each comment block in the YAML frontmatter. For each comment, apply: *"Would the agent get this wrong without this comment?"* Flag any that answer "no" as padding.
- Look for patterns like `# Optional. <long explanation>` or extensive inline guidance (more than 1–2 lines per field) that should be condensed or removed before shipping.
- This mirrors skill-audit's body-discipline check but applies to template documentation in the frontmatter — template guidance belongs in development; agent-ready files should have minimal comments.
Cite file and line number for every finding.
**Pair consistency (cross-file) — project/user scope only:**
- Both files exist — FAIL if counterpart is missing (kyberforge project convention; not a platform requirement from either CC or Copilot — label as such)
- The following checks are covered automatically by `validate.sh`; apply them manually only when the script cannot run: both system prompt bodies non-empty — FAIL if either is empty
- **Does not apply at plugin/APM scope** — there is only one file, by design; do not raise a Pair Consistency finding there under any circumstance.
## Step 4 — Report
**Unexpressable Claude-only behavior — plugin/APM scope only:**
- Read the description and body. If either implies a need the vendor-neutral frontmatter can no longer express — tool restriction, `isolation`, `memory`, or another Claude-only behavior that a hand-authored CC file could have declared — flag it as a SUGGESTION, never a FAIL. This is a known upstream schema limitation (APM's agent primitive has no per-target compile integrator, so `tools:`/`isolation`/etc. can't be emitted safely to both CC and Copilot — ADR-0016), not an authoring mistake. The finding exists to give the author visibility into the gap, not to imply the schema can be made to do something it can't.
- Example: a body that says "only use Read and Grep, never Edit" but the frontmatter has no `tools` field to enforce it — SUGGESTION, not FAIL.
## Step 3 — Report
Open with a coverage line. At project/user scope:
Open with a coverage line naming every dimension checked. At project/user scope:
```text
Checked: structure · provider-safety · description · body · comment-discipline · pair-consistency · provenance
Checked: structure · provider-safety · description · body · delegation · comment-discipline · pair-consistency · provenance
```
At plugin/APM scope, omit `pair-consistency` — it does not apply when there is no pair:
At plugin/APM scope, drop `pair-consistency` — there is no pair to check.
```text
Checked: structure · provider-safety · description · body · comment-discipline · provenance
```
Then output only the dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each. Omit clean dimensions — their absence is what confirms they passed.
Then output only dimensions that have findings, grouped under H3 headings, FAILs before SUGGESTIONs within each dimension. Omit clean dimensions entirely. `### Provenance` findings are sourced verbatim from `validate-provenance.sh` output — copy them without rephrasing.
For each finding:
Each finding:
```text
FAIL/SUGGESTION <finding> — file:line
@@ -107,18 +86,4 @@ FAIL/SUGGESTION <finding> — file:line
Fix: <exact change — quote before/after where applicable>
```
Close with:
```text
## Result
PASS
PASS · P info
PASS (N suggestions)
PASS (N suggestions) · P info
FAIL (N fails · M suggestions)
FAIL (N fails · M suggestions) · P info
Run /agent-author to address findings.
```
Omit `Run /agent-author to address findings.` when there are no findings at all. Do not apply fixes — report and propose only.
Close with a `## Result` block holding one line: `PASS`, `PASS (N suggestions)`, or `FAIL (N fails · M suggestions)`, each optionally followed by ` · P info`. INFO findings are observational and never change PASS/FAIL; omit `· P info` when there are none. Add a second line, `Run agent-author to address findings.`, whenever there is at least one finding. Do not apply fixes — report and propose only.