Files
holocron/plugins/kyberforge/.apm/skills/agent-audit/references/validation-scripts.md
Claude Code AI - Gitea MCP 598a7c326a refactor(skills): retrofit the corpus to the ADR-0020 context contract (#129)
Retrofits all 39 skills to ADR-0020's description/body context contract, then fixes what six rounds of independent review found in that retrofit — including four ways the hot gate itself failed open.

Closes #99, #107, #108, #110, #111, #114, #115, #120.

## The retrofit (waves 1-5)

| | Start | Now |
|---|---|---|
| Description FAILs (>400 chars) | 26 | **0** |
| Body FAILs (>900 words, body-only) | 9 | **0** |
| Dangling routing targets | 2 | **0** |
| `Kyberforge.CompositionNote` | 10 | **0** |
| Preload tax | 21,005 chars | **~10,500** |

Under the 12,000-char success criterion. Per-wave detail is on #99.

## The review fixes

**The gate failed open four ways, three of them found after the retrofit shipped.** An unrecognised follower token made a dangling target vanish. A skill directory with no `SKILL.md` resolved as a valid target, so a commit could be green locally and red in a fresh clone — three existing fixtures were relying on that, one of which made the install-leak A/B pass vacuously. Then the free-standing `/name` sweep turned out to be gated on the sentence carrying a boundary marker, so route notation in any other sentence was invisible — not an ERROR, not a SUGGESTION, not an INFO — which left the documented "`/name` always blocks" promise false from a second direction. All four fixed and pinned.

**Two checks were silently not running.** `validate-provenance.sh` checks 7-8 were dead across nine skills. Waking them exposed a deeper problem: they assume `Research doc:` names a source index, but 30 of 121 entries point at topic content documents, so every new check-7 INFO was a false positive and check 8 was saved from a false-FAIL flood only by an *unannounced* skip. Checks 7/8 are now scoped to source indexes and every skip announces itself (#121).

**The retrofit's own anti-goal, four times.** ADR-0020 warns that a blunt gate gets satisfied by deleting content rather than relocating it. `diagnose` and `skill-audit` relocated prose and then read it unconditionally; `prototype` and `vale-config` deleted rules outright that survived nowhere. All four addressed.

## Verification

- `bash tests/run-tests.sh --strict` — 24 suites, 0 skipped, 0 failed
- `bash tests/run-bats.sh` — 325 tests, 0 failures
- `pre-commit run --all-files` — 17/17
- `pre-commit run --hook-stage pre-push --all-files` — 16/16, with `apm marketplace check` and `apm pack --check-clean` run against the remote, not skipped
- `scripts/skill-size-check.sh` over all 39 skills — rc 0, 0 ERROR/FAIL, SUGGESTION-only
- Preload tax measured at **10,498 chars**, max description 390 — both inside budget
- Every new test proven non-vacuous by a deliberate mutation of the behaviour it covers

**Per-commit sync, stated accurately:** the ten commits from the latest review round each pass `check-plugin-content-sync` in isolation, verified by checking each out in a detached worktree with a clean between. The earlier gitea window (`dfacf05..bedbd1d`, nine commits) does **not** — its mirror was regenerated in one batch at `bbc7300`. An earlier revision of this description claimed the property held for every commit; it does not, and a bisect through that window lands on a red commit. **Squash-merge** to collapse it, or accept that this range is not bisectable.

## Version bump

Six plugins and the catalog take a **patch**, not a minor. The branch is **89 commits — 40 `fix` / 30 `refactor` / 12 `docs` / 5 `chore` / 2 `test` — zero `feat`, zero `!`, zero `BREAKING CHANGE`** — and adds no skill, agent, command or hook. (Two earlier revisions of this section cited a stale histogram, most recently 78 commits; the figures above are measured at HEAD.) Both rules this repo ships (`forge/references/version-bump.md`, landing in this PR, and `git-commits/references/conventional-commits-spec.md`) make that a patch, and the catalog set is unchanged at 7 entries.

Not settled by that: four published files were removed from the installed tree, three moved, and `caveman` gained `disable-model-invocation`, retiring its old triggers. Under a strict reading those are major-class and currently ship under `refactor:` with no marker. Whether the deployed skill surface is a public contract is written down nowhere — worth deciding, but it outlives this PR.

## Deliberately not in scope

#112 (cherry-pick ownership, now resolved in favour of `git-commits`), #113 (`rtk git` normalisation), #116 (research fan-out), #101 (audit-skill merge), #122 (non-spec skill-root files), #123 (no PRD producer) stay open. #117 is the one worth reading: the contract's remedy is to move prose into `references/`, which is exactly where neither the size gate nor Vale looks — and the blind spot is wider than #117 currently records, since there is no root `.vale.ini` at all, so every ADR, `CONTEXT.md` and `README.md` is unlinted too.

That blind spot let this branch carry two `level: error` `Kyberforge.SentenceOpenerThereIs` violations into `references/` files it created — `provider-adapter-author/references/provider-matrix.md:31` and `agent-audit/references/finding-criteria.md:95`. Both are reworded in `afadaae`, confirmed by routing each file through the audit's own `vale-wrap.sh` (1 error each before, 0 after). Five further occurrences sit in `references/` files already on `main`; those are the pre-existing corpus and stay with #117, which is the real fix.

Also unfixed and not this PR's: `apm install` appends a duplicate `SessionStart` entry to `.claude/settings.json`, so a fresh clone cannot get pre-push green without an edit AGENTS.md warns against. Reproduces identically on `main`.

Co-authored-by: Defame1297 <gitea@rkdr.net>
Reviewed-on: https://git.dev.rkdr.net/Defame1297/holocron/pulls/129
Co-authored-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net>
Co-committed-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net>
2026-09-01 13:47:46 +00:00

4.1 KiB

source_keys
source_keys
claude-code-plugins-docs
claude-code-subagents-docs
github-custom-agents-configuration

Validation Scripts Reference

Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting. Nothing here is needed on a clean run.

Report the gap, do not guess

If a script cannot run at all — Bash denied, python3 unavailable, vale not installed — say so as an INFO finding naming the script and the missing dependency, then fall back to the manual checks below. An INFO never changes PASS/FAIL. Silently omitting the dimension a script would have covered reports a clean audit that checked less than it claims to have checked.

How the scripts detect scope

validate.sh and validate-provenance.sh walk up from the agent file's directory and stop at the first of these:

  1. An apm.yml carrying a top-level type: instructions|skill|hybrid|prompts line — plugin/APM scope, and that directory is the package root. An apm.yml with no type: is a marketplace-only manifest: skip it and keep walking.
  2. $HOME — user scope, checked before .git so a dotfiles-managed home directory that is its own repo cannot shadow it.
  3. A .git directory or file — project scope.
  4. The filesystem root — project scope.

plugin.json and .claude-plugin/plugin.json are not scope signals. A directory holding only a plugin.json and no apm.yml falls through to project or user scope.

validate-provenance.sh exits 0 silently when that walk does not land on a package root, and again when the package has no provenance data. Check the exit code before you believe the silence:

  • 0 — a pass, not a skip you need to investigate. Both silent cases above land here.
  • 1 — real findings, on stdout with Why and Fix.
  • 2 — the check never ran. A missing, doubled, non-file or wrongly-named argument, an undecodable apm.yml, or an absent python3, each with a diagnostic on stderr and no findings at all. Report the ### Provenance dimension as unverified and quote the reason. An exit 2 is never a clean pass: empty stdout there means nothing was checked, not that nothing was wrong.

Manual fallback

Every scope: required fields present (name, description, non-empty body); name is kebab-case; no FILL IN: placeholders in the description or body; the description at or under 400 characters measured on the folded YAML value.

Plugin/APM scope: name matches the filename stem; no HTML comments left in the frontmatter; no frontmatter key outside the apm-agent-allowlist section of references/field-inventory.md — open that file, do not work from memory.

Project/user scope: the counterpart file exists; name matches the filename stem in the Copilot .agent.md only (Claude Code files are exempt); no key from claude-code-only-fields in the Copilot file and none from copilot-only-fields in the CC file, both read from references/field-inventory.md.

Script-specific failures

  • Error: field-inventory.md not found (exit 2). validate.sh reads its field lists from references/field-inventory.md at load time and refuses to run without it, rather than falling back to a hardcoded list that could disagree with the file (ADR-0009). Restore the file; do not work around it.
  • vale reports 0 files. Treat the pass as NOT RUN, not as clean, and fall back to full Step 3 judgment for the dimensions it would have covered. The Kyberforge style is scoped to **/agents/*.md and **/*.agent.md, and KyberforgeCopilot to **/*.agent.md alone — a file outside those globs is silently not linted.
  • E100 Runtime error ... does not exist (exit 2) from vale-wrap.sh. An explicit relative --config was passed. Pass none: the wrapper locates its own assets/vale/.vale.ini from its own path. Do not read this exit code as vale being unavailable.
  • A path argument that does not exist is a hard error in vale-wrap.sh, deliberately: bare vale would fall back to reading stdin and print a clean-looking 0 errors ... in stdin, which the 0 files guard above does not catch.