Defame1297
88866b81a3
The clean provenance bill was an artifact. Checks 7 and 8 assume `Research doc:` names a source index whose H2s are slugs, but 30 of 121 corpus entries point at topic content documents whose H2s are topics. Those 30 produced every new check-7 INFO — all false positives. Check 8 aimed at the same documents, which carry no `Status:` line at all, would have emitted a large false-FAIL flood; the only thing preventing it was an unannounced `rd_status != extracted` skip. So "0 new FAILs" rested on exactly the fail-open class this branch exists to remove, and naively fixing the skip would have turned the branch red. Checks 7/8 now run only when the research doc's basename is `sources.md`, and every other case emits a visible INFO naming the slug. The dangling-path INFO stays ahead of the basename gate, because a path that does not resolve is rot whatever it is named. `parse_status` accepts the bullet form and a trailing note after the backticked value, so a status it cannot read no longer reads as "nothing to check". Corpus: 36 INFOs of which 30 were false, to 56 of which none are. FAIL stays 0, and no Status line flipped to `extracted` under the new parser, so no FAIL was suppressed by luck. Also closed, each a silent pass: a nonexistent directory, a directory with no SKILL.md, and extra arguments now exit 2; a UTF-8 BOM no longer defeats frontmatter parsing; the bare `except Exception: return False` that turned an unreadable file into a clean pass is gone, with all reads pinned to UTF-8; check 3 walks nested `references/` subdirectories; `FILL IN:` at end of line no longer escapes checks 1 and 6; duplicate `## slug` blocks and repeated `Research doc:` lines are announced rather than half-read. The agent-audit copy carried all of the above unfixed and is now ported, minus the four fixes that are genuinely N/A at agent scope — it reads a plugin-root `sources.md` and has no checks 7/8 and no `references/` tree. Its silent exit 0 for a file outside plugin scope is preserved deliberately: that is a verdict about a valid file, not a skip, and `check-scope-walkup-sync.sh` pins it. Every exit-2 gate therefore decides from the argument alone, before the walk-up runs. `validation-scripts.md` said flatly that silence from the validator is a pass, not a skip. That sentence is what made a typo'd path dangerous, and both copies are corrected here. The matching SKILL.md exit-code guidance lands with the audit rubric change, which touches the same files. Tests: skill-audit 45 to 65, agent-audit 24 to 43, every new case proven by mutation. Refs: #111, #118, #121
77 lines
4.1 KiB
Markdown
77 lines
4.1 KiB
Markdown
---
|
|
source_keys:
|
|
- claude-code-plugins-docs
|
|
- claude-code-subagents-docs
|
|
- github-custom-agents-configuration
|
|
---
|
|
|
|
# Validation Scripts Reference
|
|
|
|
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting.
|
|
Nothing here is needed on a clean run.
|
|
|
|
## Report the gap, do not guess
|
|
|
|
If a script cannot run at all — Bash denied, `python3` unavailable, `vale` not installed — say so
|
|
as an **INFO** finding naming the script and the missing dependency, then fall back to the manual
|
|
checks below. An INFO never changes PASS/FAIL. Silently omitting the dimension a script would have
|
|
covered reports a clean audit that checked less than it claims to have checked.
|
|
|
|
## How the scripts detect scope
|
|
|
|
`validate.sh` and `validate-provenance.sh` walk up from the agent file's directory and stop at the
|
|
first of these:
|
|
|
|
1. An `apm.yml` carrying a top-level `type: instructions|skill|hybrid|prompts` line — **plugin/APM
|
|
scope**, and that directory is the package root. An `apm.yml` with no `type:` is a
|
|
marketplace-only manifest: skip it and keep walking.
|
|
2. `$HOME` — **user scope**, checked before `.git` so a dotfiles-managed home directory that is its
|
|
own repo cannot shadow it.
|
|
3. A `.git` directory or file — **project scope**.
|
|
4. The filesystem root — **project scope**.
|
|
|
|
`plugin.json` and `.claude-plugin/plugin.json` are not scope signals. A directory holding only a
|
|
`plugin.json` and no `apm.yml` falls through to project or user scope.
|
|
|
|
`validate-provenance.sh` exits 0 silently when that walk does not land on a package root, and again
|
|
when the package has no provenance data. Check the exit code before you believe the silence:
|
|
|
|
- **0** — a pass, not a skip you need to investigate. Both silent cases above land here.
|
|
- **1** — real findings, on stdout with Why and Fix.
|
|
- **2** — the check never ran. A missing, doubled, non-file or wrongly-named argument, an
|
|
undecodable `apm.yml`, or an absent `python3`, each with a diagnostic on stderr and no findings
|
|
at all. Report the `### Provenance` dimension as unverified and quote the reason. An exit 2 is
|
|
never a clean pass: empty stdout there means nothing was checked, not that nothing was wrong.
|
|
|
|
## Manual fallback
|
|
|
|
**Every scope:** required fields present (`name`, `description`, non-empty body); `name` is
|
|
kebab-case; no `FILL IN:` placeholders in the description or body; the description at or under 400
|
|
characters measured on the folded YAML value.
|
|
|
|
**Plugin/APM scope:** `name` matches the filename stem; no HTML comments left in the frontmatter;
|
|
no frontmatter key outside the `apm-agent-allowlist` section of `references/field-inventory.md` —
|
|
open that file, do not work from memory.
|
|
|
|
**Project/user scope:** the counterpart file exists; `name` matches the filename stem in the
|
|
Copilot `.agent.md` only (Claude Code files are exempt); no key from `claude-code-only-fields` in
|
|
the Copilot file and none from `copilot-only-fields` in the CC file, both read from
|
|
`references/field-inventory.md`.
|
|
|
|
## Script-specific failures
|
|
|
|
- **`Error: field-inventory.md not found` (exit 2).** `validate.sh` reads its field lists from
|
|
`references/field-inventory.md` at load time and refuses to run without it, rather than falling
|
|
back to a hardcoded list that could disagree with the file (ADR-0009). Restore the file; do not
|
|
work around it.
|
|
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
|
|
Step 3 judgment for the dimensions it would have covered. The `Kyberforge` style is scoped to
|
|
`**/agents/*.md` and `**/*.agent.md`, and `KyberforgeCopilot` to `**/*.agent.md` alone — a file
|
|
outside those globs is silently not linted.
|
|
- **`E100 Runtime error ... does not exist` (exit 2) from `vale-wrap.sh`.** An explicit relative
|
|
`--config` was passed. Pass none: the wrapper locates its own `assets/vale/.vale.ini` from its
|
|
own path. Do not read this exit code as vale being unavailable.
|
|
- **A path argument that does not exist is a hard error** in `vale-wrap.sh`, deliberately: bare
|
|
`vale` would fall back to reading stdin and print a clean-looking `0 errors ... in stdin`, which
|
|
the `0 files` guard above does not catch.
|