Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

fix(skill-audit): make check 9 reachable, wrap-safe and never silently skipped

Browse Source 
Check 9 shipped in #130 to close #118, but three defects meant it could not do the job it was
added for.

Why:
- It is INFO-only, so it always exits 0 — and SKILL.md graded exit 0 "a genuine pass" and said the
  script "prints nothing on success". Every check-9 INFO was discarded before it reached a report,
  behind three further doors that only opened on a non-zero exit.
- `parse_field_raw()` matched `(.+)`, which does not span newlines, so only the first physical line
  of a wrapped value was compared. Rewriting only the continuation line of a wrapped Description
  from a hedge to a confident claim produced no finding at all — verbatim the regression #118 was
  filed about. The bullet branch had the same shape: a wrapped bullet broke the loop and dropped
  every later entry.
- A `git show` failure at the base ref was treated as "creation, nothing to flag" and skipped the
  whole skill with no output, collapsing "absent at that ref" with "not tracked under that name".
  A gitignored `.claude/skills/` copy reported clean while the authoring path reported four changed
  claims. The script's own usage text promises this is "never a silent skip".

Implementation notes:
- Exit-code guidance re-keyed on output as well as code: 0-and-silent passes, 0-with-output is
  INFO-only findings, 1 is FAILs, 2 never ran.
- `parse_field_raw()` is line-based and joins continuation lines; `normalize_field_text()`'s
  docstring is now true rather than aspirational. A reorder deliberately fires: the two fields share
  one parser, and order-insensitivity would mean splitting a prose Description on commas.
- The discarded `show_err` is now surfaced as one whole-check INFO naming both readings.
- `--base-ref=` given empty now beats the env var, as the usage text always claimed.

`validate.sh` gains an ADR-0022 `metadata.version` check at FAIL tier, because any lower tier lets
skill-author Step 4 report done on a file the commit gate then refuses. Its `read` heuristic now
skips here-doc bodies — reflowing the one offending line would have cleared the finding and left
the cause, since every usage() heredoc is one wrap from putting the English verb in column 0.

Impact: provenance tests 73 -> 82, validate tests 64 -> 72. Test 72 previously deleted origin/main
before asserting the override, so it proved the flag works with no default rather than that it beats
one; it now moves origin/main forward first.

Refs: #118
ADR: 0022
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeH8SCbcrCAQrtymkNuhKP
This commit is contained in:
Defame1297
2026-09-09 05:15:05 +00:00
parent ed8c99efbd
commit 175ea89c0a
8 changed files with 704 additions and 65 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
plugins/kyberforge/.apm/skills/skill-audit/README.md
View File

@@ -11,7 +11,7 @@ Audit a skill directory against the agentskills.io specification and the house c
`validate.sh` enforces two independent length families that must not be conflated: the agentskills.io spec conformance ceilings (500 lines, 2,770 words, both counting the whole file) and the ADR-0020 context budget (250/400 description characters, 600/900 body-only words).
Alongside those it runs four shape checks that are not length measurements at all. Two are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent, and every `references/<file>.md` the body names must exist on disk. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
Alongside those it runs shape checks that are not length measurements at all. Three are FAILs: every routing target named in the description — in the compressed `Not <thing> -> <name>` arrow **and** in the prose form — must resolve to a real skill or agent; every `references/<file>.md` the body names must exist on disk; and `metadata.version` must be present and three-part semver (ADR-0022). That last one is FAIL rather than SUGGESTION because the `skill-frontmatter` pre-commit hook rejects the file without it — an audit grading it lower would report ready-to-ship on a file the commit gate refuses. Three are SUGGESTIONs: a missing boundary clause, a Gotchas section over five entries, and a Gotchas section over 25% of the body. The resolution universe for boundary targets is derived by walking up from the audited `SKILL.md` — the authoring root above it, its own apm package, and that package's declared `apm.yml` dependencies — so a fresh clone and a machine that has run `apm install` return the same verdict. When no universe can be determined the check prints `INFO ... DID NOT RUN` and does not silently pass.
## Usage
@@ -26,8 +26,8 @@ Provide the path to the skill directory to audit when invoking.
| File | Purpose |
|------|---------|
| `SKILL.md` | Skill instructions for agents |
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, and upstream research doc alignment |
| `scripts/validate.sh` | Structural validator — checks name format, name matches directory, description presence and length, `metadata.version` presence and semver shape (ADR-0022), body-only word count, line and whole-file word ceilings, boundary-clause presence, boundary-target resolution, `references/` pointer existence, Gotchas entry count and body share, placeholder detection, script executable bit, and interactive-prompt detection |
| `scripts/validate-provenance.sh` | Provenance validator — checks sources.md completeness, source_keys/slug consistency, Contributing files existence, bidirectional linkage, Research doc: fields, upstream research doc alignment, and (check 9, INFO only) whether a slug's `Description` or `Contributing files` text has changed since a base ref — `--base-ref=<ref>` or `VALIDATE_PROVENANCE_BASE_REF`, defaulting to the merge base with `origin/main` |
| `scripts/vale-wrap.sh` | Vale prefilter wrapper — runs the bundled `Kyberforge` Vale styles against SKILL.md and reports alerts as deterministic FAILs ahead of Step 3's qualitative review |
| `assets/vale/.vale.ini` | Vale configuration — points Vale at the bundled `Kyberforge` style path, self-located relative to `vale-wrap.sh` |
| `assets/vale/styles/Kyberforge/CompositionNote.yml` | Vale rule — flags composition and architecture notes in a description (e.g. "cross-cutting", "entry point", "rather than duplicating") |
@@ -41,7 +41,7 @@ Provide the path to the skill directory to audit when invoking.
| `references/patterns.md` | Rubric for the patterns dimension — which instruction construct fits which job, and how each is correctly formed |
| `references/file-structure.md` | Rubric for the file-structure and internal-consistency dimensions — permitted directories, cross-plugin path rules and their two structural exemptions, README drift |
| `references/formatting-and-scripts.md` | Rubric for the formatting and scripts dimensions — heading and fencing conventions, and the agentic-use criteria for bundled scripts |
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded only on a script failure) |
| `references/validation-scripts.md` | Step 1 troubleshooting — the manual structural fallback when `validate.sh` cannot run, and the script exit codes that are easy to misread (loaded on a script failure, and on any exit-0 run that printed something — `validate-provenance.sh`'s check 9 is INFO-only, so its findings arrive that way) |
| `references/sources.md` | Provenance record — agentskills.io sources that informed this skill and which files each contributed to |
| `tests/validate.bats` | (source-only) Bats test suite for validate.sh |
| `tests/validate-provenance.bats` | (source-only) Bats test suite for validate-provenance.sh |

6
plugins/kyberforge/.apm/skills/skill-audit/SKILL.md
View File

@@ -7,7 +7,7 @@ description: >
skill-author.
allowed-tools: Bash Read
metadata:
version: "1.0.0"
version: "1.0.1"
category: factory
source_keys:
- agentskills-home
@@ -36,9 +36,9 @@ bash scripts/vale-wrap.sh <skill-dir>/SKILL.md
`validate.sh` findings become the `### Structure` dimension — its FAILs and its SUGGESTIONs both, at the tier the script assigned. Report each once; never re-grade one under another dimension. Unresolved boundary targets are where this bites, because their tier turns on notation.
If any of the three cannot run, or exits non-zero for a reason other than findings, read `references/validation-scripts.md` — it carries the manual fallback and the misleading exit codes. Ordinary content FAILs are the expected outcome here and need no fallback.
Read `references/validation-scripts.md` when any of the three cannot run or exits non-zero for a reason other than findings, **and whenever `validate-provenance.sh` exits 0 having printed anything**. Ordinary content FAILs are the expected outcome here and need no fallback.
`validate-provenance.sh` prints nothing on success, so read its exit code before you read its silence. **0** is a genuine pass. **1** means real findings: its FAILs and INFOs become a separate `### Provenance` dimension, and it emits Why and Fix itself — surface those verbatim. **2** means the check never ran — a usage or environment error, reason on stderr, no findings and often no stdout at all. On a 2, report `### Provenance` as unverified and quote the stderr reason. Never grade an exit 2 as a clean pass: empty stdout there means nothing was checked, not that nothing was wrong.
`validate-provenance.sh` reports through exit code **and** output; neither alone is the verdict. **0, silent** is a genuine pass. **0 with output** is INFO-only findings — still a `### Provenance` dimension; `references/validation-scripts.md` says what each obliges — for a check-9 INFO, reading rather than relaying. **1** is FAILs plus any INFOs; it emits Why and Fix itself — surface those verbatim. **2** means it never ran — a usage or environment error, reason on stderr, often no stdout — so report `### Provenance` unverified and quote that reason. Never grade an exit 2, or an exit 0 that printed, as a clean pass.
`vale-wrap.sh` applies the bundled `Kyberforge` style as a prefilter. Pass no `--config`; the wrapper locates its own. Every rule is graded `error`, so every alert is a FAIL. Report each one citing its rule ID, filed under the dimension it belongs to, and do not re-derive it by judgment:

38
plugins/kyberforge/.apm/skills/skill-audit/references/validation-scripts.md
View File

@@ -6,8 +6,9 @@ source_keys:
# Validation Scripts Reference
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting.
Nothing here is needed on a clean run.
Read this when a Step 1 script fails, cannot run, or reports something that needs interpreting —
including `validate-provenance.sh` exiting **0 having printed something**, which is INFO findings,
not a clean run. Its silent exit 0 is the only outcome that needs nothing here.
## Report the gap, do not guess
@@ -106,18 +107,33 @@ Three ways to read the result wrong:
`python3` all exit **2** with a message on stderr. Exit 2 means the script never ran — report it
as an unaudited dimension, never as a pass and never as a finding. Exit 1 is findings.
- **A check-9 INFO — `'<field>' changed for '<slug>' since <ref>` — means go read, not just relay.**
Check 9 diffs the current `references/sources.md` against a base ref (default: the merge base with
`origin/main`) and flags a slug whose `Description` or `Contributing files` text differs. It is
structurally incapable of telling you whether the new wording is still *true* — it only detects
that the text changed — so when this INFO fires, open the Contributing files it names and the
document named in that slug's `Research doc:` field, and confirm by reading whether the (possibly
Check 9 diffs the current `references/sources.md` against a base ref and flags a slug whose
`Description` or `Contributing files` text differs. It is structurally incapable of telling you
whether the new wording is still *true* — it only detects that the text changed — so when this
INFO fires, open that slug's own entry: the document named in its `Research doc:` field, and the
files its `Contributing files` list names. Read whichever the changed field is a claim *about* —
a Description-only change often leaves the file list untouched, so "open the Contributing files"
is where to look, not proof that they are what moved. Confirm by reading whether the (possibly
strengthened) claim genuinely holds. This is the one provenance finding this script cannot verify
for you: every other check here is a structural fact you can relay as-is, but check 9's job is
only to tell you *where* to spend that reading effort, not to replace it. Acknowledging the INFO
without opening those files is not auditing it. A single INFO naming "no base ref could be
resolved" or "no repo root above the skill directory" is the same graceful-skip pattern as every
other check here that cannot run — treat it as an unaudited dimension for that reason, not as a
finding about the skill.
without opening those files is not auditing it. Its companion — `'<field>' removed for '<slug>'
since <ref>` — is the same obligation in the other direction: a claim withdrawn rather than
rewritten. No other check here requires the field, so confirm the removal was deliberate.
- **The check-9 base ref defaults to `git merge-base HEAD origin/main`, and there are two ways to
override it.** `--base-ref=<ref>` on the command line, or the `VALIDATE_PROVENANCE_BASE_REF`
environment variable; the flag wins when both are given, including when it is given empty
(`--base-ref=`), which selects the default resolution and ignores the environment. Reach for one
on a fork, a long-lived branch, or a mirror whose remote is not called `origin` — and when a
review asks what changed since a specific commit rather than since the branch point.
- **A single check-9 INFO naming a whole-check skip is an unaudited dimension, not a finding about
the skill.** There are three: "no repo root above the skill directory", "no base ref could be
resolved", and "`<path>` is not tracked at `<ref>`". The third is the one to read carefully — it
fires when the base ref resolved but `git show <ref>:<path>` did not, which covers both a
genuinely new `sources.md` (nothing to flag) and a path git does not know under that name: a
renamed skill directory, or an installed, gitignored copy such as a deployed `.claude/skills/`
tree. Auditing the deployed copy silently checks nothing; re-run against the authoring path under
`plugins/*/.apm/skills/`.
- **`vale` reports `0 files`.** Treat the pass as NOT RUN, not as clean, and fall back to full
Step 3 judgment for the dimensions it would have covered. The bundled `Kyberforge` style is
scoped by glob in `assets/vale/.vale.ini`; a file outside those globs is silently not linted.

214
plugins/kyberforge/.apm/skills/skill-audit/scripts/validate-provenance.sh
View File

@@ -15,7 +15,8 @@ Arguments:
a long-lived branch, a mirror with a different remote name).
The VALIDATE_PROVENANCE_BASE_REF environment variable is an
equivalent, lower-precedence way to set it — the flag wins
if both are given.
if both are given, including when the flag is given empty
(\`--base-ref=\`), which selects the default resolution.
Exit codes:
0 All checks passed (or nothing to validate)
@@ -47,10 +48,13 @@ Checks performed:
8 Extracted non-(none) slug in research doc present in sources.md
9 Description or Contributing files text changed since --base-ref (INFO
only — a bash script cannot verify the claim is still TRUE, only that it
changed; the auditor reads the named files to check that). A slug absent
at the base ref is a creation, not a change, and is not flagged. When the
base ref cannot be resolved at all, this is announced as ONE INFO for the
whole check, never a silent skip.
changed; the auditor reads the named files to check that). Wrapped values
are joined before comparison, so a re-wrap alone is not a change and a
rewrite of any line of one is. A slug absent at the base ref is a
creation, not a change, and is not flagged; a field that WAS there and is
now gone is announced as a removal. When the base ref cannot be resolved,
or references/sources.md is not tracked under this path at that ref, this
is announced as ONE INFO for the whole check, never a silent skip.
Checks 7 and 8 apply ONLY when the Research doc value names a research SOURCE
INDEX — a file whose basename is sources.md, whose H2 headings ARE source
@@ -70,8 +74,15 @@ fi
# never counts against them — a caller passing it alongside skill-dir sees
# the same argument-count behaviour as one who does not pass it at all, and a
# genuinely extra positional argument is still rejected.
#
# BASE_REF_OVERRIDE is deliberately left UNSET here rather than initialised to
# the empty string. `--base-ref=` (given, but empty) and "no flag at all" are
# different instructions — the first says "use the default resolution, ignoring
# the environment", the second says "fall back to the environment" — and an
# empty-string initialiser collapsed them: `${BASE_REF_OVERRIDE:-$ENV}` treats
# an empty flag value as absent, so the environment variable won and the usage
# text's "the flag wins if both are given" was false for exactly that spelling.
declare -a POSITIONAL_ARGS=()
BASE_REF_OVERRIDE=""
for arg in "$@"; do
case "$arg" in
--base-ref=*)
@@ -107,10 +118,15 @@ fi
SKILL_DIR_ARG="${POSITIONAL_ARGS[0]}"
# The flag wins over the environment variable when both are given; either is
# empty-string when unset, and an empty string tells the Python body to fall
# The flag wins over the environment variable whenever the flag was GIVEN —
# `+x` tests for presence, not for a non-empty value, which is the distinction
# `:-` could not make. An empty result either way tells the Python body to fall
# back to `git merge-base HEAD origin/main`.
BASE_REF="${BASE_REF_OVERRIDE:-${VALIDATE_PROVENANCE_BASE_REF:-}}"
if [[ -n "${BASE_REF_OVERRIDE+x}" ]]; then
BASE_REF="$BASE_REF_OVERRIDE"
else
BASE_REF="${VALIDATE_PROVENANCE_BASE_REF:-}"
fi
# python3 is a HARD dependency. Without this preflight a missing interpreter
# produced 'line NN: python3: command not found' and exit 127 — an exit code no
@@ -491,10 +507,21 @@ def find_repo_root(start_dir):
# --- Check 9 helpers ---------------------------------------------------
# Check 9 needs a raw field VALUE (as text, to diff against an earlier
# version), not the parsed structure parse_contributing_files() and
# parse_status() return — a Contributing files list that reordered its
# entries without changing them is not what this check is looking for, but
# neither is normalizing so hard that a genuine rewrite disappears. Raw text,
# whitespace-normalized, is the middle ground.
# parse_status() return. The ONE normalization applied is whitespace
# collapsing, which is what makes a re-wrap or a re-indent invisible; nothing
# else is normalized away.
#
# In particular a REORDERED Contributing files list DOES fire this check, and
# that is deliberate — the header here used to claim the opposite, which the
# code never did. Order-insensitivity cannot be had for one field without
# distorting the other: the two fields share this parser, and the only way to
# ignore order is to split the value into items and sort them, which for a
# prose Description means splitting on commas and would then hide a genuine
# rewrite that merely permuted its clauses. Check 9 is always an INFO whose
# whole job is to point a human at a place to read; a reordered list costs
# that human one glance to dismiss, whereas a hidden rewrite is the exact
# failure #118 exists to catch. False positive over false negative, on this
# check, on purpose.
def run_git(args, cwd):
"""Run `git <args>` in cwd. Returns (returncode, stdout, stderr) — never
@@ -523,14 +550,44 @@ def find_slug_block(content, slug):
m = pattern.search(content)
return m.group(1) if m else None
# A field value ENDS at the next field, the next heading, or a blank line.
# Every other non-blank line is a continuation of the value the author wrapped
# across physical lines.
#
# This boundary is what the old `(.+)$` regex did not have. `.` does not cross
# a newline, so only the FIRST physical line of a wrapped value was ever
# compared — and a rewrite confined to a continuation line produced no finding
# at all. That is verbatim the hedge-to-confident-claim regression #118 exists
# to catch, invisible to the check written to catch it. The bullet branch had
# the same defect one level down: a wrapped bullet's continuation does not
# start with '- ', so the loop broke there and silently dropped every
# remaining bullet.
#
# A continuation line that itself opens with bold text ('**note** — ...') is
# read as a boundary and truncates the value. That is a known, narrow
# false-negative, accepted because the alternative — no boundary at all —
# is what produced the wide one above.
FIELD_BOUNDARY_RE = re.compile(r'^(?:- )?\*\*|^#{1,6} ')
def _is_field_boundary(stripped_line):
"""True when a stripped line starts a new field, bullet-less heading or H2."""
return bool(FIELD_BOUNDARY_RE.match(stripped_line))
def parse_field_raw(content, slug, field_name):
"""Raw text of a '**<field_name>:**' field under a slug H2.
"""Raw text of a '**<field_name>:**' field under a slug H2, wrapping joined.
Mirrors the two authored shapes parse_contributing_files() and
parse_status() already handle (inline value on the same line, or a
bare heading followed by '- ' bullets), but returns text rather than a
parsed structure, because check 9 diffs wording, not semantics.
Continuation lines are joined into the value they belong to before the
caller normalizes and compares, so a value wrapped across two lines and
the same value on one line are the same text — and a change made on any
line of a wrapped value is visible, not just one made on the first.
Returns None when the H2 itself is absent (the slug did not exist at
this content's revision) or the field is absent — both read as "no
earlier claim to compare against" to the caller, which is deliberate:
@@ -539,28 +596,49 @@ def parse_field_raw(content, slug, field_name):
block = find_slug_block(content, slug)
if block is None:
return None
inline_re = re.compile(r'^\- \*\*' + re.escape(field_name) + r':\*\* (.+)$', re.MULTILINE)
im = inline_re.search(block)
if im:
return im.group(1).strip()
heading_re = re.compile(r'^\*\*' + re.escape(field_name) + r':\*\*\s*$', re.MULTILINE)
hm = heading_re.search(block)
if not hm:
return None
lines = []
for line in block[hm.end():].splitlines():
line = line.strip()
if not line:
if lines:
break
continue
if not line.startswith("- "):
break
lines.append(line[2:].strip())
return ", ".join(lines) if lines else None
lines = block.splitlines()
inline_re = re.compile(r'^\- \*\*' + re.escape(field_name) + r':\*\*[ \t]*(.*)$')
heading_re = re.compile(r'^\*\*' + re.escape(field_name) + r':\*\*[ \t]*$')
for idx, line in enumerate(lines):
im = inline_re.match(line)
if im:
parts = [im.group(1).strip()]
for cont in lines[idx + 1:]:
stripped = cont.strip()
if not stripped or stripped.startswith("- ") or _is_field_boundary(stripped):
break
parts.append(stripped)
joined = " ".join(p for p in parts if p).strip()
return joined or None
if heading_re.match(line):
entries = []
for cont in lines[idx + 1:]:
stripped = cont.strip()
if not stripped:
if entries:
break
continue
if _is_field_boundary(stripped):
break
if stripped.startswith("- "):
entries.append(stripped[2:].strip())
elif entries:
# A wrapped bullet: fold it back into the bullet it
# continues rather than ending the list here.
entries[-1] = (entries[-1] + " " + stripped).strip()
else:
break
return ", ".join(e for e in entries if e) or None
return None
def normalize_field_text(value):
"""Collapse whitespace so reformatting alone never registers as a change."""
"""Collapse whitespace so reformatting alone never registers as a change.
True only because parse_field_raw() joins wrapped continuation lines
first: collapsing whitespace inside a value that had already been
truncated at its first newline normalized nothing a re-wrap could change.
"""
return re.sub(r'\s+', ' ', value).strip()
findings = []
@@ -1025,28 +1103,82 @@ else:
["show", f"{resolved_base_ref}:{sources_md_relpath}"], repo_root
)
if rc != 0:
# The base ref resolved fine, but references/sources.md did not
# exist there at all — the whole file is new. Every entry in it
# is therefore a creation, not a change: nothing to flag, and
# this is not a structural failure of the check, so no INFO
# either. Same reasoning applies per-slug below when the ref
# resolved but a given '## <slug>' heading did not exist yet.
# The base ref resolved fine but `git show <ref>:<path>` did not.
# That single return code covers two situations this check cannot
# tell apart, and only one of them is harmless:
#
# the file genuinely did not exist at the base ref — the whole
# sources.md is new, every entry in it is a creation, and there
# is nothing check 9 could have flagged;
#
# the path is not TRACKED under that name at the base ref — a
# renamed skill directory, or a copy of the skill living
# somewhere untracked or gitignored (an installed .claude/skills
# tree is the everyday case).
#
# Treating both as "creation, nothing to flag" made the second one
# a silent, whole-skill skip: the same directory audited at its
# authoring path reported changed claims and at its deployed path
# reported nothing, with no way to tell that from a clean run.
# That is the exact fail-open this script's own header forbids —
# "never a silent skip" — so announce it once for the whole check
# and hand over git's own stderr, which is the only diagnostic
# that separates the two cases.
detail = show_err.strip().splitlines()
detail = detail[0] if detail else "git gave no reason"
emit_info(
f"Check 9 skipped — '{sources_md_relpath}' is not tracked at {resolved_base_ref}",
"references/sources.md",
f"`git show {resolved_base_ref}:{sources_md_relpath}` failed ({detail}). "
f"Either the file did not exist at that ref — in which case every entry is a "
f"creation and there was nothing to flag — or this path is not tracked under "
f"that name there: a renamed skill directory, or an untracked or gitignored copy "
f"of the skill such as a deployed .claude/skills/ tree. "
f"Check 9 did not run for any slug in this skill. "
f"Re-run against the tracked authoring path, or pass --base-ref=<ref> naming a "
f"commit where this path exists."
)
old_sources_content = None
if old_sources_content is not None:
for slug in unique_slugs:
changed_fields = []
removed_fields = []
for field_name in ("Description", "Contributing files"):
old_value = parse_field_raw(old_sources_content, slug, field_name)
new_value = parse_field_raw(sources_content, slug, field_name)
if old_value is None or new_value is None:
if old_value is None and new_value is None:
continue
if old_value is None:
# No earlier claim to compare against — a brand-new
# entry, or a field that did not exist yet at the
# base ref. That is a creation, not a change, and is
# never flagged.
continue
if new_value is None:
# The field existed at the base ref and is gone now.
# This was folded into the creation skip above, which
# justified only the other half: deleting a whole
# '- **Description:**' line left NO finding anywhere —
# no other check in this script requires the field, so
# a claim could be removed as invisibly as it could be
# strengthened. Announce it; the auditor decides
# whether the removal was intended.
removed_fields.append(field_name)
continue
if normalize_field_text(old_value) != normalize_field_text(new_value):
changed_fields.append(field_name)
if removed_fields:
removed_list = " and ".join(removed_fields)
emit_info(
f"'{removed_list}' removed for '{slug}' since {resolved_base_ref}",
f"references/sources.md (## {slug})",
f"The '## {slug}' entry had {removed_list} at {resolved_base_ref} and has "
f"none now. Nothing else in this script requires the field, so the removal "
f"is otherwise invisible. Confirm it was deliberate — a provenance claim "
f"withdrawn is as much a change to the chain as one rewritten — and "
f"restore the field if it was lost to an edit."
)
if changed_fields:
field_list = " and ".join(changed_fields)
emit_info(

101
plugins/kyberforge/.apm/skills/skill-audit/scripts/validate.sh
View File

@@ -1289,6 +1289,50 @@ else:
if desc:
ok("description has no unfilled placeholders")
# --- ADR-0022: metadata.version is mandatory -------------------------------
# FAIL, not SUGGESTION, and the tier is set by the gate rather than by taste.
# `.pre-commit-config.yaml`'s `skill-frontmatter` hook REJECTS a SKILL.md with
# no `metadata.version`, and rejects a value that is not three-part semver.
# skill-author's Step 4 says to run this audit and "resolve every FAIL", so any
# tier below FAIL lets that step report done on a skill the commit gate then
# refuses — the same audit-disagrees-with-the-gate failure the MAX_LINES note
# below warns about, arrived at from the other direction. Verified before this
# check existed: a SKILL.md with no `metadata:` block at all reported "All
# checks passed".
#
# The rule is DUPLICATED from that hook for the same cache-isolation reason as
# every other constant here — an installed plugin's scripts cannot read the
# repo-root config. Keep the two in step: this check must accept exactly what
# the hook accepts.
SEMVER_RE = re.compile(r'^\d+\.\d+\.\d+$')
try:
fm_data = yaml.safe_load(fm)
except Exception:
# Unreachable in practice: description_value() above parses the same text
# and hard-exits on a YAML error, so anything arriving here already parsed.
fm_data = None
metadata_block = fm_data.get('metadata') if isinstance(fm_data, dict) else None
if not isinstance(metadata_block, dict) or metadata_block.get('version') is None:
fail("frontmatter has no metadata.version — ADR-0022 makes it mandatory for "
"every skill, and the skill-frontmatter pre-commit hook rejects the file "
"without it. Add `metadata:` / ` version: \"1.0.0\"` (new skills start "
"at \"0.1.0\")")
else:
version_value = metadata_block['version']
# NOT str()-coerced blind: `version: 1.0` is a YAML float, and its "1.0"
# spelling is exactly the two-part value the hook rejects — coercing and
# then matching keeps this check and the hook agreeing on that case.
version_text = version_value if isinstance(version_value, str) else str(version_value)
version_text = version_text.strip()
if SEMVER_RE.match(version_text):
ok(f"metadata.version present: '{version_text}' (ADR-0022)")
else:
fail(f"metadata.version '{version_text}' is not three-part semver — the "
f"skill-frontmatter pre-commit hook rejects it. Use MAJOR.MINOR.PATCH, "
f"e.g. \"1.0.0\"")
# SKILL.md size ceilings (agentskills.io skill-authoring.md: 500 lines,
# ~5,000 tokens). Both constants are DUPLICATED from the repo-root pre-commit
# hook scripts/skill-size-check.sh — a plugin skill's scripts cannot read files
@@ -1515,11 +1559,66 @@ def stdin_redirected(line, prev_line):
unquoted = re.sub(r'"[^"]*"|\'[^\']*\'', '', line)
return '<' in unquoted or prev_line.rstrip().endswith('|')
# A here-doc body is DATA, not command position. Every script in this corpus
# carries a `usage() { cat <<EOF ... EOF; }`, and prose wrapped inside one puts
# ordinary English at the start of a line — "read is reported as an INFO ..."
# in this skill's own validate-provenance.sh, which made skill-audit hard-FAIL
# on its own script. Reflowing that one sentence would have cleared the finding
# and left the cause: every future usage text is one wrap away from the same
# false positive, and the remedy an author reaches for is contorting working
# source, which the note above records has already happened twice.
#
# Detection is deliberately conservative in the direction that matters. A
# here-doc body is skipped only when its terminator is actually found further
# down the file; an opener with no terminator — the shape a stray `<<` inside a
# string would produce — is ignored rather than allowed to swallow the tail,
# because swallowing the tail is a false NEGATIVE and this check exists to fail
# closed. `<<<` here-strings open nothing and are excluded by the lookbehind.
HEREDOC_START_RE = re.compile(r'(?<!<)<<-?\s*(["\']?)([A-Za-z_][A-Za-z0-9_]*)\1')
def heredoc_delimiter(line):
"""The here-doc terminator this line opens, or None."""
m = HEREDOC_START_RE.search(line)
return m.group(2) if m else None
def heredoc_body_indices(lines):
"""Line indices that are here-doc BODY (plus its terminator), not code."""
skip = set()
i, n = 0, len(lines)
while i < n:
stripped = lines[i].strip()
delim = None if stripped.startswith('#') else heredoc_delimiter(lines[i])
if delim:
# `<<-` allows an indented terminator, so compare stripped.
for j in range(i + 1, n):
if lines[j].strip() == delim:
skip.update(range(i + 1, j + 1))
i = j
break
i += 1
return skip
# The here-doc exemption applies to the `read` heuristic ONLY, and the
# asymmetry is the point. `read` is an ordinary English verb, so any prose a
# script prints is one line-wrap away from opening with it. `input(` is not a
# word — a line beginning `input(` inside a here-doc is an embedded Python
# program pausing for a keypress, which is exactly what this check is for, and
# these scripts embed Python in a here-doc as a matter of course. Exempting the
# whole body would have disarmed the check across every script in the corpus.
def interactive_reads(source):
hits = []
prev_line = ''
for line in source.splitlines():
lines = source.splitlines()
in_heredoc = heredoc_body_indices(lines)
for idx, line in enumerate(lines):
stripped = line.strip()
if idx in in_heredoc:
if re.match(r'input\(', stripped):
hits.append(stripped)
continue
if re.match(r'read(\s|$)', stripped):
if not stdin_redirected(line, prev_line):
hits.append(stripped)

253
plugins/kyberforge/.apm/skills/skill-audit/tests/validate-provenance.bats
View File

@@ -67,6 +67,25 @@ EOF
EOF
}
# Helper: a sources.md whose Description is wrapped across two physical
# lines — the shape check 9's parser used to truncate at the first newline.
make_wrapped_sources_md() {
local dir="$1"
mkdir -p "$dir/references"
cat > "$dir/references/sources.md" <<'EOF'
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source, informing the dispatch table's shape with
no forge-specific content drawn directly from it beyond that.
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Status:** `extracted`
EOF
}
# Helper: turn dir into a real git repo with one commit of its current
# contents, and a refs/remotes/origin/main pointing at that same commit.
# Check 9 diffs the skill's references/sources.md against `git merge-base
@@ -1629,23 +1648,247 @@ EOF
assert_output --partial "Check 9 skipped — no base ref could be resolved"
}
@test "check 9: --base-ref overrides the default origin/main resolution" {
@test "check 9: --base-ref overrides a default origin/main that resolves to something else" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
commit_as_base "$skill"
local base_sha
base_sha="$(git -C "$skill" rev-parse HEAD)"
git -C "$skill" update-ref -d refs/remotes/origin/main >/dev/null 2>&1
local old_sha
old_sha="$(git -C "$skill" rev-parse HEAD)"
# A SECOND commit carrying the rewritten claim, with origin/main moved onto
# it. The default base ref therefore resolves — to a commit that matches the
# working tree — so a silent default run proves there was a default here to
# override. Deleting origin/main instead, as this test used to, proved only
# that the flag works when nothing else does.
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
"$skill/references/sources.md"
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
git -C "$skill" update-ref refs/remotes/origin/main HEAD >/dev/null 2>&1
run bash "$SCRIPT" "$skill" "--base-ref=$base_sha"
run bash "$SCRIPT" "$skill"
assert_success
assert_output ""
run bash "$SCRIPT" "$skill" "--base-ref=$old_sha"
assert_success
assert_output --partial "'Description' changed for 'my-source'"
}
@test "check 9: VALIDATE_PROVENANCE_BASE_REF sets the base ref when no flag is given" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
commit_as_base "$skill"
local old_sha
old_sha="$(git -C "$skill" rev-parse HEAD)"
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
"$skill/references/sources.md"
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
git -C "$skill" update-ref refs/remotes/origin/main HEAD >/dev/null 2>&1
run env VALIDATE_PROVENANCE_BASE_REF="$old_sha" bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "'Description' changed for 'my-source'"
}
@test "check 9: the --base-ref flag wins over the environment variable" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
commit_as_base "$skill"
local old_sha
old_sha="$(git -C "$skill" rev-parse HEAD)"
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
"$skill/references/sources.md"
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
local new_sha
new_sha="$(git -C "$skill" rev-parse HEAD)"
# The environment names the old commit (which would fire), the flag names
# the new one (which would not). The usage text promises the flag wins.
run env VALIDATE_PROVENANCE_BASE_REF="$old_sha" bash "$SCRIPT" "$skill" "--base-ref=$new_sha"
assert_success
assert_output ""
}
@test "check 9: an EMPTY --base-ref is still 'given' and wins over the environment variable" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
commit_as_base "$skill"
local old_sha
old_sha="$(git -C "$skill" rev-parse HEAD)"
sed -i 's/^- \*\*Description:\*\* A test source\.$/- **Description:** A rewritten claim./' \
"$skill/references/sources.md"
git -C "$skill" -c user.email=test@example.com -c user.name=test commit -aqm rewrite >/dev/null 2>&1
git -C "$skill" update-ref refs/remotes/origin/main HEAD >/dev/null 2>&1
# `--base-ref=` selects the DEFAULT resolution (origin/main, which now
# matches the working tree), so nothing fires. Under the old `:-` spelling
# the empty value read as absent and the environment variable won, firing
# the INFO and contradicting the usage text.
run env VALIDATE_PROVENANCE_BASE_REF="$old_sha" bash "$SCRIPT" "$skill" "--base-ref="
assert_success
assert_output ""
}
# ---------------------------------------------------------------------------
# Cycle 29 — Check 9: a WRAPPED field value. The parser compared only the first
# physical line, so a rewrite confined to a continuation line — the exact
# hedge-to-confident-claim shape #118 exists to catch — produced no finding at
# all, while a pure re-wrap produced a false one.
# ---------------------------------------------------------------------------
@test "check 9: a rewrite confined to a wrapped Description's CONTINUATION line fires an INFO" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_wrapped_sources_md "$skill"
commit_as_base "$skill"
# Hedge to confident claim, on the second physical line only. This is the
# regression check 9 was written for, and the one it could not see.
sed -i "s|^ no forge-specific content drawn directly from it beyond that\.\$| it grounds Step 2's dispatch table in full.|" \
"$skill/references/sources.md"
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "'Description' changed for 'my-source'"
}
@test "check 9: re-wrapping a Description with no wording change produces no finding" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_wrapped_sources_md "$skill"
commit_as_base "$skill"
# Same words, different line breaks. normalize_field_text()'s docstring
# promises this is invisible; it was not, because the value was truncated
# at its first newline before the whitespace collapse ever ran.
python3 - "$skill/references/sources.md" <<'PY'
import sys
path = sys.argv[1]
text = open(path).read()
old = ("- **Description:** A test source, informing the dispatch table's shape with\n"
" no forge-specific content drawn directly from it beyond that.")
new = ("- **Description:** A test source, informing the dispatch\n"
" table's shape with no forge-specific content drawn\n"
" directly from it beyond that.")
assert old in text
open(path, 'w').write(text.replace(old, new))
PY
run bash "$SCRIPT" "$skill"
assert_success
assert_output ""
}
@test "check 9: the bullet form of Contributing files is compared, not skipped" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<'EOF'
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
**Contributing files:**
- SKILL.md (the dispatch table)
- references/other.md (the rubric)
- **Research doc:** (none)
- **Status:** `extracted`
EOF
printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md"
commit_as_base "$skill"
# The change is in the SECOND bullet. The old loop joined bullets in
# document order too, but broke on any wrapped one — and no test covered
# this branch at all, both existing check-9 tests using the inline form.
sed -i 's|^- references/other.md (the rubric)$|- references/other.md (the whole rubric, verbatim)|' \
"$skill/references/sources.md"
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "'Contributing files' changed for 'my-source'"
}
@test "check 9: a wrapped bullet does not silently drop the bullets after it" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<'EOF'
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
**Contributing files:**
- SKILL.md (the dispatch table, and the gates
common to every branch of it)
- references/other.md (the rubric)
- **Research doc:** (none)
- **Status:** `extracted`
EOF
printf -- '---\nsource_keys:\n - my-source\n---\n\nnotes\n' > "$skill/references/other.md"
commit_as_base "$skill"
# The old loop broke at the wrapped continuation line, so everything from
# here down was never part of the compared value — a change to the last
# bullet was invisible.
sed -i 's|^- references/other.md (the rubric)$|- references/other.md (rewritten claim)|' \
"$skill/references/sources.md"
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "'Contributing files' changed for 'my-source'"
}
@test "check 9: a field present at the base ref and deleted since is announced" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
commit_as_base "$skill"
# No other check in this script requires a Description, so a deleted one
# used to leave no finding anywhere: a claim could be withdrawn as
# invisibly as it could be strengthened.
sed -i '/^- \*\*Description:\*\*/d' "$skill/references/sources.md"
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "'Description' removed for 'my-source'"
}
@test "check 9: a sources.md untracked at the base ref is announced, not silently skipped" {
local repo="$TMPDIR/repo"
local skill="$repo/tracked-skill"
mkdir -p "$skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
commit_as_base "$repo"
# A copy of the same skill at a path git does not know — the everyday case
# being an installed, gitignored .claude/skills/ tree. The base ref
# resolves fine; `git show <ref>:<path>` does not. Treating that as
# "creation, nothing to flag" made the whole check vanish without a word,
# so the same directory reported findings at one path and silence at the
# other.
cp -r "$skill" "$repo/untracked-copy"
run bash "$SCRIPT" "$repo/untracked-copy"
assert_success
assert_output --partial "INFO"
assert_output --partial "is not tracked at"
assert_output --partial "Check 9 did not run for any slug in this skill."
}
@test "check 9: an invalid --base-ref value is reported as unresolvable, not a crash" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"

147
plugins/kyberforge/.apm/skills/skill-audit/tests/validate.bats
View File

@@ -16,6 +16,10 @@ setup() {
# SUGGESTION-freedom would be asserting the boundary check's absence instead
# of the thing it names. "anything else" is not hyphenated, so the clause adds
# a boundary marker without adding a routing target to resolve.
#
# metadata.version is equally load-bearing: ADR-0022 makes it mandatory and
# validate.sh FAILs without it, so a fixture omitting it would not be
# "otherwise clean" either.
make_valid_skill() {
local dir="$1"
local name
@@ -25,6 +29,8 @@ setup() {
---
name: $name
description: A valid skill description that is well within the limit. Do not use for anything else.
metadata:
version: "1.0.0"
---
## Step 1
@@ -59,6 +65,8 @@ PY
echo "---"
echo "name: $name"
echo "description: $desc"
echo "metadata:"
echo ' version: "1.0.0"'
echo "---"
echo ""
python3 -c "print(' '.join(['word'] * $body_words))"
@@ -294,6 +302,141 @@ EOF
assert_success
}
@test "prose inside a usage() here-doc that wraps onto a line starting with 'read' does not fail" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
# The exact shape that made skill-audit hard-FAIL on its own
# validate-provenance.sh: a usage() heredoc whose wrapped sentence puts the
# English verb "read" in column 0.
cat > "$skill/scripts/helper.sh" <<'SH'
#!/usr/bin/env bash
usage() {
cat <<EOF
Checks performed:
4 A Contributing files block this parser cannot
read is reported as an INFO, never skipped silently.
EOF
}
usage
SH
chmod +x "$skill/scripts/helper.sh"
run bash "$SCRIPT" "$skill"
assert_success
}
@test "a real interactive read AFTER a here-doc is still caught" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
# Pins that the here-doc exemption ends at its terminator. A body skip that
# ran to end-of-file would swallow this read and report the script clean.
cat > "$skill/scripts/helper.sh" <<'SH'
#!/usr/bin/env bash
cat <<EOF
read this text
EOF
read -r ANSWER
SH
chmod +x "$skill/scripts/helper.sh"
run bash "$SCRIPT" "$skill"
assert_failure
}
@test "an unterminated here-doc opener does not disarm the check for the rest of the file" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
# `<<` here is inside a string, not an opener. Treating it as one would skip
# every following line — a false negative, the direction this check must
# never fail in.
cat > "$skill/scripts/helper.sh" <<'SH'
#!/usr/bin/env bash
echo "shift left with a << b"
read -r ANSWER
SH
chmod +x "$skill/scripts/helper.sh"
run bash "$SCRIPT" "$skill"
assert_failure
}
@test "a bare input() inside an embedded-python here-doc is still caught" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
# The here-doc exemption is for the `read` heuristic only: these scripts
# embed Python in a here-doc as a matter of course, so exempting the body
# wholesale would disarm the check across the corpus.
cat > "$skill/scripts/helper.sh" <<'SH'
#!/usr/bin/env bash
python3 - <<'PY'
print("press enter")
input()
PY
SH
chmod +x "$skill/scripts/helper.sh"
run bash "$SCRIPT" "$skill"
assert_failure
}
# ---------------------------------------------------------------------------
# ADR-0022 — metadata.version is mandatory. FAIL tier, matching the
# skill-frontmatter pre-commit hook: an audit that graded this lower would
# report ready-to-ship on a file the commit gate rejects.
# ---------------------------------------------------------------------------
@test "ADR-0022: a SKILL.md with no metadata block at all FAILs" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
python3 - "$skill/SKILL.md" <<'PY'
import sys
p = sys.argv[1]
s = open(p).read().replace('metadata:\n version: "1.0.0"\n', '')
open(p, 'w').write(s)
PY
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "metadata.version"
}
@test "ADR-0022: a metadata block with no version key FAILs" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
python3 - "$skill/SKILL.md" <<'PY'
import sys
p = sys.argv[1]
s = open(p).read().replace(' version: "1.0.0"\n', ' category: factory\n')
open(p, 'w').write(s)
PY
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "metadata.version"
}
@test "ADR-0022: a two-part metadata.version FAILs as malformed, not passes as present" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
python3 - "$skill/SKILL.md" <<'PY'
import sys
p = sys.argv[1]
s = open(p).read().replace(' version: "1.0.0"\n', ' version: 1.0\n')
open(p, 'w').write(s)
PY
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "three-part semver"
}
@test "ADR-0022: an unquoted three-part metadata.version passes" {
local skill="$TMPDIR/my-skill"
make_valid_skill "$skill"
python3 - "$skill/SKILL.md" <<'PY'
import sys
p = sys.argv[1]
s = open(p).read().replace(' version: "1.0.0"\n', ' version: 0.1.3\n')
open(p, 'w').write(s)
PY
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "metadata.version present: '0.1.3'"
}
@test "fails when name contains consecutive hyphens" {
local skill="$TMPDIR/my--skill"
make_valid_skill "$skill"
@@ -608,6 +751,8 @@ make_hand_invoked_skill() {
echo "name: $name"
echo "description: $desc"
echo "disable-model-invocation: true"
echo "metadata:"
echo ' version: "1.0.0"'
echo "---"
echo ""
python3 -c "print(' '.join(['word'] * $body_words))"
@@ -761,6 +906,8 @@ make_hand_invoked_skill() {
---
name: locale-skill
description: A valid skill description that is well within the limit.
metadata:
version: "1.0.0"
---
## Step 1

2
tests/test-adr0020-contract.sh
View File

@@ -224,6 +224,8 @@ cat > "$SUBJECT_SKILL_DIR/SKILL.md" <<'EOF'
---
name: my-skill
description: A short valid description. Do not use for anything else.
metadata:
version: "1.0.0"
---
Do the thing.